Metadata-Version: 2.4
Name: haruka_parser
Version: 1.2.1
Summary: A simple HTML Parser
Home-page: https://github.com/prnake/haruka-parser
Author: papersnake
Author-email: prnake@gmail.com
Classifier: Programming Language :: Python :: 3
Classifier: License :: OSI Approved :: MIT License
Classifier: Operating System :: OS Independent
Classifier: Topic :: Text Processing
Classifier: Topic :: Text Processing :: Markup :: HTML
Classifier: Topic :: Utilities
Description-Content-Type: text/markdown
Requires-Dist: py_asciimath
Requires-Dist: inscriptis==2.5.0
Requires-Dist: tabulate
Requires-Dist: numpy
Requires-Dist: resiliparse
Requires-Dist: ftfy
Requires-Dist: faust-cchardet
Requires-Dist: lxml
Requires-Dist: lxml_html_clean
Requires-Dist: html5lib
Requires-Dist: python-dateutil
Requires-Dist: courlan
Requires-Dist: charset_normalizer
Requires-Dist: trafilatura
Requires-Dist: tldextract
Requires-Dist: pyahocorasick
Requires-Dist: nltk
Requires-Dist: uniseg
Requires-Dist: pyyaml
Provides-Extra: dev
Requires-Dist: fasttext; extra == "dev"
Provides-Extra: justext
Requires-Dist: fugashi; extra == "justext"
Requires-Dist: unidic-lite; extra == "justext"
Requires-Dist: python-mecab-ko; extra == "justext"
Requires-Dist: python-mecab-ko-dic; extra == "justext"
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: summary

# Haruka Parser

A simple HTML Parser

## Install

```bash
pip install haruka-parser
```

## Usage

### V2（推荐）

V2 是多引擎投票的抽取管线：trafilatura 与 readability 两路主内容启发式对段落投票，
只保留双引擎都命中的正文块；在此基础上叠加 math/code/table 的结构化转换与块级去重。

```python3
from haruka_parser.v2.extractors.auto_extractor import AutoExtractor

extractor = AutoExtractor()
result = extractor.extract(html, url="https://example.com/page.html")

print(result["content"])     # Markdown 正文
print(result["title"])       # 标题
print(result["time"])        # 发布时间 "2024-05-11 08:30:00"（无则 ""）
print(result["time_stamp"])  # 对应 unix timestamp（字符串，无则 ""）
```

返回字段：

| 字段 | 说明 |
| --- | --- |
| `content` | Markdown 格式正文 |
| `paragraphs` | 段落结构（含 unique_id、tag、dom meta），可回放 |
| `dom_attrs` / `links` | DOM 属性与页面链接（链接按 base_url 补全） |
| `title` / `time` / `time_stamp` | 标题、发布时间及其 timestamp |
| `base_url` / `encoding` | 最终基准 URL 与检测到的编码 |

常用参数：

```python3
extractor.extract(
    html,
    url="https://example.com/page.html",  # 用于链接补全、域名特判、标题回退
    separator="\n\n",                     # 段落分隔符
    table_format="github",                # github / grid / simple / html / html_raw / html_auto
    escape_dollars=False,                 # 非 LaTeX 上下文的 $ 转义
    use_ftfy=False,                       # ftfy 文本修复
    magic_html_heuristic=False,           # 引入 magic-html 作为第三路投票
    dedup=True,                           # 块级去重（页面内重复渲染的内容只留第一份）
)
```

#### clean 模式：已清洗的上游输入

当上游 HTML 已经清洗过（例如只有正文的 body 片段）时，用 `mode="clean"` 跳过所有会删正文
的内容启发式（traf/read/magic 投票、link_list 打标、dedup、clean_tree、wiki 特判、
交互/不可见/广告元素清理），只做 math/code/table 等结构化 DOM 处理和 paragraphs 转换。
标题/时间抽取照常执行；`script`/`head`/`iframe` 等非正文标签仍会剔除。

```python3
result = extractor.extract(cleaned_body_html, mode="clean", url="https://example.com/docs/")
```

### V1（旧接口）

```python3
from haruka_parser.extract import extract_text

html = """<!DOCTYPE html>
<html>
<body>
<!-- Using MathML -->
<p>Using MathJax:</p>
<script type="math/tex; mode=display" id="MathJax-Element-1">{e}^{i\pi }=-1</script>
<!-- Using MathML -->
<p>Using MathML:</p>
<math xmlns="http://www.w3.org/1998/Math/MathML">
  <msup>
    <mi>e</mi>
    <mrow>
      <mi>i</mi>
      <mi>&#x03C0;</mi>
    </mrow>
  </msup>
  <mo>=</mo>
  <mn>-1</mn>
</math>

<!-- Using AsciiMath -->
<p>Using AsciiMath:</p>
<script type="math/asciimath">
e^(i*pi) = -1
</script>

</body>
</html>"""

text, info = extract_text(html)
print(text)
print(info)
```

## Configurations

```python3
from haruka_parser.extract import DEFAULT_CONFIG
DEFAULT_CONFIG = {
    "readability": False,
    "skip_large_links": False,
    "extract_latex": True,
    "extract_cnki_latex": False,
    "escape_dollars": True,
    "remove_buttons": True,
    "remove_edit_buttons": True,
    "remove_image_figures": True,
    "markdown_code": True,
    "markdown_headings": True,
    "remove_chinese": False,
    "boilerplate_config": {
        "enable": False,
        "ratio_threshold": 0.18,
        "absolute_threshold": 10,
        "end_threshold": 15,
    },
}
```

## Parsing speed

10k Page:
| method | haruka-parser 0.5.2 | haruku-parser 0.4.9 | html2text | inscriptis | trafilatura |
| ------- | --------- | ------- | ------- | -------- | ----- |
| Speed   | 379.4s | 391.6s | 272.8s | 114.7s | 343.9s |
