Metadata-Version: 2.4
Name: webpage-parser
Version: 0.1.0
Summary: Extract title, content, publication time and author from raw HTML pages
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: lxml==6.1.2
Requires-Dist: trafilatura==2.2.0
Requires-Dist: dateparser==1.4.2

# webpage-parser

Extract four core fields from a raw HTML page: `title`, `content`, `pub_time`, `author`.

Rule-based extraction over the DOM, using `lxml` for parsing, `trafilatura` for
main-content candidates and `dateparser` for multilingual date fallback.
No network access is performed while parsing.

## Install

```bash
pip install webpage-parser
```

Requires Python >= 3.10 (`trafilatura` 2.2 and `dateparser` 1.4 both require it).

## Python API

```python
from webpage_parser import process_row

result = process_row({
    "html": "<html>...</html>",          # required
    "url": "https://example.com/a/b",    # optional, enables host-specific rules
    "time": "2026-01-01 00:00:00",       # optional, fetch time; upper bound for pub_time
    "headers": {},                       # optional dict
    "extra": {},                         # optional dict
})
```

Only `html` is required. The other keys refine the result: `url` selects
host-specific rules, `time` rejects publication dates later than the fetch time.

Returns a `dict`. The four core fields are always present and always strings --
an empty string when nothing was extracted, never `None`:

| key | meaning |
|---|---|
| `title` | page title |
| `content` | main body text |
| `pub_time` | publication time, formatted `YYYY-MM-DD HH:MM:SS` |
| `author` | author or publishing organisation |

Additional diagnostic keys describe how each field was obtained:
`page_status`, `title_source`, `title_suffix_stripped`, `content_path`,
`content_traceable_ratio`, `content_noise_by_class`, `pub_time_source`,
`pub_time_precision`, `pub_time_tz_offset`, `pub_time_tz_inferred`,
`pub_time_raw`, `author_source`, `author_is_org`.

`page_status` is one of `ok`, `no_article`, `list_page`, `spa_empty`, `non_html`.
Body text is only extracted from pages that reach `ok`; thin, list-like or
client-rendered pages report the corresponding status and leave `content` empty.

`empty_result()` is also exported and returns the baseline dict with every field
empty. `process_row` may add `pub_time_tz_inferred` on top of those keys, so treat
the returned key set as a superset rather than a fixed schema.

## Time zone

`pub_time` is normalised to **UTC+8**, the default assumed site time zone:

- a value carrying an explicit offset (`Z`, `+HH:MM`, `+HHMM`) is converted to
  UTC+8, and `pub_time_tz_offset` keeps the original offset in minutes
- a value without any offset is interpreted using `extra["time_zone"]`, an offset
  in hours (for example `0` or `-5`), falling back to UTC+8 when absent
- day-precision values are not shifted

## Command line

```bash
webpage-parser --input rows.jsonl --output preds.jsonl
```

- one JSON object per input line, same keys as `process_row`
- output lines align 1:1 with input lines, in the same order
- a malformed or failing row degrades to empty values instead of aborting;
  the exit code is always 0
- streaming read and write, the input is never loaded as a whole

## Behaviour notes

- blank or non-string `html` yields `page_status = "non_html"`
- `html` longer than 8,000,000 characters is truncated before parsing
