Metadata-Version: 2.4
Name: webpage-parser
Version: 0.2.0
Summary: Extract structured fields (title, body, publication time, authors, language and more) from raw HTML pages
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: lxml==6.1.2
Requires-Dist: trafilatura==2.2.0
Requires-Dist: dateparser==1.4.2

# webpage-parser

Extract structured fields from a raw HTML page. Given one crawled web record,
`process_row` returns a fixed set of **35 fields** covering document identity,
URLs, the main article (title, body, headings, description, authors),
timestamps, language and content type.

## Install

```bash
pip install webpage-parser
```

Python 3.10+ is required (`trafilatura` and `dateparser` need it).

## Python API

```python
from webpage_parser import process_row

row = {
    "html": "<html>…</html>",       # the main input
    "url": "https://example.com/a",  # everything except html is optional
}
result = process_row(row)
print(result["title"], result["published_at"], result["language"])
```

`process_row(row)` reads the following keys from the input dict. Only `html`
really matters; without usable HTML most output fields stay empty.

| key | meaning |
| --- | --- |
| `html` | raw HTML, the main input |
| `url`, `durl`, `redirect_url` | candidate page URLs; first non-empty of `redirect_url` > `durl` > `url` wins |
| `purl` | passed through unchanged |
| `domain` | registrable-domain hint |
| `host` | full host-name hint |
| `headers` | HTTP response headers, a dict or a JSON string |
| `crawled_time`, `first_found_time`, `last_found_time` | epoch seconds (or milliseconds) |
| `anchor` | inbound anchor text |
| `in_domain_links`, `out_domain_links` | link counts; `(0, 0)` is treated as unknown |
| `raw_id`, `raw_id_version` | opaque document id / version |
| `extra` | optional dict; recognised sub-keys `time_zone` (offset in hours), `anchor`, `parser_name` |
| `img_ref` | optional dict of image references |

## Output

`process_row` returns a dict with exactly these 35 keys:

`document_id`, `document_version`, `url`, `url_hash`, `canonical_url`,
`canonical_url_hash`, `domain`, `registrable_domain`, `purl`, `site_name`,
`favicon_url`, `host_logo_url`, `image_url`, `title`, `headings`, `body`,
`description`, `authors`, `published_at`, `published_at_confidence`,
`published_at_source`, `updated_at`, `language`, `language_confidence`,
`country_codes`, `content_type`, `mime_type`, `content_return_policy`,
`parser_version`, `crawled_at`, `first_seen_at`, `last_seen_at`,
`anchor_text`, `in_domain_links`, `out_domain_links`.

Notes on selected fields:

- `title`, `body`, `description` are strings; `headings` and `authors` are lists.
- `domain` is the full host name; `registrable_domain` is the eTLD+1.
- `favicon_url`, `host_logo_url`, `image_url`, `country_codes` and
  `content_return_policy` are placeholders that are currently always empty.
- `parser_version` is a constant string.

`empty_result(row)` is also exported and returns the same 35-key dict with the
HTML-derived fields left empty. `process_row` degrades to it whenever a page
cannot be parsed, so callers always receive the complete key set.

## Time zone

- `published_at` and `updated_at` are wall-clock strings `YYYY-MM-DD HH:MM:SS`
  normalised to **UTC+8** (the default assumed site time zone). A value without
  an explicit offset is interpreted using `extra["time_zone"]`, an offset in
  hours, falling back to UTC+8 when absent.
- `crawled_at`, `first_seen_at` and `last_seen_at` are RFC 3339 timestamps in
  **UTC** with a trailing `Z`, derived from the epoch inputs.

## Command line

```bash
webpage-parser --input rows.jsonl --output parsed.jsonl
```

Each input line is a JSON object shaped like the `process_row` argument; each
output line is the 35-field JSON result.
