Metadata-Version: 2.4
Name: webpage-parser
Version: 0.3.0
Summary: Extract structured fields (title, body, publication time, authors, language and more) from raw HTML pages
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: beautifulsoup4==4.15.0
Requires-Dist: lxml==6.1.2
Requires-Dist: tldextract==5.3.2

# webpage-parser

Extract structured fields from a raw HTML page. Given one crawled web record,
`process_row` returns a fixed set of **35 fields** covering document identity,
URLs, the main article (title, body, headings, description, authors),
timestamps, language and content type.

## Install

```bash
pip install webpage-parser
```

Python 3.10+ is required. (The dependencies themselves only need 3.9 at most;
the floor is aligned with the cp311 runtime used to build the MaxCompute UDF.)

## Python API

`process_row` is the only entry point: hand it one dict, get back a dict with
exactly 35 keys.

```python
import json

from webpage_parser import empty_result, process_row

HTML = """<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Quantum computing reaches 100 qubits - Example Times</title>
  <meta name="description" content="Researchers entangled 100 superconducting qubits.">
  <meta property="og:site_name" content="Example Times">
  <meta name="author" content="Jane Doe">
  <meta property="article:published_time" content="2026-09-05T10:30:00+00:00">
  <link rel="canonical" href="https://tech.example.com/news/quantum-2026">
</head>
<body>
  <article>
    <h1>Quantum computing reaches 100 qubits</h1>
    <p>The team said a superconducting chip held a stable entangled state across
       100 qubits, tripling coherence time compared with the previous generation.</p>
    <h2>How it works</h2>
    <p>Tunable couplers suppress crosstalk, and dynamical decoupling extends the
       coherence time.</p>
  </article>
</body>
</html>
"""

# 1) Minimal call: html is the only key that really matters
result = process_row({"html": HTML})
print(result["title"], result["language"], len(result))
# Quantum computing reaches 100 qubits en 35

# 2) Full call: every key the parser looks at
row = {
    "raw_id": "doc-0001",
    "raw_id_version": "v1",
    "url": "https://tech.example.com/news/quantum?utm_source=x",
    "durl": None,
    "redirect_url": None,
    "purl": "https://tech.example.com/",
    "domain": "example.com",
    "host": "tech.example.com",
    "html": HTML,
    "headers": '{"Content-Type": "text/html; charset=utf-8"}',
    "crawled_time": 1789000000,
    "first_found_time": 1788000000,
    "last_found_time": 1789000000,
    "anchor": "quantum milestone",
    "in_domain_links": 12,
    "out_domain_links": 3,
}
out = process_row(row)

# 3) Batch: one JSON object per line in, one 35-field object per line out
with open("parsed.jsonl", "w") as f:
    for one in [row]:
        f.write(json.dumps(process_row(one), ensure_ascii=False) + "\n")

# 4) Fallback: the empty result carries the very same key set
blank = empty_result(row)
assert set(blank) == set(out)
```

What `out` actually holds:

```text
document_id             = 'doc-0001'
document_version        = 'v1'
url                     = 'https://tech.example.com/news/quantum?utm_source=x'
canonical_url           = 'https://tech.example.com/news/quantum-2026'
domain                  = 'tech.example.com'
registrable_domain      = 'example.com'
site_name               = 'Example Times'
title                   = 'Quantum computing reaches 100 qubits'
headings                = ['How it works']
description             = 'Researchers entangled 100 superconducting qubits.'
authors                 = ['Jane Doe']
published_at            = '2026-09-05T10:30:00+00:00'
published_at_source     = 'meta.article_published_time'
published_at_confidence = 0.95
language                = 'en'
language_confidence     = 0.7
content_type            = 'html'
mime_type               = 'text/html'
crawled_at              = '2026-09-10T00:26:40Z'
first_seen_at           = '2026-08-29T10:40:00Z'
last_seen_at            = '2026-09-10T00:26:40Z'
anchor_text             = 'quantum milestone'
in_domain_links         = 12
out_domain_links        = 3
```

Four things that tend to surprise people:

- `title` drops the trailing site suffix (`- Example Times`); the site name is
  reported separately as `site_name`.
- The `h1` was taken as the title, so it is not repeated in `headings`.
- `published_at` keeps the page's own UTC offset. The page declares
  `10:30:00+00:00`, so the output stays `2026-09-05T10:30:00+00:00`; see the
  Time zone section below.
- Every call returns all 35 keys, parsed or not.

`process_row(row)` reads the following keys from the input dict. Only `html`
really matters; without usable HTML most output fields stay empty.

| key | meaning |
| --- | --- |
| `html` | raw HTML, the main input |
| `url`, `durl`, `redirect_url` | candidate page URLs; first non-empty of `redirect_url` > `durl` > `url` wins |
| `purl` | passed through unchanged |
| `domain` | registrable-domain hint |
| `host` | full host-name hint |
| `headers` | HTTP response headers, a dict or a JSON string |
| `crawled_time`, `first_found_time`, `last_found_time` | epoch seconds (or milliseconds) |
| `anchor` | inbound anchor text |
| `in_domain_links`, `out_domain_links` | link counts; `(0, 0)` is treated as unknown |
| `raw_id`, `raw_id_version` | opaque document id / version |

## Output

`process_row` returns a dict with exactly these 35 keys:

`document_id`, `document_version`, `url`, `url_hash`, `canonical_url`,
`canonical_url_hash`, `domain`, `registrable_domain`, `purl`, `site_name`,
`favicon_url`, `host_logo_url`, `image_url`, `title`, `headings`, `body`,
`description`, `authors`, `published_at`, `published_at_confidence`,
`published_at_source`, `updated_at`, `language`, `language_confidence`,
`country_codes`, `content_type`, `mime_type`, `content_return_policy`,
`parser_version`, `crawled_at`, `first_seen_at`, `last_seen_at`,
`anchor_text`, `in_domain_links`, `out_domain_links`.

Notes on selected fields:

- `title`, `body`, `description` are strings; `headings` and `authors` are lists.
- `domain` is the full host name; `registrable_domain` is the eTLD+1.
- `favicon_url`, `host_logo_url`, `image_url`, `country_codes` and
  `content_return_policy` are placeholders that are currently always `None`.
- `parser_version` is a constant string (currently `'1.1.0'`).

`empty_result(row)` is also exported and returns the same 35-key dict with the
HTML-derived fields left empty. `process_row` degrades to it whenever a page
cannot be parsed, so callers always receive the complete key set.

## Time zone

- `published_at` and `updated_at` are ISO 8601 strings that **preserve the
  page's own UTC offset** — `Z` stays `Z`, `+00:00` stays `+00:00`, `+08:00`
  stays `+08:00`. When the page carries no explicit offset, **UTC+8** is
  assumed (a date-only value becomes `...T00:00:00+08:00`).
- `crawled_at`, `first_seen_at` and `last_seen_at` are RFC 3339 timestamps in
  **UTC** with a trailing `Z`, derived from the epoch inputs.

## Command line

```bash
webpage-parser --input rows.jsonl --output parsed.jsonl
```

Each input line is a JSON object shaped like the `process_row` argument; each
output line is the 35-field JSON result.
