Metadata-Version: 2.4
Name: forktex-scraping
Version: 0.1.0
Summary: Acquire public data and prove what you acquired: captured request contracts, verified filters, and a coverage report that names what it could not reach.
License-Expression: AGPL-3.0-or-later OR LicenseRef-ForkTex-Commercial
License-File: LICENSE
License-File: NOTICE
Author: FORKTEX
Author-email: info@forktex.com
Requires-Python: >=3.14,<4.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Internet :: WWW/HTTP
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing :: Markup :: HTML
Classifier: Typing :: Typed
Provides-Extra: browser
Provides-Extra: html
Provides-Extra: http
Requires-Dist: forktex (>=0.10,<0.11)
Requires-Dist: httpx (>=0.28) ; extra == "http"
Requires-Dist: playwright (>=1.45) ; extra == "browser"
Requires-Dist: pydantic (>=2.12)
Requires-Dist: selectolax (>=0.3) ; extra == "html"
Project-URL: Bug Tracker, https://github.com/forktex/forktex-scraping/issues
Project-URL: Changelog, https://github.com/forktex/forktex-scraping/blob/master/CHANGELOG.md
Project-URL: Documentation, https://github.com/forktex/forktex-scraping/tree/master/docs
Project-URL: Homepage, https://forktex.com
Project-URL: Repository, https://github.com/forktex/forktex-scraping
Description-Content-Type: text/markdown

# forktex-scraping

Acquire public data, and be able to prove what you acquired.

```bash
pip install forktex-scraping[http]
```

An acquisition framework, not a scraper. Scraping a rendered page is its *lowest* tier, used when
nothing better exists — and across the four sources it was built against, two are official JSON
APIs and only one needs a browser at all.

> A remote that answers `200` to a request it did not understand is the adversary.
> Everything here exists to turn a silently wrong answer into a loud one.

## The problem it solves

A public procurement portal was measured to do all of this at once:

- reject any request without a `Referer`, with a message and no status code to match;
- answer every unfiltered query with `total: 3000, searchTooLong: true` — a **cap**, not a count;
- accept a filter field it did not recognise, **ignore it**, and return `200` with the capped,
  unfiltered set — byte-identical to sending no filter at all.

That last one is the reason this library exists. A guessed field name does not fail. It produces a
*confidently wrong* dataset, which is worse than an error, and no amount of care in the calling
code detects it.

## What it does about that

**A request contract is captured, not guessed.** A browser drives the site's own filter controls
once, and the wire name of every filter it honours is read off the traffic. That contract is data;
replaying it never needs a browser again.

**Every filter is verified on every page.** A contract's filter declares where its effect must
appear in a returned row. A row that violates it raises `ContractDriftError` — the remote answered
a different question than the one asked, and that is not something to patch client-side.

**Coverage is a claim that has to be earned.** A capped query is split along its axis until the
remote answers honestly. A window that cannot be split further and is still capped is neither
dropped nor quietly accepted:

```python
coverage = await harvest(source, contract, transport=..., politeness=..., sink=..., run_id=...)

if not coverage.complete:
    for window in coverage.truncated:   # named, not silently missing
        ...
```

**Politeness is a refusal, not a log line.** `robots.txt` is honoured — and it fails *open*, since
absence is not prohibition. The rate limiter is per-origin and shared across callers, so eight
concurrent workers at "one request per second" produce one request per second, not eight.
`Retry-After` wins over the computed backoff.

There is no stealth, no user-agent rotation and no proxy pool. The agent string says who you are.

## The seam: stateless versus stateful

The central distinction is not HTTP-versus-browser; it is **whether a thing can be replayed**.

| | `Transport` | `Session` |
|---|---|---|
| snapshot and replay | yes — a request hashes to a stable identity | **no** — order *is* the state |
| retry, pagination, cap-splitting | yes | not applicable |
| safe to run concurrently | yes, under the pacer | one session, one caller |

The rule that follows: **a session's output is never a result.** It is data — a `RequestContract`
or a `SourceDefinition` — which the stateless path then replays forever. An agent explores a site
statefully; what comes *out* is never records. That is what keeps a harvest reproducible.

## Sources are data

A source is a JSON file naming the model it loads into. Adding a **source** is a data change;
adding a **kind** is a code change, because a kind with no runner acquires nothing, silently.

```json
{ "$model": "ApiSource",
  "id": "source.example.notices",
  "tier": "captured_contract",
  "contract": "example/notices.json",
  "partition": {"axis": "publishedAt", "start_field": "from", "end_field": "to",
                "initial": "P1M", "floor": "P1D", "end_bound": "inclusive_day"},
  "politeness": {"rate_per_second": 2.0, "respect_robots": true} }
```

Five kinds, each earned by a real source: `ApiSource`, `BatchKeySource`, `CatalogSource`,
`FormStateSource`, `SitemapSource`.

## Driving it from an agent

`forktex_scraping.operations` describes the library as typed operations whose JSON Schemas come
free from Pydantic — with no MCP, agent-framework or SDK dependency. A harness binds them:

```python
from forktex_scraping import OPERATIONS, schemas

for op in OPERATIONS:
    register(name=op.name, description=op.summary, parameters=op.schema())
    # op.holds_resource marks the session group — scope those, or you leak browsers
```

## Install

| Extra | Pulls | For |
|---|---|---|
| `[http]` | `httpx` | the stateless transport — almost everything |
| `[browser]` | `playwright` | capturing a contract, once per source |
| `[html]` | `selectolax` | parsing a server-rendered table |

Extras import lazily: `import forktex_scraping` works with none of them installed, and asking for
one you did not install raises an `ImportError` that names the extra.

## Documentation

One page per module in [`docs/`](docs/), plus [`DESIGN.md`](DESIGN.md) for why it is shaped this
way and [`docs/development.md`](docs/development.md) for the gate.

## Licence

Dual-licensed: **AGPL-3.0-or-later**, or a commercial licence for use in proprietary products and
SaaS deployments where AGPL obligations cannot be met — <info@forktex.com>. See
[`LICENSE`](LICENSE) and [`NOTICE`](NOTICE).

