Metadata-Version: 2.5
Name: opencraw
Version: 0.3.1
Summary: Crawl with OpenCraw from Python: the engine through its driver, downloaded once with no Node to install, and hooks written in Python.
Project-URL: Homepage, https://github.com/Open-Craw/OpenCraw
Project-URL: Repository, https://github.com/Open-Craw/OpenCraw
Project-URL: Issues, https://github.com/Open-Craw/OpenCraw/issues
License-Expression: MIT
Keywords: crawler,opencraw,recipes,scraping
Requires-Python: >=3.12
Description-Content-Type: text/markdown

# opencraw (Python)

Crawl with [OpenCraw](https://github.com/Open-Craw/OpenCraw) from Python, with no Node to install: `Crawler`
downloads the driver's runtime for this platform once (a Node binary and the engine in one archive, checked
against its SHA-256 and cached in `~/.cache/opencraw/driver/`), starts it, and talks to it over its pipes.
Standard library only; Python 3.12+.

```python
from opencraw import Crawler

with Crawler(version="0.6.0") as crawler:

    @crawler.hook()
    def tidy(value, args, request):  # a hook the recipes call: plain Python
        return str(value).strip().lower()

    run = crawler.run(paths=["recipes/"])
    for record in run.records():  # as they are emitted
        print(record)
    print(run.report()["records"])
```

- **`crawler.run(...)`** takes the driver's run options (`out`, `only`, `dryRun`, `parallel`, `access`, ...);
  `run.records()` and `run.events()` yield as it crawls, `run.report()` waits for the end, `run.cancel()` stops it.
- **`crawler.work(next_item, ...)`** is worker mode: `next_item()` returns each window's item
  (`{"id": ..., "vars": {...}}`), or `None` to end.
- **`@crawler.hook()`, `@crawler.captcha_solver()`, `@crawler.access_plugin()`** register Python functions by
  name; raising fails that one call (`CalloutError` for a plain message), never the run.
- **`crawler.validate`, `probe`, `diff`, `export`** answer as the driver does.
- **`AsyncCrawler`** is the asyncio twin: `async with AsyncCrawler(version=...) as crawler`, `await crawler.run(...)`,
  `async for record in run.records()`, and hooks, plugins and `next_item` may be `async def`.
- `OPENCRAW_DRIVER=<folder>` uses an unpacked runtime; `OPENCRAW_DRIVER_OFFLINE=1` never downloads; `command=`
  starts any other driver (`["node", "path/to/main.esm.js"]`).

The rest of this page is the older way, which stays: build recipes in Python and run the `opencraw` CLI once
per call (`npm i -g @opencraw/cli`, or `OPENCRAW_CLI` set to the command that starts it; with neither,
`npx @opencraw/cli` is used).

A recipe stays a plain dict in the engine's own shape, so the recipe guide
([docs/recipes/authoring.md](https://github.com/Open-Craw/OpenCraw/blob/main/docs/recipes/authoring.md)) is the reference for every key. The constructors only save typing, and
the engine, not this package, says what is valid.

```python
from opencraw import input_recipe, output_recipe, run, step, validate

product = output_recipe(
    "product",
    {
        "url": {"type": "url", "required": True, "key": True},
        "title": {"type": "string", "required": True},
    },
)

shops = [
    input_recipe(
        f"shop-{name}",
        "product",
        "web",
        [url],
        [
            step("goto", url="{{start.url}}"),
            step("extract", id="title", selector=selector, kind="css", take="text"),
            step("emit"),
        ],
        {"url": {"from": "start.url"}, "title": {"from": "title"}},
    )
    for name, url, selector in [
        ("a", "https://a.example", "h1"),
        ("b", "https://b.example", ".name"),
    ]
]

result = validate(
    [product, *shops]
)  # Validation(ok, output): problems are data, not exceptions
records = run(
    [product, *shops], "--only", "shop-a"
)  # records as dicts; OpenCrawError if the crawl fails
```

Reserved words: pass a step's `as` or `from` through `**{"as": "json"}`.

`write_recipes(folder, recipes)` writes `<id>.input.json` and `<id>.output.json` for the CLI, the Studio
and the MCP server.

## A hook written in Python

A recipe's hook (`{ "op": "hook", "name": "slug" }`) can be a Python script. The hooks module registers it
with `commandHook` (`@opencraw/core`), and the script is a few lines:

```python
# hooks/slug.py
import re
from opencraw import serve_hook

serve_hook(
    lambda value, args, request: re.sub(r"[^a-z0-9]+", "-", str(value).lower()).strip(
        "-"
    )
)
```

`serve_hook(handler)` reads the request from stdin, calls `handler(input, args, request)` (`input` is `None`
for a `hook` step; `request` carries `context` and `idempotencyKey`) and writes the answer to stdout. What the
handler returns is the output; an exception becomes an error answer with its message, so the step fails with
that reason. A handler whose work takes longer raises `HookPending(retry_after_ms)`: OpenCraw runs the script
again after that wait, with the same `idempotencyKey`, until it returns or the hook's `maxWaitMs` passes, so the
script starts the work once and looks it up by that key on later runs. The request and answer shapes are in
`packages/core/schemas/callout-*.schema.json`.

Not in this package: a Python engine and generated typed models. The crawl itself always runs in the
OpenCraw CLI. It is published to PyPI as `opencraw`: `mnci ci release` versions and publishes what sits
under `python-packages/`.

## Development

`smith run test -p opencraw` (pytest; the CLI integration tests use this repository's built CLI and skip
without it), `smith run lint -p opencraw` (Ruff), `smith run typecheck -p opencraw` (mypy strict).

**Layout.** One slice package per outcome: `recipe_building/` (recipes in Python, written to files),
`hook_serving/` (hooks served as command callouts), `cli_bridge/` (the CLI, run once per call),
`driver_runtime/` (the runtime's download and cache), `driver_channel/` (JSON-RPC over the driver's pipes) and
`crawl_session/` (`Crawler` and `Run`). Its tests replay the driver's conformance transcripts against the
repository's built driver.
