Metadata-Version: 2.4
Name: hadoku-scrape-sdk
Version: 0.1.5
Summary: The runtime hadoku-scrape plugins build on: plugin contract, browser stack, run log, events, test floor
License-Expression: ISC
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: browser-cookie3<1.0,>=0.19.0
Requires-Dist: fastapi<1.0,>=0.109.0
Requires-Dist: hadoku-execlog>=0.1.0
Requires-Dist: httpx[socks]<1.0,>=0.27.0
Requires-Dist: patchright<2.0,>=1.49.0
Requires-Dist: playwright<2.0,>=1.62.0
Requires-Dist: pydantic<3.0,>=2.0
Requires-Dist: pyvirtualdisplay<4.0,>=3.0
Requires-Dist: structlog<25.0,>=24.0.0
Requires-Dist: tenacity<9.0,>=8.2.0
Provides-Extra: testing
Requires-Dist: pytest==8.4.2; extra == "testing"
Requires-Dist: pytest-asyncio<2,>=1.2; extra == "testing"
Requires-Dist: pytest-timeout<3.0,>=2.2.0; extra == "testing"
Requires-Dist: respx<1.0,>=0.21.0; extra == "testing"
Requires-Dist: httpx2<3.0,>=2.9; extra == "testing"
Requires-Dist: typer<1.0,>=0.15.0; extra == "testing"
Dynamic: license-file

# hadoku-scrape-sdk

The runtime that hadoku-scrape plugins are built on. A plugin is a scraper that
lives in its own repo and runs inside the hadoku-scrape service. This package is
everything such a plugin imports, and nothing of the service around it.

```sh
pip install hadoku-scrape-sdk            # runtime
pip install 'hadoku-scrape-sdk[testing]' # + the pinned test stack and test floor
```

## What's in it

| Module | What it gives a plugin |
|---|---|
| `hadoku_sdk.plugins` | `ScraperPlugin`, `UrlScraper`: the contract. `python -m hadoku_sdk.plugins` checks every enabled plugin loads |
| `hadoku_sdk.db.engine`, `.db.execution_log` | The SQLite engine (`init_db`, `get_connection`) and `execution_run`, the run log every scrape writes through |
| `hadoku_sdk.runtime` | `Severity` and events (`ctx.event(...)`), run pacing, the run gate |
| `hadoku_sdk.browser` | Browser manager (Patchright, persistent profiles, stealth), sessions, preflight checks |
| `hadoku_sdk.proxy` | HTTP client with proxy escalation |
| `hadoku_sdk.auth.cookies` | Browser-cookie import |
| `hadoku_sdk.api.middleware` | Auth dependencies (`require_service`, ...), error types, `spawn_tracked` for fire-and-forget routes |
| `hadoku_sdk.scrapers.base` | `BaseScraper`, `ScraperResult` for URL-dispatch scrapers |
| `hadoku_sdk.observability.logging` | `get_logger` (structlog) |
| `hadoku_sdk.testing` | The autouse test floor: an isolated DB per test, no real browsers |

## A plugin

```python
# my_plugins/thing/plugin.py
from pathlib import Path
from hadoku_sdk.plugins import ScraperPlugin

PLUGIN = ScraperPlugin(
    name="thing",
    routers=[router],                 # FastAPI, prefix /api/v1/<yours>
    cli={"thing": thing_app},         # Typer, mounted as `hs thing`
    migrations=Path(__file__).parent / "migrations",   # NNN_*.sql
    on_startup=[seed],
)
```

```toml
# pyproject.toml
[project.entry-points."hadoku_scrape.plugins"]
thing = "my_plugins.thing.plugin:PLUGIN"
```

The host's `plugins.toml` decides which installed plugins actually load.

## Tests

```python
# tests/conftest.py
import os
from pathlib import Path
os.environ["HADOKU_PLUGINS_MANIFEST"] = str(Path(__file__).parent / "resources" / "plugins.toml")
pytest_plugins = ["hadoku_sdk.testing.pytest_plugin"]
```

```python
# tests/unit/conftest.py
from hadoku_sdk.testing.unit import _instant_sleep, _silence_discord_notify  # noqa: F401
```

## Rules a plugin follows

- Notifications go through `ctx.event(name, payload, severity=...)` on the run
  context, never a direct Discord call. The host routes them.
- Timestamps are UTC. Never `datetime.utcnow()`.
- Never log credentials, cookies or API keys.
- No lazy imports inside functions that run on background threads.
- Drive browsers through `hadoku_sdk.browser.manager`. Never hardcode locale or
  timezone.

License: ISC.
