Metadata-Version: 2.4
Name: keble-scraper-api
Version: 0.2.0
Summary: Asynchronous policy-bounded scraping API, workers, proxy router, and artifact service.
Author-email: zma <bob0103779@gmail.com>
Requires-Python: <3.14,>=3.13
Requires-Dist: aiohttp<4,>=3.11
Requires-Dist: celery<6,>=5.6
Requires-Dist: cryptography<47,>=45
Requires-Dist: fastapi<1,>=0.116
Requires-Dist: google-auth<3,>=2.40
Requires-Dist: httpx<1,>=0.28
Requires-Dist: keble-data-infra-contract<1,>=0.9.0
Requires-Dist: keble-helpers<2,>=1.54.0
Requires-Dist: keble-scraper-contract==0.3.0
Requires-Dist: nh3<1,>=0.3
Requires-Dist: oss2<3,>=2.19
Requires-Dist: pydantic-settings<3,>=2.10
Requires-Dist: pymongo<5,>=4.12
Requires-Dist: redis<7,>=5
Requires-Dist: sentry-sdk[celery,fastapi]<3,>=2.32
Requires-Dist: uvicorn[standard]<1,>=0.35
Description-Content-Type: text/markdown

# Keble Scraper API

The API package owns FastAPI composition, Mongo lifecycle/index persistence,
Redis-backed egress budgets and sessions, Celery workers/beat, secure HTTP and
Cloudflare fetching, private OSS artifacts, HMAC callbacks, authentication,
auditing, and Sentry instrumentation. It imports `keble-scraper-contract`; the
contract never imports this package.

Version `0.2.0` retains the undiscounted Bright pay-as-you-go baseline for
pre-dispatch cost reservation so temporary coupon pricing cannot understate
production spend and releases typed provider recovery, crash-safe callback and
job budgets, bounded history, artifact-namespace ownership, ordered retention,
and the guarded first-release reset.

The workspace consumes `keble-scraper-contract 0.3.0` as the foundation for
the API `0.2.0` recovery release. Attempt/job/callback
persistence stores discriminated typed failure snapshots instead of terminal
strings, and retry classification uses their finite dispositions. Public HTTP
errors use the same envelope; mandatory retention revisions prevent delayed pin
delivery from reversing a newer paid-storage policy.

Each job and callback row owns its immutable attempt budget;
jobs also own a fixed deadline. Lease acquisition and recovery enforce those
stored values, temporary capacity retains its future horizon, and unexpected
runtime faults remain leased for expiry recovery rather than becoming a false
network failure. Cache pointers cannot outlive referenced artifacts, missing
cache metadata invalidates the pointer before refetch, and expired callback
leases are recovered or dead-lettered by maintenance.

The WP8 clean-break command lives at `scripts/reset_pre_release_state.py` and
delegates to the package-owned typed maintenance module. DRY_RUN is the default.
Local/test-only APPLY deletes referenced binaries before their Mongo metadata,
then the fixed Scraper collection allowlist and exact Redis namespace. It
requires explicit writer-stop, database, Redis, artifact-owner, snapshot,
artifact-inventory, campaign, and immutable-audit evidence; failures propagate
with metadata or RUNNING evidence retained for replay. See `../docs/RUNBOOK.md`.

HTTP, Cloudflare, and registered callback transports own reusable connection
pools closed by `ScraperContainer`. The shared Data Infra Retry-After parser is
used at every response boundary, with durable due times rounded upward to Mongo
millisecond precision. Maintenance projects due job ID plus transport policy in
one indexed read and publishes with configured bounded concurrency.

Maintenance also owns bounded historical retirement. Terminal Jobs without a
callback receive an absolute retention clock immediately; callback Jobs receive
one only in the same transaction that marks their outbox event delivered.
Pending, leased, and dead-letter outbox rows therefore cannot be purged. One
Mongo singleton lease serializes each minute's bounded purge, and the majority
transaction deletes each expired Job with all immutable Attempt children before
reporting success. Delivered callbacks and audit facts use independent absolute
clocks. Defaults are 90 days for Job/Attempt evidence, 30 days for delivered
callback replay, 400 days for security audit, and 1,000 rows per collection per
minute; every value is a typed `SCRAPER_*` environment setting.

Sentry is initialized explicitly for the API, Celery parent, every worker child,
and beat. Runtime tags distinguish those processes while credentials, URLs,
request bodies, and provider payloads stay out of telemetry configuration.

Side effects if changes:
- Shopify must repin and exhaustively map local versus upstream failures;
- frontend error presentation and OpenAPI generation ship with the API release;
- pre-production `0.1.x` scraper job rows require the guarded reset/repair step.

Runtime commands:

```bash
uv run --package keble-scraper-api keble-scraper-api
uv run --package keble-scraper-api keble-scraper-worker fetch
uv run --package keble-scraper-api keble-scraper-worker render
uv run --package keble-scraper-api keble-scraper-worker proxy-health
uv run --package keble-scraper-api keble-scraper-worker maintenance
uv run --package keble-scraper-api keble-scraper-beat
```

Service-authenticated consumers may download a verified artifact through the
audited signed-URL endpoint and synchronize a downstream retention pin with a
required positive `retentionRevision`. The Mongo transaction increments the
artifact reference only on first owner-pin creation, applies only newer policy
revisions, ignores stale deliveries, and rejects a divergent replay at the same
revision. Both finite archive expiry and indefinite latest retention are valid
ordered policies.
The operator and service paths share the same public download route but retain
separate authorization and audit identities.

The download service reserves a server UUID, places it in the immutable audit
event and the signed URL's `x-keble-audit-id` query, and returns the URL only
after the audit append succeeds. OSS `GetObject` logs can therefore reconcile
reads exactly. New OSS objects also carry their canonical SHA-256 as private
user metadata for bounded integrity checks.

For a multi-process local canary without OSS, configure both
`SCRAPER_LOCAL_ARTIFACT_DIRECTORY` and `SCRAPER_LOCAL_ARTIFACT_BASE_URL`, then
serve only that isolated directory on localhost. The development adapter keeps
content-addressed bytes shared between API and Celery workers; staging and
production reject it and continue to require private OSS.

Empty or whitespace-only values for those two optional local-artifact settings
are treated as absent. This supports explicit Compose overrides when private OSS
replaces a legacy local-artifact `env_file` configuration; all backend-completeness
and environment restrictions still apply.

The optional OSS endpoint, bucket, access-key ID, and access-key secret are also
normalized from blank to absent as one typed tuple. Partial or empty production
configuration fails startup before any external request or artifact write.

Startup now applies two independent artifact fences before readiness. Mongo
persists and compares the complete non-secret backend identity: backend kind,
deterministic prefix, OSS endpoint/bucket or resolved local directory, and the
configured owner. The binary store also contains one deterministic owner canary
inside that prefix. Existing rows without a complete marker, backend/location
changes, and a second owner all fail closed before workers accept jobs.

Set `SCRAPER_ARTIFACT_NAMESPACE_OWNER` in staging and production to a stable,
non-secret value unique to the Mongo deployment/database that owns the prefix,
for example `cn-hangzhou/mongo-primary/keble_scraper`. Do not reuse it for a
second database. Development/test may derive `environment/database` locally.
The pre-production prefix-only marker is intentionally incompatible and must be
cleared or explicitly migrated before this release line starts.

When a local VPN returns fake-IP DNS answers, configure both
`SCRAPER_DEVELOPMENT_HTTP_PROXY_URL` and
`SCRAPER_DEVELOPMENT_DNS_OVER_HTTPS_URL`. The worker validates real DoH A/AAAA
answers through the explicit proxy, preserves IP pinning, and applies the proxy
only to the verified `DIRECT` route. The pair is rejected outside
development/test.

Every route admission is atomically assigned a future Redis timestamp derived
from its requests-per-second policy. A fetch worker asynchronously waits for
that timestamp, preserving the requested route and job attempt; internal burst
pressure therefore does not become terminal `ROUTE_UNAVAILABLE` evidence or a
paid fallback.

The repository release workflow builds this package's runtime image only from
an annotated `keble-scraper-api-vX.Y.Z` tag whose version matches this package.
GHCR version/source tags are discovery aids; production consumes the digest
evidence artifact and pins `ghcr.io/keble-ai/keble-scraper-api@sha256:...`.
