Metadata-Version: 2.4
Name: botshield
Version: 0.1.2
Summary: BotShield -- layered bot & fraud request-filtering pipeline (signals -> Tier 1 rules -> Tier 2 scoring -> logging -> optional ML)
License-Expression: Apache-2.0
License-File: LICENSE
Requires-Python: >=3.11
Requires-Dist: pyyaml>=6.0
Provides-Extra: agent
Requires-Dist: openai>=1.50; extra == 'agent'
Requires-Dist: playwright>=1.47; extra == 'agent'
Requires-Dist: python-dotenv>=1.0; extra == 'agent'
Requires-Dist: requests>=2.31; extra == 'agent'
Provides-Extra: dashboard
Requires-Dist: pandas>=2.2; extra == 'dashboard'
Requires-Dist: streamlit>=1.35; extra == 'dashboard'
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == 'dev'
Provides-Extra: django
Requires-Dist: django>=4.2; extra == 'django'
Provides-Extra: flask
Requires-Dist: flask>=3.0; extra == 'flask'
Provides-Extra: ml
Requires-Dist: joblib>=1.4; extra == 'ml'
Requires-Dist: pandas>=2.2; extra == 'ml'
Requires-Dist: scikit-learn>=1.4; extra == 'ml'
Provides-Extra: mongodb
Requires-Dist: pymongo>=4.6; extra == 'mongodb'
Requires-Dist: python-dotenv>=1.0; extra == 'mongodb'
Provides-Extra: redis
Requires-Dist: redis>=5.0; extra == 'redis'
Provides-Extra: web
Requires-Dist: fastapi>=0.111; extra == 'web'
Requires-Dist: requests>=2.31; extra == 'web'
Requires-Dist: uvicorn[standard]>=0.30; extra == 'web'
Description-Content-Type: text/markdown

# BotShield

A layered bot/fraud request-filtering pipeline:

```
request -> SDK middleware (extracts signals)
         -> Tier 1: fast rules engine (in-process, sub-millisecond)
         -> Tier 2: risk scoring (weighted signals, later ML)
         -> action: allow / challenge / block
         -> log outcome (feeds your own review/labeling pipeline)
```

Tier 1 hard-blocks known-bad traffic (documented AI crawlers, missing UA on
sensitive paths, spoofed-browser headers, burst traffic) in a handful of
deterministic checks. Everything that survives falls through to Tier 2,
which scores it with hand-tuned weights across ~15 signals. Every decision
gets logged so you can review and tune thresholds against real traffic.

## What's in this package

```
botshield/
  types.py                  RequestData / Decision dataclasses
  config.py                 BotShieldConfig + insecure-default-secret fail-fast
  known_lists.py            curated known-bad UA patterns + KNOWN_BAD_TLS_FINGERPRINTS
  signals.py                extract_signals(request) -> dict (browser-family aware)
  client_ip.py              trusted-proxy-aware real client IP + TLS fingerprint
  history.py                in-memory per-IP rate tracking (LRU-bounded)
  history_redis.py          opt-in Redis rate backend (`redis` extra)
  rules.py + rules.yaml     Tier 1 config-driven rules engine
  scoring.py                Tier 2 hand-weighted scoring
  db.py                     SQLite event log (async, non-blocking writer)
  db_mongodb.py             opt-in MongoDB event-log backend (`mongodb` extra)
  protocols.py              HistoryBackend / EventLogBackend interfaces
  session.py                per-visitor botshield_sid correlation cookie
  challenge.py              challenge gate: SHA-256 PoW + fingerprint check
  fingerprint.py             server-side validation of the client fingerprint bundle
  core.py                   orchestration: BotShield.evaluate()
  middleware_{fastapi,flask,django}.py   framework adapters
  routes_{fastapi,django}.py             POST /botshield/verify endpoints
```

## Install

```
pip install botshield                 # core (pyyaml only)
pip install "botshield[web]"          # + FastAPI adapter deps
pip install "botshield[flask]"        # + Flask adapter deps
pip install "botshield[django]"       # + Django adapter deps
pip install "botshield[redis]"        # + Redis rate-tracking backend
pip install "botshield[mongodb]"      # + MongoDB event-log backend
```

Minimal usage (FastAPI):

```python
from botshield import BotShield, BotShieldConfig, EventLog
from botshield.middleware_fastapi import BotShieldMiddleware
from botshield.routes_fastapi import router as botshield_router

shield = BotShield(
    event_log=EventLog(),
    config=BotShieldConfig(trusted_proxies=frozenset({"10.0.0.1"})),
)
app.add_middleware(BotShieldMiddleware, shield=shield)
app.include_router(botshield_router)   # the /botshield/verify endpoint
```

Set `BOTSHIELD_CHALLENGE_SECRET` before deploying; with
`BotShieldConfig(production_mode=True)` the shield refuses to start on the
insecure default. Runtime data (the SQLite event log) goes to `./data` by
default -- override with `BOTSHIELD_DATA_DIR` or `EventLog(db_path=...)`.

## The challenge gate

When a request scores into the challenge band, `BotShieldMiddleware` serves
a small HTML+JS page instead of the real response. That page does two
things before the client is let through:

1. **Proof-of-work** (`botshield/challenge.py`): the browser must find an
   answer whose SHA-256 hash has N leading zeros, computed via the real Web
   Crypto API. This is genuine work -- but *any* language can compute
   SHA-256, so PoW alone does **not** prove the client is a browser. It
   raises attacker cost slightly; that's all.

2. **Fingerprint validation** (`botshield/fingerprint.py`): the verify
   endpoint *also* requires a plausible client-side fingerprint bundle
   (canvas hash, WebGL, `navigator` details) submitted alongside the solved
   PoW. This is the layer that actually stops a plain script: a
   `curl`/`requests` client has no canvas or WebGL context to report and
   can't fabricate a convincing one without implementing a browser engine.

Only when **both** pass does the client get a signed, time-limited clearance
cookie and get to reload the original URL. This is the same category of
defense real vendors (e.g. Cloudflare's JS challenge) use -- a cost-raising
heuristic that filters non-browser automation, not a cryptographic proof of
humanity. A determined attacker running a real headless browser can pass
it; that's expected, and where Tier 2 behavioral scoring takes over.

Any FastAPI app using `BotShieldMiddleware` needs
`app.include_router(botshield.routes_fastapi.router)` for the verify
endpoint to exist.

### Privacy note: the session cookie

The middleware issues an opaque `botshield_sid` cookie to **every** visitor
on first contact (not only challenged ones), so Tier 2 scoring can correlate
a session's fingerprint history even on a first "allow." It's httponly,
samesite=lax, and holds no personal data -- but it is a persistent
identifier set for 100% of traffic, so a real deployment very likely needs
to disclose it in a cookie-consent banner under GDPR/ePrivacy.

## Tuning

- **Tier 1 rules** (`botshield/rules.yaml`): edit the YAML, no redeploy of
  code needed. Underlying curated bad-UA data lives in
  `botshield/known_lists.py`.
- **Tier 2 weights** (`botshield/scoring.py`): `DEFAULT_WEIGHTS` is a plain
  dict -- retune it against your own traffic before reaching for ML.
  Thresholds (`allow_threshold` / `block_threshold`) live on
  `ScoringConfig`.

## Known limitations

- `RequestHistory` (rate tracking) is in-memory and per-process -- fine for
  a single worker, not for multiple processes or machines. Swap it for a
  Redis-backed sliding-window counter with the same `.stats()` shape
  (`history_redis.py`).
- `EventLog` uses SQLite by default -- fine for a demo or small deployment;
  swap in the MongoDB backend (`db_mongodb.py`) or move to Postgres/
  ClickHouse at real volume.
- There's no IP reputation / datacenter-ASN / TLS-JA3 signal wired in yet --
  `RequestData.tls_fingerprint` is a hook for a reverse proxy to populate
  (e.g. an `X-JA3` header from nginx/Cloudflare).
- The `block_known_ai_training_crawlers` rule blocks by User-Agent string
  alone, which is trivially spoofable -- it stops honest crawlers that
  respect their own self-identification, not a determined attacker. Layer
  in Tier 2 behavioral scoring and IP reputation for anything adversarial.
- The challenge gate is only wired into `middleware_fastapi.py`.
  `middleware_flask.py` and `middleware_django.py` currently let
  `challenge` verdicts through untouched -- port the same clearance-cookie
  check into those if you deploy on Flask/Django.

## License

Apache-2.0
