Metadata-Version: 2.4
Name: dpdp-scrub
Version: 0.0.5
Summary: Detect, validate, and redact Indian PII (Aadhaar, PAN, GSTIN, UPI, and more) — self-hosted, checksum-validated, DPDP-ready.
Author: Palkush Dave
License: MIT
Keywords: pii,dpdp,aadhaar,pan,gstin,privacy,redaction,india
Classifier: Development Status :: 2 - Pre-Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Security
Classifier: Topic :: Text Processing
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: mypy>=1.10; extra == "dev"
Provides-Extra: server
Requires-Dist: fastapi>=0.110; extra == "server"
Requires-Dist: uvicorn>=0.29; extra == "server"
Provides-Extra: redis
Requires-Dist: redis>=5; extra == "redis"
Provides-Extra: ai
Requires-Dist: onnxruntime>=1.17; extra == "ai"
Requires-Dist: transformers>=4.40; extra == "ai"
Requires-Dist: numpy>=1.21; extra == "ai"
Dynamic: license-file

# dpdp-scrub

[![PyPI](https://img.shields.io/pypi/v/dpdp-scrub)](https://pypi.org/project/dpdp-scrub/)
[![CI](https://github.com/ramcharanteja0307/dpdp-scrub/actions/workflows/ci.yml/badge.svg)](https://github.com/ramcharanteja0307/dpdp-scrub/actions)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
[![Python 3.9+](https://img.shields.io/badge/python-3.9%2B-blue)](https://pypi.org/project/dpdp-scrub/)

**Detect, validate, and redact Indian PII before it reaches an LLM — self-hosted, checksum-validated, DPDP-ready.**

Your org keeps the keys; the LLM gets the placeholders.

```
IN  : Sir aadhaar 2345 6789 0124 hai, call karo 9876543210 pe, PAN ABCPE1234F
LLM : Sir aadhaar XXXX XXXX 0124 hai, call karo <PHONE_IN_1> pe, PAN <PAN_1>
BACK: Done! 9876543210 pe call schedule ho gaya.        ← rehydrated, agent-side
```

## Why this exists

Every Indian company wiring LLMs into support, sales, and ops is streaming
Aadhaar numbers, PANs, GSTINs, UPI IDs, and phone numbers to third-party model
providers — and copying them into logs, vector DBs, and fine-tuning sets.
The DPDP Act's substantive obligations become enforceable in **May 2027**
(penalties up to ₹250 crore); UIDAI's Aadhaar-masking rules and RBI's card
rules already apply.

The redaction loop itself is well-established engineering (Presidio, Google
DLP). What didn't exist is **eyes that can see Indian PII** — validated,
context-aware, and open. Google Cloud DLP ships 3 Indian infoTypes, AWS
Comprehend 4, Presidio 6. dpdp-scrub detects **20**, most with mathematical
or reference-data validation:

| Entity | Validation |
|---|---|
| Aadhaar / Aadhaar VID | **Verhoeff checksum** + first-digit rule |
| GSTIN | **mod-36 check character** + embedded-PAN structure + state code |
| Credit/debit card | **Luhn checksum** |
| PAN | holder-type structure rule |
| UPI ID | handle verified against **known PSP list** (`@ybl`, `@oksbi`, …) |
| Driving licence | **RTO state-code** verified + year structure |
| IP address | per-octet range validation |
| API keys / tokens | provider prefixes (`sk-`, `AKIA`, `ghp_`, JWT…) |
| Password | captured only after an explicit `password:` / `pwd=` label |
| UUID | RFC-4122 device / advertising / session IDs |
| IFSC, phone, email, voter ID, passport | structural + context scoring |
| Vehicle registration | state/RTO code verified against reference list (incl. BH series) |
| Bank account, EPF UAN, ABHA | **context-required** (surface only with evidence like "A/c", "UAN") |

Secrets (API keys, passwords) are **redacted**, never tokenized — there is no
reason to ever reverse them.

The context engine is what separates a phone number from an order ID:

```
order 9876543210 ka status batao   →  nothing        (suppressed by "order")
call karo 9876543210 pe            →  PHONE_IN 0.99  (boosted by "call")
mera a/c 34512345678901 hai        →  BANK_ACCOUNT   (earned by "a/c")
code 34512345678901 likha hai      →  nothing
```

## Quick start

```bash
pip install dpdp-scrub
```

```python
from dpdp_scrub import Scrubber

s = Scrubber()

result = s.scrub("mera PAN ABCPE1234F hai, call 9876543210", session_id="ticket-42")
result.text    # "mera PAN <PAN_1> hai, call <PHONE_IN_1>"
result.spans   # typed spans: entity, offsets, confidence

reply = call_your_llm(result.text)          # provider never sees real values
s.rehydrate(reply, "ticket-42")             # real values restored, your side only
```

Detection only:

```python
from dpdp_scrub import detect
detect("payment 9876543210@ybl pe karo")
# [Span(entity='UPI_ID', start=8, end=22, ..., confidence=0.92)]
```

Zero runtime dependencies. Python 3.9+.

### Names and addresses (optional)

Aadhaar has a checksum. "Ravi Sharma, Flat 302, Andheri West" has nothing — no
pattern exists for it, so it needs a model. That pass is opt-in and stays out of
the default install:

```bash
pip install "dpdp-scrub[ai]"       # onnxruntime + tokenizer, no torch
export DPDP_SCRUB_NER_MODEL=/path/to/model
```

```python
from dpdp_scrub import Scrubber, NerTagger

s = Scrubber(ner=NerTagger())
s.ner_status                       # 'onnx' | 'torch' | 'unavailable: <reason>'
s.scrub("mera naam Ravi Sharma hai, ghar Andheri West")
# "mera naam <PERSON_1> hai, ghar <ADDRESS_1>"
```

Three guarantees, each covered by tests:

- **Additive only.** A model span overlapping a rules span is discarded, always.
  A checksum-validated Aadhaar can never be relabelled because a neural net felt
  strongly about it — not even when the model reports 1.00 against the rules
  layer's 0.51.
- **Provenance.** Every span and audit line carries `detected_by: rules | model`,
  so a reviewer can tell arithmetic from inference.
- **Never fails open silently.** No weights, or no `[ai]` extra → rules-only, and
  `ner_status` says why. A model outage cannot masquerade as a clean run.

The model sits behind a cheap lexical gate, so messages that cannot contain a
name ("what is the bulk price for 500 units") skip it entirely. The gate keys on
name/address *cues* — deliberately **not** on whether the rules layer found
anything, because `mera naam Ravi Sharma hai` carries no checksum-bearing
identifier, so gating on rules hits would leak the name.

## Trust model

```
Ring 0: the vault (your Redis/memory)   — real values, session-scoped, droppable
Ring 1: your org's services             — placeholders + scoped right to resolve
Ring 2: LLM providers, logs, vector DBs — placeholders only, no resolution path
```

- Same value → same placeholder within a session (multi-turn consistency).
- `vault.drop(session)` makes every old copy of its placeholders permanently
  unresolvable — retroactive redaction for logs you already wrote.
- No cloud, no telemetry, no phoning home. This is a library, not a service:
  **your data never leaves your infrastructure.**

Per-entity actions are policy: `tokenize` (reversible), `mask` (UIDAI-format
`XXXX XXXX 0124` for Aadhaar, last-4 for cards), `redact` (ABHA health IDs by
default), `ignore`.

## Benchmark

IndicPII-Bench v0 (in `bench/`): 500 synthetic messages — Hinglish, English,
Devanagari, OCR-noise spacing, multi-entity — with 20% traps (order IDs,
invoice refs, phones-in-txn-context). Presidio runs at its best
configuration — its IN_* recognizers explicitly registered (they are shipped
but **not loaded by default**). F1, value-level matching:

| entity | dpdp-scrub | presidio |
|---|---|---|
| AADHAAR | **1.000** | 1.000 |
| GSTIN | **1.000** | 0.971 |
| PAN | **1.000** | 0.383 |
| PHONE_IN | **1.000** | 0.602 |
| UPI_ID | **1.000** | 0.000 |
| IFSC | **1.000** | 0.000 |
| EMAIL | **1.000** | 1.000 |
| **OVERALL** | **1.000** | 0.595 |
| trap false positives | **0/100** | 200/100 |

Honest caveats: this is our own v0 corpus (templated, synthetic, Hinglish-
heavy) — a perfect self-score mainly means the corpus is still too easy, and
it will get harder (Devanagari, OCR noise, multi-entity). Presidio was not
built for code-mixed text; the 107 trap FPs come from it having no context
suppression (e.g. any 10-digit number near "txn" flags as phone). Reproduce:
`python3 bench/compare.py 500`.

## Status

Pre-alpha, moving fast. **199 tests, all synthetic data** — no real PII is
committed anywhere in this repo, including the author's own.

Built: 20 rules entities with checksum validators, the context-scoring engine,
the scrub/rehydrate loop with a session vault, policy profiles + audit log,
Redis backend, LiteLLM/LangChain/Presidio adapters, IndicPII-Bench, and the
optional NER pass for names and addresses.

- [x] Policy profiles (`dpdp-strict`, `uidai-mask`, `audit-only`) + audit log
- [x] Redis vault backend
- [x] LiteLLM / LangChain / Presidio adapters
- [x] **IndicPII-Bench** — a benchmark for Indian PII detection (`bench/`)
- [x] Code-mixed NER for names/addresses — `[ai]` extra, MuRIL fine-tune
- [ ] Publish the NER weights to the Hugging Face Hub
- [ ] `scrub_json()` for structured payloads
- [ ] OpenAI-compatible gateway (one-line `base_url` change, self-hosted)
- [ ] User-defined entity patterns for company-internal ID formats

### 0.0.5

- Five new entities: `API_KEY`, `IP_ADDRESS`, `UUID`, `DRIVING_LICENSE`,
  `PASSWORD`. Live secrets are **redacted**, never tokenized — a partial view of
  a credential is still a credential, and it must not be rehydratable.
- Glued labels now detected for IFSC, GSTIN, passport and voter ID, so
  `ifscICIC0000599` no longer slips through.
- Rehydration is case-insensitive: an LLM that echoes `<pan_1>` back in
  lowercase still resolves.
- New `[ai]` extra: optional PERSON/ADDRESS detection, additive-only, with
  `detected_by` provenance on every span.
- `Span` gained a defaulted `detected_by` field. Existing code is unaffected.

**Known limitations are published, not hidden** — see
[`LIMITATIONS.md`](LIMITATIONS.md) for the register, including what is
deliberately left open and why. The NER model's language scope (Odia
underperforms; Assamese is unmeasured) is documented in
[`ner/README.md`](ner/README.md).

## Contributing

The reference-data files are designed for one-line PRs: new UPI PSP handles,
RTO codes, context words (English/Hindi/regional). Entity requests welcome —
include format, any checksum, and public documentation.

All test data must be synthetic (checksum-generated) or published samples.
Never commit real PII, including your own.

## License

MIT
