Metadata-Version: 2.5
Name: rahasya
Version: 0.0.2
Summary: thirty-four kinds of personal identifier, found by arithmetic: checksummed patterns read in context, no model and no network
Project-URL: Repository, https://github.com/vedicreader/rahasya
Project-URL: Documentation, https://vedicreader.github.io/rahasya/
Author-email: Karthik <karthik.rajgopal@hotmail.com>
License: Apache-2.0
License-File: LICENSE
Keywords: checksum,dlp,gdpr,nbdev,ner,pii,privacy,redaction
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Security
Classifier: Topic :: Text Processing
Requires-Python: >=3.12
Requires-Dist: fastcore>=2.2.15
Description-Content-Type: text/markdown

# rahasya


<!-- WARNING: THIS FILE WAS AUTOGENERATED! DO NOT EDIT! -->

Thirty-four kinds of personal identifier, each a pattern with a checksum, read in context.
1.000 precision at recall 1.000 on a held-out corpus, 0.996 on the one they were tuned against. No model, no network, no gigabyte on disk.

## Install

``` sh
uv add rahasya
```

## The one call

``` python
r = pii_report('Invoice for ada@example.com. Card 4111 1111 1111 1111. '
                'KEY=sk-abcdefghijklmnopqrstuvwxyz123456')
r.has_pii, r.identifying
```

`has_pii` counts only `IDENTIFYING` kinds. An IP address is reported and does not tip the
gate.

``` python
redact('Card 4111 1111 1111 1111 charged to ada@example.com')
```

## Digits are read in context

A number failing its checksum is not the thing the checksum protects. A number introduced by a
reference word is not an identifier.

``` python
[pii_report(t).has_pii for t in
 ('Order 4111 1111 1111 1112 shipped',   # fails Luhn
  'Conforms to EN 60601-1',              # a standard, not a ZIP
  'Card 4111 1111 1111 1111')]           # this one is a card
```

`DESIGNATOR` holds the words that do it: `ISO`, `RFC`, `invoice`, `page`, `commit`, `order`
and thirty more. It is matched against the 48 characters to the left of a candidate.

## Nineteen regional identifiers, each with its own checksum

| where | kinds |
|----|----|
| India | Aadhaar, PAN, GSTIN |
| Australia | TFN, ABN, Medicare |
| Europe | Dutch BSN, French NIR, Spanish DNI and NIE, Italian codice fiscale, Polish PESEL, Swedish personnummer, Norwegian fødselsnummer, German Steuer-IdNr, UK NINO |
| Asia-Pacific | Singapore NRIC, Thai national ID |
| anywhere | IMEI |

``` python
import random
a = gen_aadhaar(random.Random(0))            # a valid one, for a fixture
a, aadhaar_ok(a), pesel_ok('44051401359'), nino_ok('AB123456C')
```

## Names are opt-in

`ner=True` turns on an honorific-anchored pass. `Dr Charles Babbage` matches; a bare
`Ada Lovelace` does not. `scanned_ner` says whether anything looked, which keeps “none found”
apart from “not looked for”.

``` python
(pii_report('Dr Charles Babbage signed it.').has_pii,
 pii_report('Dr Charles Babbage signed it.', ner=True).identifying,
 redact('Dr Charles Babbage signed it.', ner=True))
```

## What is here

| name | what it does |
|----|----|
| [`pii_report`](https://vedicreader.github.io/rahasya/core.html#pii_report) | spans found, and whether they tip `has_pii` |
| [`pii_spans`](https://vedicreader.github.io/rahasya/core.html#pii_spans) | `(start, end, kind, value)`, de-overlapped longest-first |
| [`redact`](https://vedicreader.github.io/rahasya/core.html#redact) | mask matched spans; `[EMAIL]`, `[CARD]`, or a mask you name |
| [`redact_obj`](https://vedicreader.github.io/rahasya/core.html#redact_obj) | [`redact`](https://vedicreader.github.io/rahasya/core.html#redact) over the strings inside a nested dict or list |
| [`person_names`](https://vedicreader.github.io/rahasya/core.html#person_names) / [`person_spans`](https://vedicreader.github.io/rahasya/core.html#person_spans) | the honorific-anchored name pass |
| `PATTERNS` | kind to `(pattern, validator)`; the whole detector as data |
| `IDENTIFYING` | which kinds tip the gate |

[`luhn`](https://vedicreader.github.io/rahasya/core.html#luhn), [`aadhaar_ok`](https://vedicreader.github.io/rahasya/core.html#aadhaar_ok), [`pesel_ok`](https://vedicreader.github.io/rahasya/core.html#pesel_ok) and the rest are exported too, because a checksum is useful on
its own.

## No model

Two learned detectors were measured against these patterns. Both lose on precision and recall, and
are 200 times slower. A DeBERTa-v3 ONNX classifier earns its gigabyte
only on names no honorific introduces (2/8 to 5/8 of them). A 350M tflite encoder finds no names
at all.

## Development

The notebooks in `nbs/` are the source; the modules are generated.

``` sh
pip install -e .
nbdev_prepare
```
