Metadata-Version: 2.5
Name: euredact
Version: 0.6.0
Summary: European PII redaction SDK — rule engine for 31 countries
Project-URL: Homepage, https://euredact.dev
Project-URL: Documentation, https://euredact.dev/docs
Project-URL: Repository, https://github.com/euredact/euredact
Project-URL: Changelog, https://github.com/euredact/euredact/blob/main/euredact-python/CHANGELOG.md
Project-URL: Issues, https://github.com/euredact/euredact/issues
Author: JNJS
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: anonymization,europe,gdpr,masking,pii,privacy,redaction,referential-integrity
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Provides-Extra: cloud
Requires-Dist: httpx>=0.27; extra == 'cloud'
Provides-Extra: coref
Provides-Extra: dev
Requires-Dist: google-re2>=1.1; extra == 'dev'
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pyahocorasick>=2.0.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
Requires-Dist: pytest-benchmark; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Provides-Extra: fast
Requires-Dist: google-re2>=1.1; extra == 'fast'
Requires-Dist: pyahocorasick>=2.0.0; extra == 'fast'
Description-Content-Type: text/markdown

# EuRedact

**European PII redaction SDK -- rule engine**

---

EuRedact is a pure-Python SDK for detecting and redacting personally identifiable
information (PII) in European text data. It covers 31 countries with a two-pass
architecture: liberal pattern matching in the first pass, followed by suppression
filters and checksum validation in the second. The library has zero required
dependencies, is thread-safe, and produces immutable detection objects.

## Quick Start

```bash
pip install euredact
```

```python
import euredact

result = euredact.redact("Mijn BSN is 111222333 en IBAN NL91ABNA0417164300.")
print(result.redacted_text)
# "Mijn BSN is [NATIONAL_ID] en IBAN [BANK_ACCOUNT]."

print(result.detections)
# [Detection(entity_type=<EntityType.NATIONAL_ID>, ...), Detection(entity_type=<EntityType.BANK_ACCOUNT>, ...)]
```

## Features

- **31 European countries** (see list below)
- **41 PII entity types** (30 from the rules engine, 11 from the
  [cloud tier](#cloud-tier)): national IDs, IBANs, phone numbers, email addresses,
  VAT numbers, license plates, VIN, credit cards, BIC/SWIFT, IMEI, GPS
  coordinates, UUIDs, social handles, MAC addresses, IP/IPv6 addresses, health
  insurance numbers, passport numbers, driver's licenses, secrets/API keys, and more
- **Secret/API key detection:** known-prefix patterns for AWS, GitHub, Stripe,
  OpenAI, Slack, JWT, SendGrid, plus Shannon entropy-based detection for generic
  high-entropy tokens near context keywords
- **Custom patterns:** register your own regex patterns for domain-specific PII
  types at runtime via `add_custom_pattern()`
- **Reversible tokenization:** `tokenize=True` swaps values for `EMAIL_K7Q2`-style
  tokens and `restore()` puts them back — for prompts that go to an LLM and
  come back
- **Allowlist:** values that are never redacted, such as your own organisation's
  name and addresses, per call or per instance
- **Checksum validation:** IBAN mod-97, Luhn (credit cards), and 30+ country-specific
  national ID checksums (e.g., Dutch BSN 11-proof, Belgian national number modulo)
- **Priority-aware deduplication:** when matches overlap, validated patterns
  (with passing checksums, corroborated by the document's country) win over
  custom patterns, which win over regex-only patterns; a failed checksum demotes
  same-type matches rather than deleting them, so it can never silence a
  detection
- **Two-pass detection:** liberal regex matching followed by suppression filters
  that eliminate false positives
- **Context-aware:** keyword proximity checks and structural detection (JSON field
  names, CSV headers) for ambiguous patterns like dates of birth
- **Country self-detection:** infers a document's countries from the entities
  that carry one, so an ambiguous value resolves without the caller naming a
  country — and `countries=` never gates what is looked for
- **Fast:** ~1.1 ms for a short record, ~9 ms for a 3,500-character document
  with `[fast]` installed — though cost tracks identifier density more than
  length (see [Performance](#performance))
- **Zero required dependencies** (`pip install euredact[fast]` adds optional acceleration)
- **Thread-safe,** immutable `Detection` objects (frozen dataclasses)

### Supported Countries

| Region | Countries |
|---|---|
| Western Europe | AT, BE, CH, DE, FR, LU, NL |
| Southern Europe | CY, EL, ES, IT, MT, PT |
| Northern Europe | DK, EE, FI, IS, LT, LV, NO, SE |
| Eastern Europe | BG, CZ, HR, HU, PL, RO, SI, SK |
| British Isles | IE, UK |

## API Reference

### Which option do I need?

| I want to… | Use |
|---|---|
| redact a document with no further setup | `redact(text)` — all 31 countries, `[ENTITY_TYPE]` output |
| restrict scope to the countries I operate in | `countries=["NL", "BE"]` |
| keep detection wide but resolve ambiguity | `country_hint=["DE"]` |
| send a prompt to an LLM and restore the reply | `tokenize=True`, then [`restore()`](#euredactrestore) |
| keep relationships visible across a whole session | `referential_integrity=True` |
| never redact my own company name or addresses | `allowlist=[...]`, or `EuRedact(allowlist=[...])` |
| never redact anything at my own domain | `allowlist_domains=["acme.be"]` |
| audit what the allowlist kept in the document | `result.exempted` |
| catch person names, employers, job titles, diagnoses | `mode="cloud"` — see [Cloud tier](#cloud-tier) |
| include dates of birth | `detect_dates=True` |
| redact a document too large for one call | `context=` + `chunk_offset=` — see [Chunked documents](#chunked-documents) |
| detect an identifier the engine does not know | [`add_custom_pattern()`](#euredactadd_custom_pattern) |
| isolate tenants from each other | one `EuRedact()` instance each — see [Instance Isolation](#instance-isolation) |
| free PII held in memory | `clear()` |


EuRedact provides both module-level functions (using a shared singleton) and an
instance-based `EuRedact` class. The module-level API is the easiest way to get
started; the class-based API gives you isolated instances with separate caches
and custom pattern registrations.

### Module-Level Functions

#### `euredact.redact()`

```python
euredact.redact(
    text: str,
    *,
    countries: list[str] | None = None,
    country_hint: list[str] | None = None,
    context: DocumentContext | None = None,
    chunk_offset: int = 0,
    mode: str = "rules",
    referential_integrity: bool = False,
    tokenize: bool = False,
    allowlist: list[str] | None = None,
    detect_dates: bool = False,
    coref: bool = False,
    coref_model: str = "default",
    cache: bool = True,
) -> RedactResult
```

Main entry point. Detects and redacts PII in the given text. Every option is
keyword-only.

**What is looked for**

| Parameter | Default | Description |
|---|---|---|
| `text` | — | Input text to scan. Longer than `max_input_length` raises `ValueError`. |
| `countries` | `None` | ISO 3166-1 alpha-2 codes that define **scope** (e.g. `["NL", "BE"]`). `None` loads all 31. This never gates what is looked for: a detection attributed elsewhere is flagged `out_of_scope`, never dropped. Passing a bare string raises `TypeError`. See [Country codes](#country-codes). |
| `country_hint` | `None` | A **prior only**. Helps resolve an ambiguous value without narrowing scope or flagging anything out of scope. See [Country Hints](#country-hints). |
| `detect_dates` | `False` | Include `DOB` and `DATE_OF_DEATH`. Off by default: a bare date without keyword or structural context is deferred to the cloud tier. When on, the engine applies keyword and JSON/CSV-header checks. |

**How the output looks**

The three output styles are mutually exclusive in practice — pick at most one.

| Parameter | Default | Description |
|---|---|---|
| *(none)* | — | Default: each span becomes `[ENTITY_TYPE]`. |
| `referential_integrity` | `False` | Replace each distinct value with a consistent label (`EMAIL_1`). Labels persist **on the instance** across calls, so two documents sharing a value get the same label. See [Referential Integrity](#referential-integrity). |
| `tokenize` | `False` | Replace each value with a reversible token (`EMAIL_K7Q2`) and return the token → value mapping in `result.tokens`; pass it to [`restore()`](#euredactrestore). Tokens are unique to the **call**, so nothing is retained. Cannot be combined with `referential_integrity`. See [Reversible tokenization](#reversible-tokenization). |
| `allowlist` | `None` | Values never to redact, matched whole and case-insensitively, merged with the instance's list. Structured identifiers (IBAN, phone, VAT, …) also match across spacing and hyphenation. Applies to cloud-tier types too. See [Allowlist](#allowlist). |
| `allowlist_domains` | `None` | Domains whose addresses are never redacted, e.g. `["acme.be"]`. Applies to `EMAIL` and `URL` only, and covers subdomains. See [Allowlist](#allowlist). |

**Long documents and tiers**

| Parameter | Default | Description |
|---|---|---|
| `mode` | `"rules"` | `"rules"` runs locally. `"cloud"` masks locally first, then sends the masked text to the euRedact service, which adds the model-only types (person names, organisations, job titles, diagnoses). See [Cloud tier](#cloud-tier). |
| `context` | `None` | Share country evidence across the chunks of one document, so a chunk with no country signal of its own is still scored against the rest. Pass the same [`DocumentContext`](#documentcontext) to every chunk. Disables the cache. See [Chunked documents](#chunked-documents). |
| `chunk_offset` | `0` | Where this chunk starts in the whole document. Used only to rebase spans recorded in `context`; returned detections are always relative to `text`. |
| `cache` | `True` | Reuse the result for an identical input and configuration. Set `False` for one-off calls on sensitive text, or when timing the engine. |

**Not yet implemented**

| Parameter | Default | Description |
|---|---|---|
| `coref` | `False` | Reserved for pronoun/coreference resolution. Accepted and ignored in rules mode; raises in cloud mode. |
| `coref_model` | `"default"` | Reserved, with `coref`. |

#### `euredact.redact_batch()`

```python
euredact.redact_batch(
    texts: list[str],
    *,
    countries: list[str] | None = None,
    mode: str = "rules",
    referential_integrity: bool = False,
    tokenize: bool = False,
    allowlist: list[str] | None = None,
    detect_dates: bool = False,
    cache: bool = True,
) -> list[RedactResult]
```

Redact PII from multiple texts at once. More efficient than calling `redact()` in
a loop because country configs are loaded once. Returns results in the same order
as the input.

#### `euredact.aredact()`

```python
async euredact.aredact(
    text: str,
    **kwargs,
) -> RedactResult
```

Async version of `redact()`. Offloads CPU-bound work to a thread pool so it
doesn't block the event loop. Accepts the same keyword arguments.

#### `euredact.aredact_batch()`

```python
async euredact.aredact_batch(
    texts: list[str],
    *,
    max_concurrency: int = 4,
    **kwargs,
) -> list[RedactResult]
```

Async batch redaction with controlled concurrency. Processes texts concurrently in
a thread pool. `max_concurrency` limits parallel threads (default 4). Returns
results in input order.

#### `euredact.redact_iter()`

```python
euredact.redact_iter(
    texts: Iterator[str],
    **kwargs,
) -> Iterator[RedactResult]
```

Lazy iterator that yields results one at a time. Useful for processing large
datasets without loading all results into memory. Loads country configs once on
the first item.

#### `euredact.add_custom_pattern()`

```python
euredact.add_custom_pattern(name: str, pattern: str) -> None
```

Register a custom regex pattern. Matches are reported with `name` as the entity
type. See [Custom Patterns](#custom-patterns) below for details and examples.

#### `euredact.restore()`

```python
euredact.restore(text: str, tokens: Mapping[str, str]) -> str
```

Put the original values back into text that derives from a `tokenize=True`
result — typically an LLM's reply to the tokenized prompt. `tokens` is
`result.tokens`. See [Reversible tokenization](#reversible-tokenization).

#### `euredact.available_countries()`

```python
euredact.available_countries() -> list[str]
```

Returns a sorted list of supported ISO country codes (e.g. `["AT", "BE", "BG", ...]`).

### Instance-Based API (`EuRedact` Class)

For applications that need isolated instances (separate caches, separate custom
patterns), use the `EuRedact` class directly:

```python
from euredact import EuRedact

instance = EuRedact()

# Register custom patterns on this instance only
instance.add_custom_pattern("CASE_REF", r"CASE-\d{8}")

# Redact using this instance's configuration
result = instance.redact("See CASE-20260401 for details", countries=["NL", "BE"])
print(result.redacted_text)
# "See [CASE_REF] for details"
```

The `EuRedact` class exposes the same methods as the module-level API: `redact()`,
`redact_batch()`, `aredact()`, `aredact_batch()`, `redact_iter()`, `clear()` and
`add_custom_pattern()`.

```python
EuRedact(
    *,
    max_input_length: int = 10_485_760,
    allowlist: list[str] | None = None,
)
```

| Parameter | Default | Description |
|---|---|---|
| `max_input_length` | `10_485_760` (~10 MB) | Longest document `redact()` accepts, in characters. Above it, `ValueError` — split the input or raise the ceiling. |
| `allowlist` | `None` | Values never to redact, for every call on this instance: your own organisation's name, its own addresses. Merged with the per-call `allowlist`. See [Allowlist](#allowlist). |
| `allowlist_domains` | `None` | Domains never to redact, for every call on this instance. Merged with the per-call value. |

State held on the instance — the result cache, referential-integrity labels and
custom patterns — is per instance, which is what makes one instance per tenant
the right default. `clear()` releases the cache and the label mapping.

### Return Types

#### `DocumentContext`

```python
from euredact import DocumentContext

ctx = DocumentContext()
for offset, chunk in chunks:
    result = euredact.redact(chunk, context=ctx, chunk_offset=offset)
```

Shares country evidence across the chunks of one document, so a chunk carrying
no country signal of its own is still scored against what the rest of the
document showed. Pass the same object to every chunk, with `chunk_offset` set
to where the chunk starts. Passing a context disables the result cache, because
the result then depends on evidence from other chunks rather than on the text
alone.

| Method | Description |
|---|---|
| `add(evidence, chunk_offset=0)` | Record a chunk's evidence. `redact()` calls this for you. |
| `evidence()` | Every signal gathered so far, spans rebased onto the whole document. |

Not supported in cloud mode: the model has never seen a chunk boundary, so the
service rejects oversized input rather than splitting it.

#### `RedactResult`

Returned by `redact()` and all batch/async variants.

```python
@dataclass
class RedactResult:
    redacted_text: str          # The input text with PII replaced
    detections: list[Detection] # All PII spans found
    source: str = "rules"       # Detection backend ("rules")
    degraded: bool = False      # True if the engine fell back to a simpler mode

    # Country inference — see "Country-independent detection"
    inferred_countries: tuple[tuple[str, float], ...] = ()  # (country, confidence), strongest first
    evidence: tuple[CountryEvidence, ...] = ()              # every signal, with the span behind it
    detection_mode: str = "declared"                        # "declared" if countries= was passed,
                                                            # "inferred" otherwise
    tokens: dict[str, str] = {}                             # token -> original value; only with tokenize=True
    exempted: list[Exemption] = []                          # spans the allowlist kept, with the rule that matched
```

#### `Detection`

A single PII span. Frozen dataclass (immutable, hashable).

```python
@dataclass(frozen=True)
class Detection:
    entity_type: EntityType | str  # PII category (EntityType enum or custom name)
    start: int                     # Start offset in the original text
    end: int                       # End offset (exclusive) in the original text
    text: str                      # The matched substring
    source: DetectionSource        # "rules" or "cloud"
    country: str | None            # ISO code of the matched country, or None for shared/custom patterns
    confidence: str = "high"       # "high" | "medium" | "low" — see below
    country_confidence: float = 0.0  # How strongly the document supports `country`, in [0, 1].
                                     # 0.0 means the attribution rests on a checksum alone.
    out_of_scope: bool = False       # True when attributed outside the declared `countries`.
                                     # Flagged, never dropped.
```

**`confidence`** describes how the *type* was arrived at. It never says anything
about whether the span is masked — every detection is, at every level.

| value | meaning |
|---|---|
| `"high"` | a pattern matched and, where one exists, its checksum passed |
| `"medium"` | the type comes from a label touching the span, because no pattern of that type claimed it — `Αρ. Ταυτότητας: 00892341` is a `NATIONAL_ID` although nothing can checksum it |
| `"low"` | a pattern matched, its checksum *failed*, and the document labels the span as that very type — `Rijksregisternummer: 85.03.19-284.73` is a national number with a bad check digit |

Filter on it when you need only checksum-backed detections:

```python
strict = [d for d in result.detections if d.confidence == "high"]
```

A `"low"` detection is the honest description of a mistyped, OCR'd or invented
identifier: the shape and the label agree, the check digit does not. Earlier
versions dropped these, which meant a redaction library printed in full an
identifier it had recognised and rejected.

#### `EntityType`

String enum with all supported PII categories:

```
PERSON_NAME       ADDRESS           BANK_ACCOUNT      BIC
CREDIT_CARD       PHONE             EMAIL             DOB
DATE_OF_DEATH     NATIONAL_ID       SSN               TAX_ID
PASSPORT          DRIVERS_LICENSE   RESIDENCE_PERMIT  LICENSE_PLATE
VIN               VAT               POSTAL_CODE       IP_ADDRESS
IPV6_ADDRESS      MAC_ADDRESS       HEALTH_INSURANCE  HEALTHCARE_PROVIDER
CHAMBER_OF_COMMERCE  IMEI          GPS_COORDINATES   UUID
SOCIAL_HANDLE     SECRET            INTERNAL_ID       OTHER

Cloud tier only — the rule engine never emits these, because there is no shape
to match on. That is precisely why the model exists:

ORGANISATION_NAME JOB_TITLE         MEDICAL_CONDITION SENSITIVE_ATTRIBUTE
BIOMETRIC_REF     FINANCIAL_AMOUNT  QUASI_IDENTIFIER  CREDENTIAL
URL
```

`INTERNAL_ID` — an employee, badge or customer number tied to a person — is
emitted **only** when an explicit label names it (`medarbejdernummer:`,
`Personalnummer:`, `Employee No:`, `Betriebsstättennr.`, `Badge`). There is no
pattern for one, because there is no shape for one: without the label, a digit
run is not distinguishable from any other. The type exists so that a labelled
employee number is filed correctly instead of being claimed by the phone
pattern.

#### What a label in front of a value decides

A label touching the left edge of a value decides what that value is called.
The label may be the abbreviation or the word it abbreviates — `BSN:` and
`Burgerservicenummer:` both reach `NATIONAL_ID` — and it may be the official
name in the document's own language:

| label | type |
|---|---|
| `Companies House Registration:`, `Company Registration Number:` | `CHAMBER_OF_COMMERCE` |
| `Sozialversicherungsnummer:` | `SSN` |
| `Passport No.:`, `Paspoortnummer:` | `PASSPORT` |
| `AGB-code:`, `LANR`, `GMC Number:` | `HEALTHCARE_PROVIDER` |
| `Medical Card No.:` | `HEALTH_INSURANCE` |
| `sort code`, `account number` | `BANK_ACCOUNT` |
| `TAN-activatiecode`, `activation code` | `SECRET` |

A label can also *rule a type out*. A four-digit run is not a postal code when
a founding or payment participle introduces it (`Opgericht in 2016`,
`Fondée en 2017`) or when it sits in a telephone parenthetical (`(toest. 3841)`,
`(ext. 2219)`). Postal codes that merely look like years are unaffected —
Antwerp's `2018` in `rustige ligging in 2018` is still a `POSTAL_CODE`.

A label never moves a span; it only decides the label. Which characters are
masked is unchanged either way.

For custom patterns registered via `add_custom_pattern()`, `entity_type` is a
plain string (e.g. `"EMPLOYEE_ID"`) rather than an `EntityType` enum member.

#### `DetectionSource`

String enum: `"rules"` or `"cloud"`.

## Batch processing and concurrency

Three entry points, and the difference between them is memory and ordering, not
detection. All three load the country configurations once instead of per
document, which is the whole reason to prefer them to a loop.

| Call | Shape | Use when |
|---|---|---|
| `redact_batch(texts)` | `list[str] -> list[RedactResult]` | you have a list and want it back in order |
| `aredact_batch(texts, max_concurrency=4)` | `list[str] -> list[RedactResult]`, awaited | you are in async code; work runs in a thread pool |
| `redact_iter(texts)` | `Iterator[str] -> Iterator[RedactResult]` | the input does not fit in memory |

```python
import asyncio
import euredact

docs = ["Mijn BSN is 111222333.", "IBAN NL91ABNA0417164300."]

# Synchronous, order preserved.
for result in euredact.redact_batch(docs, countries=["NL"]):
    print(result.redacted_text)

# Async, four at a time. The engine is CPU-bound, so this offloads to threads
# rather than pretending to be I/O.
results = asyncio.run(euredact.aredact_batch(docs, countries=["NL"], max_concurrency=8))

# Streaming: nothing accumulates.
with open("corpus.txt") as fh:
    for result in euredact.redact_iter(fh, countries=["NL"]):
        ...
```

`max_concurrency` defaults to 4. Raising it past your core count does not help —
the work is regex matching, not waiting.

### Reuse one instance

The module-level functions share a single hidden instance, which is usually what
you want. Construct your own when you need the configuration to be per-tenant:
custom patterns, an allowlist and the result cache all live on the instance.

```python
from euredact import EuRedact

engine = EuRedact(allowlist=["ACME Corporation"])
engine.add_custom_pattern("CASE_REF", r"CASE-\d{8}")

engine.redact_batch(
    ["ACME Corporation, ref CASE-20260401, BSN 111222333."], countries=["NL"]
)[0].redacted_text
# 'ACME Corporation, ref [CASE_REF], BSN [NATIONAL_ID].'
```

Reusing an instance is what makes the cache useful across documents. Instances
are safe to share across threads and across concurrent async tasks.

### Tokens and referential labels do not span a batch

`tokenize=True` mints tokens **per call**, so item 3 of a batch does not share a
token with item 1 even for the same value. If two documents must agree on a
label, redact them as one text, or use `referential_integrity=True`, whose
`TYPE_n` numbering is per instance rather than per call.

## Cloud tier

> **Status: private alpha.** The cloud tier is in closed testing — it is **not**
> in public beta and is not generally available. Keys are issued to alpha
> participants only, and the request/response surface may still change between
> releases. The rules engine below is unaffected and is the supported path:
> `mode="rules"` is the default and carries no alpha caveat.

The rule engine catches what has a shape: IBANs, national IDs, phone numbers,
anything with a checksum. It cannot catch what does not — a person's name, an
employer, a diagnosis, a job title. The cloud tier adds a fine-tuned model
asked only *what did the rules miss?*

```bash
pip install 'euredact[cloud]'
```

```python
import euredact

euredact.configure(api_key="erk_...")          # or set EUREDACT_API_KEY
result = euredact.redact(text, countries=["BE"], mode="cloud")

result.source                                   # "cloud"
[(d.entity_type, d.text) for d in result.detections]
# [(<EntityType.PERSON_NAME>, 'Bas Verhoeven'), (<EntityType.PHONE>, '+32 ...')]
```

`mode="cloud"` **raises** `NotConfiguredError` when the tier is not configured.
It never falls back to rules-only output: a caller who believes names and
diagnoses were checked, and ships a document that only had its phone numbers
masked, is the one failure this library must not have.

`aredact()` takes the same arguments from async code:

```python
result = await euredact.aredact(text, countries=["BE"], mode="cloud")
```

`CloudClient` and `AsyncCloudClient` in `euredact.cloud` are the transport
underneath. They send exactly the text they are given and return spans relative
to it: the local masking happens in `redact()`, not in the client. Use them
directly only with text you have already masked.

Retries carry an `Idempotency-Key`, so a retry after a timeout cannot bill
twice. `Retry-After` is obeyed. A document that outlives the service's sync
window is polled transparently — callers never write that branch. Oversized
input raises `TooLargeError` (413): the service refuses it rather than
chunking, because the model has never seen a chunk boundary.

### What leaves your machine

Be precise about this, because it is the question a security review asks first.

In `mode="cloud"` the rules engine runs **on your machine first**, and only
the text it leaves behind is sent to the service over TLS. Every value it finds
is already a `[TYPE]` placeholder by the time the request is built:

```
your text      IBAN NL91 ABNA 0417 1643 00 belongs to Nick Bols, tel +31 6 12345678
                 │  rules engine, on your machine
what is sent   IBAN [BANK_ACCOUNT] belongs to Nick Bols, tel [PHONE]
                 │  TLS ──▶ service (model) ──▶ spans relative to the masked text
what you get   IBAN [BANK_ACCOUNT] belongs to [PERSON_NAME], tel [PHONE]
```

The request body is that one `text` field plus the country. No list of types,
no offsets and no original values travel beside it. The service answers with
spans that index the masked text, and the SDK maps them back onto your
original, so `detections` always index the document you passed in.

**This is minimisation, not an exemption.** Three things still leave:

- what the model is there to find — names, employers, job titles, diagnoses;
- anything the rules engine missed, such as an identifier in a format it does
  not know;
- the surrounding prose.

So the service still processes personal data on your behalf, and a DPIA should
say so. What it no longer receives is every IBAN, national ID, phone number and
email address the local engine could place.

Three things behave differently here than in rules mode:

- `detect_dates` has no effect. The local pass always runs with dates on,
  because that is the rules output the model was trained against. It can only
  cause more to be masked, never less.
- `tokenize` and the allowlists are applied locally, after the response, and do
  not change what is sent: placeholders on the wire are always `[TYPE]`, and an
  allowlisted value is masked in the request and put back in your result.
- [Custom patterns](#custom-patterns) run in the local pass, so a value only
  you know how to recognise is masked before the request too.

Every span the service returns is checked against the text that was sent. One
that does not match raises `CloudError` rather than being placed by guesswork.

### Keeping identifiers local

`mode="cloud"` does this for you with our model. The same pattern works with
any other model — your own, or a general-purpose LLM — and that is what
`tokenize=True` is for:

```python
import euredact

prompt = "Stuur een mail naar Bas Verhoeven (bas@example.nl) over NL91ABNA0417164300."

# 1. Mask locally. Nothing has left the process.
local = euredact.redact(prompt, countries=["NL"], tokenize=True)
local.redacted_text
# 'Stuur een mail naar Bas Verhoeven (EMAIL_S5SH) over BANK_ACCOUNT_5XXR.'
local.tokens
# {'BANK_ACCOUNT_5XXR': 'NL91ABNA0417164300', 'EMAIL_S5SH': 'bas@example.nl'}

# 2. Send only the masked text to any model, ours or someone else's.
answer = call_your_model(local.redacted_text)

# 3. Put the real values back locally.
euredact.restore(answer, local.tokens)
```

The token suffixes are random per call, so yours will differ.

**Read that output carefully: `Bas Verhoeven` is still there.** That is the
whole trade-off and it is why the cloud tier exists. The local engine masks what
has a shape — the IBAN and the address — and cannot mask a name, an employer or
a diagnosis, because it cannot find them. So this pattern keeps every identifier
the engine detects inside your process and sends the prose, names included.
`mode="cloud"` sends that same masked prose to a model trained to find the
names in it.

Either way the prose leaves. If names must not leave your infrastructure
either, no hosted model can help; run one yourself against
`local.redacted_text`.


### `euredact.configure()`

```python
euredact.configure(
    api_key: str | None = None,
    *,
    base_url: str | None = None,
    timeout_s: float = 30.0,
    poll_timeout_s: float = 300.0,
    max_retries: int = 3,
    headers: dict[str, str] | None = None,
) -> CloudConfig
```

| Parameter | Default | Environment variable | Description |
|---|---|---|---|
| `api_key` | — | `EUREDACT_API_KEY` | Your alpha key. Required; `configure()` raises without one. |
| `base_url` | `https://api.euredact.dev` | `EUREDACT_BASE_URL` | Service endpoint. |
| `timeout_s` | `30.0` | — | Per-request timeout. |
| `poll_timeout_s` | `300.0` | — | Ceiling for polling a document that outlives the synchronous window. |
| `max_retries` | `3` | — | Retries for `429` and `5xx`. Each carries an `Idempotency-Key`, so a retry after a timeout cannot bill twice, and `Retry-After` is obeyed. |
| `headers` | `None` | — | Extra headers sent with every request. |

`configure()` reads the two environment variables itself, so it must still be
called — but with no arguments if the environment is set. Errors are
`NotConfiguredError`, `QuotaExceededError`, `TooLargeError` (413, the service
refuses oversized input rather than chunking it) and `CloudError`.

Options the service cannot honour raise rather than being ignored: multiple
`countries`, `country_hint`, `context`/`chunk_offset`, `referential_integrity`
and `coref`. `tokenize`, `allowlist` and `allowlist_domains` are honoured: the SDK applies them to
the spans the service returns and rebuilds the text from those.

## `NAME` is now `PERSON_NAME`

The canonical type name is `PERSON_NAME`; `NAME` is a legacy alias, exactly as
`IBAN` aliases `BANK_ACCOUNT`. The placeholder written into redacted text is
`[PERSON_NAME]`.

Nothing could have depended on the old value: the type is cloud-only and the
cloud tier was stubbed until this release, so it was never emitted. Code
matching the *string* `"NAME"` should be updated; `LEGACY_TYPE_ALIASES`
publishes the mapping, and `STREET_ADDRESS` → `ADDRESS` and
`NATIONALITY_ETHNICITY` → `SENSITIVE_ATTRIBUTE` are recognised the same way.

## What `countries` actually controls

This is the parameter most often misread, and reading it wrongly leads to
under-redaction in exactly the case you care about — a foreign identifier in a
domestic document. So, precisely:

> `countries` decides how a detection is **attributed and scored**.
> It does **not** decide what gets **found**.

Country-specific patterns for every supported country run on every document,
whatever you declare. Declaring `countries=["NL"]` does not switch the Belgian
patterns off; it says "this document is Dutch", and anything Belgian that turns
up is still detected and still masked — it is just flagged as sitting outside
what you declared.

The same Belgian national number, under four different calls:

```python
import euredact

text = "Rijksregisternummer 85.07.30-033.61 van onze klant."

for kwargs in ({"countries": ["NL"]},
               {"countries": ["BE"]},
               {},
               {"countries": ["NL"], "country_hint": ["BE"]}):
    result = euredact.redact(text, **kwargs)
    detection = result.detections[0]
    print(result.redacted_text, "|", result.detection_mode,
          "| country:", detection.country,
          "| country_confidence:", round(detection.country_confidence or 0, 2),
          "| out_of_scope:", detection.out_of_scope)
```

```
Rijksregisternummer [NATIONAL_ID] van onze klant. | declared | country: BE | country_confidence: 0 | out_of_scope: True
Rijksregisternummer [NATIONAL_ID] van onze klant. | declared | country: BE | country_confidence: 0.88 | out_of_scope: False
Rijksregisternummer [NATIONAL_ID] van onze klant. | inferred | country: BE | country_confidence: 0 | out_of_scope: False
Rijksregisternummer [NATIONAL_ID] van onze klant. | declared | country: BE | country_confidence: 0.88 | out_of_scope: True
```

**The masked output is byte-identical in all four.** What moved is the metadata.

| You pass | Effect on what is found | Effect on labelling |
|---|---|---|
| `countries=["NL"]` | None — every country's patterns still run | Attributed normally; a non-Dutch hit gets `out_of_scope=True` and `country_confidence=0.0` |
| `countries=["BE"]` | None | The Belgian hit is corroborated: `country_confidence=0.88`, `out_of_scope=False` |
| `countries=None` | None | `detection_mode="inferred"`; the country is inferred from the document and nothing is out of scope, because nothing was declared |
| `country_hint=["BE"]` | None | A *prior*: it corroborates Belgian attribution **without** widening or narrowing scope, so `out_of_scope` still reflects `countries` alone |

### How to use each one

- **`countries=[...]`** — you know where the document is from. Use it. It
  improves attribution and makes `out_of_scope` meaningful, and it is the
  cheapest path because country inference is skipped.
- **`countries=None`** — you do not know. The engine infers from the document
  and reports `inferred_countries` with confidences.
- **`country_hint=[...]`** — you have a weak signal (the customer's billing
  country, the mailbox a document arrived in) that should resolve ambiguity
  without declaring scope. Added in 0.3.2 precisely so that a hint could not be
  mistaken for a scope restriction.

### Why it works this way

Until 0.3.2, `countries` *did* gate detection, and 0.3.3 fixed the remaining
case where it "could change which spans were found, not just how they were
labelled". Both were changed deliberately: a redaction library that hides a
Belgian national number because you told it the document was Dutch has failed
at the only job it has. Scoping is a reporting concern; masking is not.

The practical consequence: **do not use `countries` as a filter.** If you only
want Dutch detections in your output, filter on
`detection.country == "NL"` or on `detection.out_of_scope` after the call —
the value was still masked in the text either way.

## Country codes

`countries=[...]` accepts **ISO 3166-1 alpha-2** codes. The two EU/VAT
spellings are accepted as equivalents:

| ISO 3166-1 | EU/VAT | |
|---|---|---|
| `GB` | `UK` | United Kingdom |
| `GR` | `EL` | Greece |

Codes are case-insensitive and whitespace-tolerant. An **unrecognised** code
does not raise — it emits an `UnknownCountryWarning` and detection continues
with the shared, country-independent patterns:

```python
euredact.redact(text, countries=["ZZ"])
# UnknownCountryWarning: Unknown country code: 'ZZ'. Continuing with shared
# country-independent patterns only (email, IBAN, international phone, ...).
```

This is deliberate. Raising on an unknown locale invites callers to wrap the
call in `try/except` and skip redaction entirely — failing open, with
unredacted PII in the output.

## Country-independent detection

**`countries=[...]` never gates detection.** Every pattern runs on every
document, whatever you pass. The country you declare decides *how a match is
labelled*, never *whether it is found*.

This is the engine's central invariant, enforced by
`tests/test_invariant_generation.py`: no value of `countries=` may change which
spans are detected. A wrong or missing country cannot cause a miss — silent
recall loss is invisible in testing and surfaces in a breach report, whereas a
false positive is recoverable.

It was not always so. `countries=["BE"]` used to make a valid Dutch BSN vanish
entirely, because the Dutch patterns were never run:

```python
euredact.redact("Werknemer met BSN 111222333", countries=["BE"])
# before: 'Werknemer met BSN 111222333'   <- leaked
# now:    'Werknemer met BSN [NATIONAL_ID]'
```

Entities found outside the countries you declared are **flagged, not dropped**:

```python
det = euredact.redact("BSN 111222333", countries=["BE"]).detections[0]
det.out_of_scope   # True — detected, masked, and marked as outside your scope
```

So a Belgian IBAN in a document processed with `countries=["AT"]` is still
detected — cross-border traffic (a foreign invoice in a local file, an employee
paid to a foreign account) does not leak:

```python
euredact.redact("Rekening: BE68 5390 0754 7034", countries=["AT"])
# -> 'Rekening: [BANK_ACCOUNT]'
```

### Which country a value belongs to

Because every pattern runs, the same digits often match several countries'
schemes. Of the national-ID values in our corpus that pass any country's checksum,
34.7% pass more than one country's (32,827 of 94,528), so the digits alone
cannot decide it — the *document* does.

The engine infers the document's countries from entities that carry their
country in the string, then uses that to resolve the ambiguity:

```python
r = euredact.redact("Bereikbaar op telefoon 0612345678, mail jan@test.nl")
r.inferred_countries          # (('NL', 0.98),)
r.detections[0].entity_type   # PHONE

r = euredact.redact("Kontakt: 0612345678, e-mail jens@test.dk")
r.inferred_countries          # (('DK', 0.98),)
r.detections[0].entity_type   # NATIONAL_ID
```

Identical digits, different answer — `0612345678` is both a valid Dutch mobile
number and a valid Danish CPR. Only the surrounding document distinguishes them.

Every inference is auditable: `result.evidence` lists each signal, its weight,
and the span that produced it.

| Signal | Weight (log-odds) | Measured reliability |
|---|---:|---|
| `e164_prefix` | 4.00 (capped) | 41,402 / 41,402 |
| `bic_country` | 4.00 (capped) | 2,588 / 2,588 |
| `email_tld` | 4.00 (capped) | 97,865 / 98,949 |
| `vat_prefix` | 2.84 | 19,022 / 20,136 |
| `iban_prefix` | 1.94 | 110,572 / 126,428 |

Weights are derived from the corpus, not chosen by hand. The IBAN prefix being
weakest is real and worth knowing: a Belgian IBAN in a Dutch invoice is
ordinary, so an account's country is only weak evidence about the document's.

Confidences are per-country and do **not** sum to 1 — document countries are
not mutually exclusive. A Belgian supplier invoicing a German customer is
genuinely both.

International phone numbers are matched by a generic E.164 pattern (`+`
followed by 8-15 digits, any grouping or separators, including `(0)` trunk
prefixes) that runs alongside the per-country patterns.

## BIC detection

BIC is the only bank identifier in the engine with **no check digit** — IBAN
has mod-97, VAT has country-specific checksums. ISO 9362 structure alone
cannot decide a match, because characters 5-6 of ordinary uppercase words are
frequently valid ISO 3166 country codes (`DRINGEND` → `GE`, `HOSPITAL` →
`IT`). Detection is therefore gated:

| Stage | Condition | Result |
|---|---|---|
| Gate 0 | the token also occurs as an ordinary lowercase word in the same document | never emitted |
| Tier 1 | registry hit on the BIC6 institution+country prefix | emitted |
| Gate 2 | heading / shouted-word shape | never emitted |
| Tier 2 | `BIC`/`SWIFT` keyword, an IBAN, or a bank block in the enclosing line, record or paragraph | emitted |
| — | none of the above | never emitted |

The context window is the enclosing **line, record or paragraph**, not a
character count — a banking cue often sits several fields away in the same
CSV row.

### Supplying your own BIC registry

The package bundles **no licensed BIC data**. The authoritative SWIFTRef BIC
Directory is a commercial product, and redistributing it inside a package
requires a specific redistribution licence. What ships is a small seed list of
BIC6 prefixes for major European banks, compiled from publicly published bank
data.

Deployments holding a licensed directory install it at startup:

```python
import euredact

# A path to a newline-delimited file of BICs...
euredact.set_bic_registry("/etc/euredact/swiftref-bics.txt")

# ...an iterable...
euredact.set_bic_registry({"ABNANL2A", "INGBNL2A", "BBRUBE"})

# ...or any membership callable.
euredact.set_bic_registry(lambda bic: bic in my_directory)

# Remove it again:
euredact.set_bic_registry(None)
```

Entries may be full BIC8/BIC11 codes or bare BIC6 prefixes; both are matched,
case-insensitively and ignoring spaces.

The registry is an **accept** signal, never a filter. A code missing from it
falls through to the context gate and is still detected when banking context
is present, so a stale list costs a little recall on bare, contextless BICs —
it never causes a leak. Annual review is sufficient.

## Custom Patterns

Register domain-specific PII patterns at runtime. Custom patterns are detected
alongside built-in patterns and participate in the same deduplication pipeline.

```python
import euredact

# Register patterns
euredact.add_custom_pattern("EMPLOYEE_ID", r"EMP-\d{6}")
euredact.add_custom_pattern("CASE_REF", r"CASE-\d{8}")

# They are detected alongside built-in PII
result = euredact.redact(
    "Employee EMP-123456, email jan@example.com, ref CASE-20260401"
)
print(result.redacted_text)
# "Employee [EMPLOYEE_ID], email [EMAIL], ref [CASE_REF]"

# Check detections
for d in result.detections:
    print(f"  {d.entity_type}: {d.text}")
# EMPLOYEE_ID: EMP-123456
# EMAIL: jan@example.com
# CASE_REF: CASE-20260401
```

### How Custom Patterns Work

- `name` becomes the entity type reported in detections and used in replacement
  tags (e.g. `[EMPLOYEE_ID]`)
- `pattern` is a Python regular expression (same syntax as `re` module)
- Custom patterns are always active regardless of the `countries` parameter
- Custom patterns have no validator (they are purely regex-based)
- In overlap resolution, custom patterns have higher priority than built-in
  regex-only patterns but lower priority than built-in patterns with a passing
  checksum validator

### Instance Isolation

Custom patterns registered on the module-level function apply to the shared
singleton. For isolated pattern registrations, use separate `EuRedact` instances:

```python
from euredact import EuRedact

# Instance A detects employee IDs
a = EuRedact()
a.add_custom_pattern("EMPLOYEE_ID", r"EMP-\d{6}")

# Instance B detects case references
b = EuRedact()
b.add_custom_pattern("CASE_REF", r"CASE-\d{8}")

# Each instance only detects its own custom patterns
result_a = a.redact("EMP-123456 CASE-20260401")
result_b = b.redact("EMP-123456 CASE-20260401")
```

## Secret and API Key Detection

EuRedact includes built-in detection for API keys, tokens, and passwords. This is
always active -- no configuration required.

### Known-Prefix Patterns

The following token formats are detected with high confidence based on their
distinctive prefixes:

| Pattern | Description |
|---|---|
| `AKIA...` | AWS Access Key ID |
| `ghp_`, `gho_`, `ghs_`, `github_pat_` | GitHub tokens (PAT, OAuth, app, server) |
| `sk_live_`, `pk_live_`, `sk_test_`, `pk_test_` | Stripe secret and publishable keys |
| `sk-`, `sk-ant-` | OpenAI and Anthropic API keys |
| `xoxb-`, `xoxp-`, `xoxa-`, `xoxs-` | Slack tokens |
| `eyJ...` (3-part base64url) | JWT tokens |
| `SG.` | SendGrid API keys |

```python
result = euredact.redact("My API key is sk-proj-abc123def456ghi789jkl0")
print(result.redacted_text)
# "My API key is [SECRET]"
```

### Entropy-Based Detection

For secrets that don't have a recognizable prefix, EuRedact uses Shannon entropy
analysis. A high-entropy string (32+ characters of alphanumeric/base64 content) is
flagged as `SECRET` when it appears near context keywords like `key`, `token`,
`secret`, `password`, `credential`, `auth`, or `bearer` (including translations in
12 European languages).

```python
result = euredact.redact("The api_key is xK9mPqR7vLnW2bFjY8cGhT4sDfAeU6iO")
print(result.redacted_text)
# "The api_key is [SECRET]"

# Without a context keyword, the same string is not flagged:
result = euredact.redact("identifier: xK9mPqR7vLnW2bFjY8cGhT4sDfAeU6iO")
print(result.redacted_text)
# "identifier: xK9mPqR7vLnW2bFjY8cGhT4sDfAeU6iO"  (unchanged)

# Low-entropy strings are also not flagged, even with context:
result = euredact.redact("The password is aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")
print(result.redacted_text)
# "The password is aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"  (unchanged)
```

## Country Hints

Two arguments tell the engine about country, and neither restricts what is
looked for:

| Argument | Meaning |
|---|---|
| `countries=[...]` | **Scope.** Resolves ambiguity, and flags anything attributed elsewhere as `out_of_scope`. |
| `country_hint=[...]` | **Prior only.** Resolves ambiguity without narrowing scope or flagging anything. |

```python
# You know this batch is Swedish, but do not want foreign PII marked out of scope:
euredact.redact(text, country_hint=["SE"])

# You want anything non-Swedish flagged for review:
euredact.redact(text, countries=["SE"])
```

Declaring a country helps where a value is genuinely ambiguous and the document
carries no other signal:

```python
euredact.redact("Telefon: 0708787668", country_hint=["SE"]).detections[0]
# PHONE / SE  — without the hint this is a valid Danish CPR and nothing says otherwise
```

`result.detection_mode` reports which happened: `"declared"` if you passed
`countries=`, `"inferred"` otherwise.

Custom patterns are always active regardless of either parameter.

Passing neither is safe, and is the right default for mixed-origin data: the
engine infers what it can and reports it in `result.inferred_countries`. On our
152,300-document corpus, blind detection scores 99.6% precision against 99.8%
with country hints — a 0.2-point gap, down from 3.4 points before inference.

## Chunked documents

A long document is usually redacted in pieces. Each piece is scanned
independently, so each infers its own country — and a chunk that happens to
contain no IBAN, no `+CC` number and no ccTLD infers nothing at all, even when
page 1 identified the document beyond doubt:

```python
euredact.redact("Telefoon 0612345678")
# -> NATIONAL_ID (DK)   <- a valid Danish CPR, and nothing here says otherwise
```

Pass a `DocumentContext` to carry evidence forward across chunks:

```python
from euredact import DocumentContext

ctx = DocumentContext()
offset = 0
for page in pages:
    result = euredact.redact(page, context=ctx, chunk_offset=offset)
    offset += len(page)

# page 1: "Factuur — IBAN NL91ABNA0417164300, info@example.nl"
# page 7: "Telefoon 0612345678"  -> PHONE (NL)
```

`chunk_offset` rebases spans recorded in the context so they point into the
whole document. Returned detections stay relative to the chunk you passed in.

A context is thread-safe — `aredact_batch` fans chunks across a thread pool.
Caching is disabled automatically while one is in use, because the result then
depends on evidence the text alone does not determine.

Reuse a context only for chunks of the **same** document. Sharing one across
unrelated documents mixes their countries together. That cannot hide anything
— the invariant still holds — but it will attribute values to the wrong
national scheme.

## Referential Integrity

When `referential_integrity=True`, each unique PII value is mapped to a consistent
label within the session. The same input always produces the same label:

```python
import euredact

text = "BSN 111222333 en later weer 111222333, IBAN NL91ABNA0417164300."
result = euredact.redact(text, referential_integrity=True)
print(result.redacted_text)
# "BSN NATIONAL_ID_1 en later weer NATIONAL_ID_1, IBAN BANK_ACCOUNT_1."
```

The mapping is scoped to the `EuRedact` instance. The module-level `redact()`
function uses a shared singleton, so labels are consistent across calls within
the same process.

## Reversible tokenization

`tokenize=True` is for text that has to come back. Each value is replaced by a
token that names its type and nothing else, and the result carries the mapping
that turns tokens back into values:

```python
import euredact

prompt = "Write an email to Joren at joren.janssens@euredact.be or call +32 475 12 34 56 about the invoice."
result = euredact.redact(prompt, countries=["BE"], tokenize=True)
print(result.redacted_text)
# "Write an email to Joren at EMAIL_K7Q2 or call PHONE_P4RT about the invoice."
print(result.tokens)
# {'EMAIL_K7Q2': 'joren.janssens@euredact.be', 'PHONE_P4RT': '+32 475 12 34 56'}

reply = call_your_llm(result.redacted_text)   # sees tokens, never the values
print(euredact.restore(reply, result.tokens))
# the reply, with the real address and number back in it
```

The suffixes are random; yours will differ. With the [cloud tier](#cloud-tier)
the name is tokenized too (`PERSON_NAME_W3NB`), since person names have no
shape for the rules tier to match on.

Within one call the same value gets the same token, so a prompt that names
someone twice still reads as one person. Across calls it gets a different
token: nothing is retained on the instance, `result.tokens` is the only copy,
and two tokenized documents never reveal that they share a value. That is the
opposite retention model from `referential_integrity`, which is why the two
cannot be combined. Batch and iterator variants tokenize each text on its own.

A token is `TYPE_` plus four characters from `ABCDEFGHJKLMNPQRSTUVWXYZ23456789`
(no vowels, no `0`/`1`/`I`/`O`). Tokens are kept clear of any token-shaped
string already in the document, so an LLM's reply to a tokenized prompt can
itself be redacted without `restore()` putting the wrong value back.
`restore()` replaces every occurrence, including a token an LLM glued to other
characters (`EMAIL_P4RTs`) — leaving a token behind is the worse failure.

Works in cloud mode: tokens are minted locally, after the response, over the
local spans and the service's. What is sent to the service always carries
`[TYPE]` placeholders, never tokens.

## Allowlist

Values a caller declares are not PII to them — their own organisation's name,
their own addresses. Set it per call, on the instance, or both; the two merge:

```python
import euredact

sdk = euredact.EuRedact(allowlist=["ACME NV", "info@acme.be"])

text = "ACME NV: mail info@acme.be or jan@acme.be about IBAN NL91 ABNA 0417 1643 00."
print(sdk.redact(text, countries=["NL"]).redacted_text)
# "ACME NV: mail info@acme.be or [EMAIL] about IBAN [BANK_ACCOUNT]."

print(sdk.redact(text, countries=["NL"], allowlist=["jan@acme.be"]).redacted_text)
# "ACME NV: mail info@acme.be or jan@acme.be about IBAN [BANK_ACCOUNT]."
```

Matching is whole-span and case-insensitive. A bare string
(`allowlist="ACME NV"`) raises `TypeError` rather than being iterated into
single letters that exempt nothing. Works in cloud mode — including on types
only the model finds, such as `ORGANISATION_NAME` — and together with
`tokenize`.

### Spacing and punctuation

For **structured identifiers** separators are presentational, so an allowlisted
value matches however the document writes it:

```python
sdk = euredact.EuRedact(allowlist=["NL91ABNA0417164300"])
sdk.redact("Pay to NL91 ABNA 0417 1643 00.", countries=["NL"]).redacted_text
# 'Pay to NL91 ABNA 0417 1643 00.'   — exempt, despite the spacing
```

Applies to `BANK_ACCOUNT`, `BIC`, `CREDIT_CARD`, `PHONE`, `VAT`, `NATIONAL_ID`,
`SSN`, `TAX_ID`, `PASSPORT`, `DRIVERS_LICENSE`, `RESIDENCE_PERMIT`,
`HEALTH_INSURANCE`, `CHAMBER_OF_COMMERCE`, `IMEI` and `VIN`. It widens the
*spelling*, never the *scope*: a different account is still redacted.

Free-text types (`EMAIL`, `PERSON_NAME`, `ORGANISATION_NAME`, …) stay literal,
because folding would exempt values you never listed —
`jan.devries@acme.be` and `jandevries@acme.be` are different mailboxes at most
providers, and `ACME NV` folded would also match `ACMENV`. Case still does not
matter, so list a company's legal spellings if they vary:
`["ACME NV", "ACME N.V."]`.

### Exempting a whole domain

Enumerating every mailbox does not scale — a new hire's address is redacted in
your own documents until someone updates the list. `allowlist_domains` takes
the domain instead:

```python
sdk = euredact.EuRedact(allowlist_domains=["acme.be"])
sdk.redact("Mail jan@acme.be or piet@acme.be", countries=["NL"]).redacted_text
# 'Mail jan@acme.be or piet@acme.be'
```

It applies to `EMAIL` and `URL` only and covers subdomains (`mail.acme.be`).
The match is on a label boundary, so `acme.be` does **not** exempt
`evilacme.be`. Entries may be written `acme.be`, `@acme.be` or `.acme.be`.

There are deliberately **no wildcards**. The allowlist is the only option that
turns redaction *off*, so an over-broad entry fails toward under-redaction and
does so silently: `*@*` would disable email redaction entirely and `* Maes` a
family's names. A domain rule cannot be widened that way — it structurally
cannot match a person's name or an IBAN.

### What was exempted

Every exemption is reported, so a document that keeps a direct identifier can be
audited rather than taken on trust:

```python
r = euredact.redact("Mail jan@acme.be", countries=["NL"], allowlist_domains=["acme.be"])
for e in r.exempted:
    print(e.entity_type, e.text, e.rule, e.rule_kind)
# EntityType.EMAIL jan@acme.be acme.be domain
```

`Exemption` carries `entity_type`, `start`, `end`, `text`, the `rule` that
matched as you wrote it, and `rule_kind` (`"value"` or `"domain"`). An exempted
span is absent from `detections`, so `exempted` is the only record that it was
found at all.

## Architecture

```
Input text
    |
    v
[Normalizer] -- Unicode normalization, whitespace cleanup
    |
    v
[Pass 1: Pattern Matching] -- All country + shared + custom regexes
    |                          via MultiPatternMatcher (Aho-Corasick optional)
    v
[Pass 2a: Validation] -- Checksum validators (mod-97, Luhn, entropy, ...)
    |                     Failed spans are recorded, per entity type
    v
[Evidence]     -- Which countries does this document belong to? From IBAN
    |              prefixes, +CC codes, VAT prefixes, BIC, email ccTLDs
    v
[Pass 2b: Suppression] -- Remove false positives (currency amounts, units,
    |                      references). Failed checksums demote same-type
    |                      matches rather than deleting them
    v
[Deduplication] -- Priority-aware, country-evidence-weighted
    |               Longer span outranks declared country
    v
[Replacement] -- Right-to-left substitution with [ENTITY_TYPE] labels
    |               or labels
    v
RedactResult
```

The engine is **thread-safe**: a `threading.Lock` guards country loading and
custom-pattern registration, and all detection state is local to each `detect()`
call. `Detection` objects are frozen dataclasses and can be safely shared across
threads.

Registering a pattern while other threads are detecting is safe too:
`add_custom_pattern` builds a whole new scan plan and publishes it in one
assignment, so a scan in flight keeps using the plan it started with and sees
the pattern set as of either before or after the registration — never a mixture.
It will not observe the new pattern until it next calls `detect()`.

Two caveats that are about *sharing*, not locking:

- **Referential integrity is per instance.** Labels (`EMAIL_1`, …) come from a
  mapper owned by the `EuRedact` instance, so every caller of the module-level
  `euredact.redact()` shares one namespace. A label appearing in two documents
  tells you they contain the same underlying value. Give each tenant its own
  `EuRedact()` instance.
- **The mapper is never evicted**, because evicting would give a previously seen
  value a second label. It grows until you call `clear()`, which is the right
  thing to do between workloads in a long-running process.

### Suppression Zones

When a regex matches a pattern that has a checksum validator but the checksum
fails, the matched span becomes a "suppression zone." Any purely regex-based
detection fully contained within that zone is suppressed as a false positive.

For example, the text `BE71 0012 3456 7890` matches the Belgian IBAN regex but
fails mod-97 validation. Without suppression zones, sub-parts of this span might
be incorrectly detected as a license plate (`BE71`) or a phone number
(`0012 3456`). The suppression zone prevents these false positives while
correctly not reporting an invalid IBAN.

### Deduplication Priority

When multiple patterns match overlapping spans, the engine resolves conflicts
using a priority system:

| Tier | What |
|---:|---|
| 3 | **Validated** — a checksum validator passes *and* the document corroborates its country |
| 2 | **Custom patterns** registered via `add_custom_pattern()` |
| 1 | **Regex-only**, and validated patterns whose country the document does not corroborate |
| 0 | **Postal codes** — a bare digit run, the weakest evidence in the engine |
| -1 | **Demoted** — a validator-less match inside a failed checksum of *its own type* |

Within a tier, ranking is: longer span, then stronger country evidence, then
whether the country was declared. **Span length outranks country** deliberately:
preferring the declared country over the longest match truncates entities — with
`countries=["BE"]` the Belgian phone pattern claimed 11 of the 14 characters of
`06 12 34 56 78` and left three digits exposed. Country can change *which*
country is attributed, never *what* is masked.

Two tiers are less obvious than they look:

- **A passing checksum does not automatically win.** A weak checksum fits by
  luck — a mod-11 scheme accepts a random number about one time in eleven — so a
  validated candidate from a country the document shows no trace of drops to
  tier 1. Entities that carry their own country vouch for themselves (an IBAN
  emits evidence for its own country), so a foreign IBAN in a domestic invoice
  keeps tier 3.
- **Nothing is deleted for failing a checksum.** A failed checksum demotes
  rather than removes, and only candidates of *the same entity type*: it is
  evidence against that type, not against the span. Deleting instead removed 454
  detections across the corpus, of which 454 overlapped real labelled PII.

## Adding a New Country

Each country is a single Python file in `src/euredact/rules/countries/`. The
registry discovers new countries automatically -- no manual registration required.

1. Create a file, e.g. `src/euredact/rules/countries/gr.py`.
2. Define a `CountryConfig` subclass with patterns:

```python
"""Greece (GR) PII patterns."""

from euredact.rules.countries._base import CountryConfig, PatternDef
from euredact.types import EntityType


class GRConfig(CountryConfig):
    def __post_init__(self) -> None:
        self.code = "GR"
        self.name = "Greece"
        self.patterns = [
            PatternDef(
                entity_type=EntityType.NATIONAL_ID,
                pattern=r"\b[A-Z]{2}[0-9]{6}\b",
                validator=None,
                description="Greek national ID (example)",
            ),
        ]
```

That is all. The `CountryRegistry` scans the `countries/` package at startup and
picks up any module that defines a `CountryConfig` subclass with a non-empty
`code`. Files prefixed with `_` (like `_base.py` and `_shared.py`) receive
special treatment and are not treated as standalone countries.

Each `PatternDef` can specify:
- `entity_type` -- which `EntityType` this pattern detects
- `pattern` -- a regular expression
- `validator` -- an optional named validator (e.g. `"bsn"`, `"luhn"`, `"iban"`)
- `context_keywords` -- proximity keywords that boost confidence
- `requires_context` -- if `True`, the match is discarded without a nearby keyword

## Performance

Measured on one core (Apple Silicon M3 Pro, CPython 3.12.13), all 31 countries
loaded, `detect_dates=True`, cache off, over two cohorts sampled evenly from the
corpus: 3,000 distinct short records (~190 chars) and 611 distinct real
documents (~3,450 chars).

**Median** per document, 10th–90th percentile in brackets:

| Input | Pure Python | Aho-Corasick | With `[fast]` |
|---|---:|---:|---:|
| Short record (~190 chars) | 1,612 µs <br><sub>1,072 – 2,417</sub> | 1,507 µs <br><sub>956 – 2,293</sub> | **1,141 µs — 876/s** <br><sub>599 – 1,958</sub> |
| Real document (~3,450 chars) | 18.88 ms <br><sub>15.28 – 25.54</sub> | 15.95 ms <br><sub>12.51 – 22.70</sub> | **9.36 ms — 107/s** <br><sub>6.22 – 15.83</sub> |
| Memory per country | ~50 KB | ~50 KB | ~50 KB |

The spread matters more than the median: **cost tracks identifier density, not
length.** At an identical 3,424 characters, the cheapest document in the corpus
takes 4.8 ms on `[fast]` and the dearest 9.5 ms; across the whole long cohort
the range is 6.2–15.8 ms. A profile shows why — the dominant cost is not the
pattern scan but the cue and suppressor checks that run per candidate match, so
a document dense in identifiers pays for every one of them. Quote a figure for
*your* documents, not this table.

Making `\b` catch identifiers glued to a non-ASCII letter costs something, but
only where it must: next to a digit the ASCII reading alone is exactly
equivalent, so one lookaround replaces a three-way alternation, and plain `\b`
stays everywhere else. 0.3.3 applied the union to all 303 patterns and was
1.8×/2.8× slower for it.

```bash
pip install euredact[fast]
```

The `fast` extra installs two optional accelerators. Neither changes what is
detected — both are covered by `tests/test_scan_path_parity.py`, which runs
every available scan path against the plain-Python one and requires them to
agree:

- **`google-re2`** builds a prefilter over every pattern it can express (334 of
  345). One DFA pass per 1 KB window reports which patterns match anywhere in
  it — typically 42 of 314 for a real document — and only those are then run.
  Patterns RE2 cannot express, such as the lookbehind-based `SECRET` rules,
  always run.

  Asking the question per window matters: over a long document nearly every
  pattern matches *somewhere*, so a whole-document prefilter stops filtering
  (2.48× on a short record, decaying to 1.11× at 12 KB).

- **`pyahocorasick`** indexes patterns that begin with a literal and runs them
  only near a prefix hit. Used when `google-re2` is unavailable.

Detection cost is dominated by how much prose surrounds the PII, not by
document size alone: the same engine runs 1.4× faster on dense records and
~3× faster on prose-heavy documents where whole windows can be skipped.

The TypeScript SDK needs no such extra — V8's regex engine has literal
prefilters CPython lacks, and measured on the same documents it runs roughly 3×
faster than the accelerated Python path.

## License

Apache 2.0
