Metadata-Version: 2.5
Name: euredact
Version: 0.8.0
Summary: euRedact Core: European PII redaction SDK for 31 countries
Project-URL: Homepage, https://euredact.dev
Project-URL: Documentation, https://euredact.dev/docs
Project-URL: Repository, https://github.com/euredact/euredact
Project-URL: Changelog, https://github.com/euredact/euredact/blob/main/euredact-python/CHANGELOG.md
Project-URL: Issues, https://github.com/euredact/euredact/issues
Author: JNJS
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: anonymization,europe,gdpr,masking,pii,privacy,redaction,referential-integrity
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Provides-Extra: cloud
Requires-Dist: httpx>=0.27; extra == 'cloud'
Provides-Extra: coref
Provides-Extra: dev
Requires-Dist: google-re2>=1.1; extra == 'dev'
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pyahocorasick>=2.0.0; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
Requires-Dist: pytest-benchmark; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Provides-Extra: fast
Requires-Dist: google-re2>=1.1; extra == 'fast'
Requires-Dist: pyahocorasick>=2.0.0; extra == 'fast'
Description-Content-Type: text/markdown

# EuRedact

**European PII redaction SDK -- euRedact Core**

---

euRedact Core is the open-source, pure-Python SDK for detecting and redacting
personally identifiable information (PII) in European text data. It covers 31 countries with a two-pass
architecture: liberal pattern matching in the first pass, followed by suppression
filters and checksum validation in the second. The library has zero required
dependencies, is thread-safe, and produces immutable detection objects.

## Quick Start

```bash
pip install euredact
```

```python
import euredact

result = euredact.redact("Mijn BSN is 111222333 en IBAN NL91ABNA0417164300.")
print(result.redacted_text)
# "Mijn BSN is [NATIONAL_ID] en IBAN [BANK_ACCOUNT]."

print(result.detections)
# [Detection(entity_type=<EntityType.NATIONAL_ID>, ...), Detection(entity_type=<EntityType.BANK_ACCOUNT>, ...)]
```

## Features

- **31 European countries** (see list below)
- **41 PII entity types** (29 from the rules engine, 11 from
  [euRedact Cloud](#cloud-tier), and `OTHER`, the cloud client's fallback):
  national IDs, IBANs, phone numbers, email addresses, VAT numbers, license plates, VIN, credit cards, BIC/SWIFT, IMEI, GPS
  coordinates, UUIDs, social handles, MAC addresses, IP/IPv6 addresses, health
  insurance numbers, passport numbers, driver's licenses, secrets/API keys, and more
- **Secret/API key detection:** known-prefix patterns for AWS, GitHub, Stripe,
  OpenAI, Slack, JWT, SendGrid, plus Shannon entropy-based detection for generic
  high-entropy tokens near context keywords
- **Custom patterns:** register your own regex patterns for domain-specific PII
  types at runtime via `add_custom_pattern()`
- **Reversible tokenization:** `tokenize=True` swaps values for `EMAIL_K7Q2`-style
  tokens and `restore()` puts them back — for prompts that go to an LLM and
  come back
- **Allowlist:** values that are never redacted, such as your own organisation's
  name and addresses, per call or per instance
- **Checksum validation:** IBAN mod-97, Luhn (credit cards), and 30+ country-specific
  national ID checksums (e.g., Dutch BSN 11-proof, Belgian national number modulo)
- **Priority-aware deduplication:** when matches overlap, validated patterns
  (with passing checksums, corroborated by the document's country) win over
  custom patterns, which win over regex-only patterns; a failed checksum demotes
  same-type matches rather than deleting them, so it can never silence a
  detection
- **Two-pass detection:** liberal regex matching followed by suppression filters
  that eliminate false positives
- **Context-aware:** keyword proximity checks and structural detection (JSON field
  names, CSV headers) for ambiguous patterns like dates of birth
- **Country self-detection:** infers a document's countries from the entities
  that carry one, so an ambiguous value resolves without the caller naming a
  country — and `countries=` never gates what is looked for
- **Fast:** ~1.1 ms for a short record, ~9 ms for a 3,500-character document
  with `[fast]` installed — though cost tracks identifier density more than
  length (see [Performance](#performance))
- **Zero required dependencies** (`pip install euredact[fast]` adds optional acceleration)
- **Thread-safe,** immutable `Detection` objects (frozen dataclasses)

### Supported Countries

| Region | Countries |
|---|---|
| Western Europe | AT, BE, CH, DE, FR, LU, NL |
| Southern Europe | CY, EL, ES, IT, MT, PT |
| Northern Europe | DK, EE, FI, IS, LT, LV, NO, SE |
| Eastern Europe | BG, CZ, HR, HU, PL, RO, SI, SK |
| British Isles | IE, UK |

## API Reference

### Which option do I need?

| I want to… | Use |
|---|---|
| redact a document with no further setup | `redact(text)` — all 31 countries, `[ENTITY_TYPE]` output |
| restrict scope to the countries I operate in | `countries=["NL", "BE"]` |
| keep detection wide but resolve ambiguity | `country_hint=["DE"]` |
| send a prompt to an LLM and restore the reply | `tokenize=True`, then [`restore()`](#euredactrestore) |
| keep relationships visible across a whole session | `referential_integrity=True` |
| never redact my own company name or addresses | `allowlist=[...]`, or `EuRedact(allowlist=[...])` |
| never redact anything at my own domain | `allowlist_domains=["acme.be"]` |
| leave whole types in the text (job titles, amounts) | `exclude_types=["JOB_TITLE", "FINANCIAL_AMOUNT"]`, or `EuRedact(exclude_types=[...])` — see [Excluding entity types](#excluding-entity-types) |
| audit what the allowlist or `exclude_types` kept in the document | `result.exempted` |
| catch person names, employers, job titles, diagnoses | `mode="cloud"` — see [Cloud tier](#cloud-tier) |
| include dates of birth | `detect_dates=True` |
| redact a document too large for one call | `context=` + `chunk_offset=` — see [Chunked documents](#chunked-documents) |
| detect an identifier the engine does not know | [`add_custom_pattern()`](#euredactadd_custom_pattern) |
| isolate tenants from each other | one `EuRedact()` instance each — see [Instance Isolation](#instance-isolation) |
| free PII held in memory | `clear()` |


EuRedact provides both module-level functions (using a shared singleton) and an
instance-based `EuRedact` class. The module-level API is the easiest way to get
started; the class-based API gives you isolated instances with separate caches
and custom pattern registrations.

### Module-Level Functions

#### `euredact.redact()`

```python
euredact.redact(
    text: str,
    *,
    countries: list[str] | None = None,
    country_hint: list[str] | None = None,
    context: DocumentContext | None = None,
    chunk_offset: int = 0,
    mode: str = "rules",
    referential_integrity: bool = False,
    tokenize: bool = False,
    allowlist: list[str] | None = None,
    detect_dates: bool = False,
    coref: bool = False,
    coref_model: str = "default",
    cache: bool = True,
    idempotency_key: str | None = None,
) -> RedactResult
```

Main entry point. Detects and redacts PII in the given text. Every option is
keyword-only.

**What is looked for**

| Parameter | Default | Description |
|---|---|---|
| `text` | — | Input text to scan. Longer than `max_input_length` raises `ValueError`. |
| `countries` | `None` | ISO 3166-1 alpha-2 codes that define **scope** (e.g. `["NL", "BE"]`). `None` loads all 31. This never gates what is looked for: a detection attributed elsewhere is flagged `out_of_scope`, never dropped. Passing a bare string raises `TypeError`. See [Country codes](#country-codes). |
| `country_hint` | `None` | A **prior only**. Helps resolve an ambiguous value without narrowing scope or flagging anything out of scope. See [Country Hints](#country-hints). |
| `detect_dates` | `False` | Include `DOB` and `DATE_OF_DEATH`. Off by default: a bare date without keyword or structural context is deferred to the cloud tier. When on, the engine applies keyword and JSON/CSV-header checks. |

**How the output looks**

The three output styles are mutually exclusive in practice — pick at most one.

| Parameter | Default | Description |
|---|---|---|
| *(none)* | — | Default: each span becomes `[ENTITY_TYPE]`. |
| `referential_integrity` | `False` | Replace each distinct value with a consistent label (`EMAIL_1`). Labels persist **on the instance** across calls, so two documents sharing a value get the same label. See [Referential Integrity](#referential-integrity). |
| `tokenize` | `False` | Replace each value with a reversible token (`EMAIL_K7Q2`) and return the token → value mapping in `result.tokens`; pass it to [`restore()`](#euredactrestore). Tokens are unique to the **call**, so nothing is retained. Cannot be combined with `referential_integrity`. See [Reversible tokenization](#reversible-tokenization). |
| `allowlist` | `None` | Values never to redact, matched whole and case-insensitively, merged with the instance's list. Structured identifiers (IBAN, phone, VAT, …) also match across spacing and hyphenation. Applies to cloud-tier types too. See [Allowlist](#allowlist). |
| `allowlist_domains` | `None` | Domains whose addresses are never redacted, e.g. `["acme.be"]`. Applies to `EMAIL` and `URL` only, and covers subdomains. See [Allowlist](#allowlist). |
| `exclude_types` | `None` | Entity types neither reported nor masked, e.g. `["JOB_TITLE"]`; merged with the instance's list. Unknown names raise. See [Excluding entity types](#excluding-entity-types). |

**Long documents and tiers**

| Parameter | Default | Description |
|---|---|---|
| `mode` | `"rules"` | `"rules"` runs locally. `"cloud"` masks locally first, then sends the masked text to the euRedact service, which adds the model-only types (person names, organisations, job titles, diagnoses). See [Cloud tier](#cloud-tier). |
| `idempotency_key` | `None` | `mode="cloud"` only: sent as the request's `Idempotency-Key`, so the same key returns the same job instead of running and billing the document again. Generated per call when omitted. Raises `ValueError` in rules mode. See [What the service said](#what-the-service-said-resultcloud). |
| `context` | `None` | Share country evidence across the chunks of one document, so a chunk with no country signal of its own is still scored against the rest. Pass the same [`DocumentContext`](#documentcontext) to every chunk. Disables the cache. See [Chunked documents](#chunked-documents). |
| `chunk_offset` | `0` | Where this chunk starts in the whole document. Used only to rebase spans recorded in `context`; returned detections are always relative to `text`. |
| `cache` | `True` | Reuse the result for an identical input and configuration. Set `False` for one-off calls on sensitive text, or when timing the engine. |

**Not yet implemented**

| Parameter | Default | Description |
|---|---|---|
| `coref` | `False` | Reserved for pronoun/coreference resolution. Accepted and ignored in rules mode; raises in cloud mode. |
| `coref_model` | `"default"` | Reserved, with `coref`. |

#### `euredact.redact_batch()`

```python
euredact.redact_batch(
    texts: list[str],
    *,
    countries: list[str] | None = None,
    mode: str = "rules",
    referential_integrity: bool = False,
    tokenize: bool = False,
    allowlist: list[str] | None = None,
    detect_dates: bool = False,
    cache: bool = True,
) -> list[RedactResult]
```

Redact PII from multiple texts at once. More efficient than calling `redact()` in
a loop because country configs are loaded once. Returns results in the same order
as the input.

#### `euredact.aredact()`

```python
async euredact.aredact(
    text: str,
    **kwargs,
) -> RedactResult
```

Async version of `redact()`. Offloads CPU-bound work to a thread pool so it
doesn't block the event loop. Accepts the same keyword arguments.

#### `euredact.aredact_batch()`

```python
async euredact.aredact_batch(
    texts: list[str],
    *,
    max_concurrency: int = 4,
    **kwargs,
) -> list[RedactResult]
```

Async batch redaction with controlled concurrency. Processes texts concurrently in
a thread pool. `max_concurrency` limits parallel threads (default 4). Returns
results in input order.

#### `euredact.redact_iter()`

```python
euredact.redact_iter(
    texts: Iterator[str],
    **kwargs,
) -> Iterator[RedactResult]
```

Lazy iterator that yields results one at a time. Useful for processing large
datasets without loading all results into memory. Loads country configs once on
the first item.

#### `euredact.add_custom_pattern()`

```python
euredact.add_custom_pattern(name: str, pattern: str) -> None
```

Register a custom regex pattern. Matches are reported with `name` as the entity
type. See [Custom Patterns](#custom-patterns) below for details and examples.

#### `euredact.restore()`

```python
euredact.restore(text: str, tokens: Mapping[str, str]) -> str
```

Put the original values back into text that derives from a `tokenize=True`
result — typically an LLM's reply to the tokenized prompt. `tokens` is
`result.tokens`. See [Reversible tokenization](#reversible-tokenization).

#### `euredact.available_countries()`

```python
euredact.available_countries() -> list[str]
```

Returns a sorted list of supported ISO country codes (e.g. `["AT", "BE", "BG", ...]`).

### Instance-Based API (`EuRedact` Class)

For applications that need isolated instances (separate caches, separate custom
patterns), use the `EuRedact` class directly:

```python
from euredact import EuRedact

instance = EuRedact()

# Register custom patterns on this instance only
instance.add_custom_pattern("CASE_REF", r"CASE-\d{8}")

# Redact using this instance's configuration
result = instance.redact("See CASE-20260401 for details", countries=["NL", "BE"])
print(result.redacted_text)
# "See [CASE_REF] for details"
```

The `EuRedact` class exposes the same methods as the module-level API: `redact()`,
`redact_batch()`, `aredact()`, `aredact_batch()`, `redact_iter()`, `clear()` and
`add_custom_pattern()`.

```python
EuRedact(
    *,
    max_input_length: int = 10_485_760,
    allowlist: list[str] | None = None,
    allowlist_domains: list[str] | None = None,
    exclude_types: list[str] | None = None,
)
```

| Parameter | Default | Description |
|---|---|---|
| `max_input_length` | `10_485_760` (~10 MB) | Longest document `redact()` accepts, in characters. Above it, `ValueError` — split the input or raise the ceiling. |
| `allowlist` | `None` | Values never to redact, for every call on this instance: your own organisation's name, its own addresses. Merged with the per-call `allowlist`. See [Allowlist](#allowlist). |
| `allowlist_domains` | `None` | Domains never to redact, for every call on this instance. Merged with the per-call value. |
| `exclude_types` | `None` | Entity types never reported or masked, for every call on this instance. Merged with the per-call value. See [Excluding entity types](#excluding-entity-types). |

State held on the instance — the result cache, referential-integrity labels and
custom patterns — is per instance, which is what makes one instance per tenant
the right default. `clear()` releases the cache and the label mapping.

### Return Types

#### `DocumentContext`

```python
from euredact import DocumentContext

ctx = DocumentContext()
for offset, chunk in chunks:
    result = euredact.redact(chunk, context=ctx, chunk_offset=offset)
```

Shares country evidence across the chunks of one document, so a chunk carrying
no country signal of its own is still scored against what the rest of the
document showed. Pass the same object to every chunk, with `chunk_offset` set
to where the chunk starts. Passing a context disables the result cache, because
the result then depends on evidence from other chunks rather than on the text
alone.

| Method | Description |
|---|---|
| `add(evidence, chunk_offset=0)` | Record a chunk's evidence. `redact()` calls this for you. |
| `evidence()` | Every signal gathered so far, spans rebased onto the whole document. |

Not supported in cloud mode: the model has never seen a chunk boundary, so the
service rejects oversized input rather than splitting it.

#### `RedactResult`

Returned by `redact()` and all batch/async variants.

```python
@dataclass
class RedactResult:
    redacted_text: str          # The input text with PII replaced
    detections: list[Detection] # All PII spans found
    source: str = "rules"       # Detection backend ("rules")
    degraded: bool = False      # True if the engine fell back to a simpler mode

    # Country inference — see "Country-independent detection"
    inferred_countries: tuple[tuple[str, float], ...] = ()  # (country, confidence), strongest first
    evidence: tuple[CountryEvidence, ...] = ()              # every signal, with the span behind it
    detection_mode: str = "declared"                        # "declared" if countries= was passed,
                                                            # "inferred" otherwise
    tokens: dict[str, str] = {}                             # token -> original value; only with tokenize=True
    exempted: list[Exemption] = []                          # spans the allowlist or exclude_types kept, with the rule
    cloud: CloudInfo | None = None                          # mode="cloud" only: job id, model, cost,
                                                            # unplaced findings — see "What the service said"
```

#### `Detection`

A single PII span. Frozen dataclass (immutable, hashable).

```python
@dataclass(frozen=True)
class Detection:
    entity_type: EntityType | str  # PII category (EntityType enum or custom name)
    start: int                     # Start offset in the original text
    end: int                       # End offset (exclusive) in the original text
    text: str                      # The matched substring
    source: DetectionSource        # "rules" or "cloud"
    country: str | None            # ISO code of the matched country, or None for shared/custom patterns
    confidence: str | None = "high"  # "high" | "medium" | "low"; None for a cloud-model span — see below
    country_confidence: float = 0.0  # How strongly the document supports `country`, in [0, 1].
                                     # 0.0 means the attribution rests on a checksum alone.
    out_of_scope: bool = False       # True when attributed outside the declared `countries`.
                                     # Flagged, never dropped.
```

**`confidence`** describes how the *type* was arrived at. It never says anything
about whether the span is masked — every detection is, at every level.

| value | meaning |
|---|---|
| `"high"` | a pattern matched and, where one exists, its checksum passed |
| `"medium"` | the type comes from a label touching the span, because no pattern of that type claimed it — `Αρ. Ταυτότητας: 00892341` is a `NATIONAL_ID` although nothing can checksum it |
| `"low"` | a pattern matched, its checksum *failed*, and the document labels the span as that very type — `Rijksregisternummer: 85.03.19-284.73` is a national number with a bad check digit |
| `None` | a span the cloud model found (`source` is `cloud`). The model reports no confidence, so none is invented: a model span is not checksum-backed, and filtering on `"high"` keeps rules detections only *(rules-engine#108)* |

Filter on it when you need only checksum-backed detections:

```python
strict = [d for d in result.detections if d.confidence == "high"]
```

A `"low"` detection is the honest description of a mistyped, OCR'd or invented
identifier: the shape and the label agree, the check digit does not. Earlier
versions dropped these, which meant a redaction library printed in full an
identifier it had recognised and rejected.

#### `EntityType`

String enum with all supported PII categories:

```
BANK_ACCOUNT      BIC               CREDIT_CARD       PHONE
EMAIL             DOB               DATE_OF_DEATH     NATIONAL_ID
SSN               TAX_ID            PASSPORT          DRIVERS_LICENSE
RESIDENCE_PERMIT  LICENSE_PLATE     VIN               VAT
POSTAL_CODE       IP_ADDRESS        IPV6_ADDRESS      MAC_ADDRESS
HEALTH_INSURANCE  HEALTHCARE_PROVIDER  CHAMBER_OF_COMMERCE  IMEI
GPS_COORDINATES   UUID              SOCIAL_HANDLE     SECRET
INTERNAL_ID

Cloud tier only — the rule engine never emits these, because there is no shape
to match on. That is precisely why the model exists:

PERSON_NAME       ADDRESS           ORGANISATION_NAME JOB_TITLE
MEDICAL_CONDITION SENSITIVE_ATTRIBUTE  BIOMETRIC_REF  FINANCIAL_AMOUNT
QUASI_IDENTIFIER  URL               LOCATION

OTHER: the cloud client's fallback for a label the service returns and the
SDK does not know. Neither layer detects anything as OTHER.
```

`SECRET` is the one type for credentials, whichever layer finds them: the rules
engine's shaped keys and tokens, and the cloud model's prose passwords and
connection-string secrets, which the service labels `CREDENTIAL`. The SDK
reports those as `SECRET`, and `CREDENTIAL` remains only as a legacy alias of
it (rules-engine#111).

`INTERNAL_ID` — an employee, badge or customer number tied to a person — is
emitted **only** when an explicit label names it (`medarbejdernummer:`,
`Personalnummer:`, `Employee No:`, `Betriebsstättennr.`, `Badge`). There is no
pattern for one, because there is no shape for one: without the label, a digit
run is not distinguishable from any other. The type exists so that a labelled
employee number is filed correctly instead of being claimed by the phone
pattern.

#### What a label in front of a value decides

A label touching the left edge of a value decides what that value is called.
The label may be the abbreviation or the word it abbreviates — `BSN:` and
`Burgerservicenummer:` both reach `NATIONAL_ID` — and it may be the official
name in the document's own language:

| label | type |
|---|---|
| `Companies House Registration:`, `Company Registration Number:` | `CHAMBER_OF_COMMERCE` |
| `Sozialversicherungsnummer:` | `SSN` |
| `Passport No.:`, `Paspoortnummer:` | `PASSPORT` |
| `AGB-code:`, `LANR`, `GMC Number:` | `HEALTHCARE_PROVIDER` |
| `Medical Card No.:` | `HEALTH_INSURANCE` |
| `sort code`, `account number` | `BANK_ACCOUNT` |
| `TAN-activatiecode`, `activation code` | `SECRET` |

A label can also *rule a type out*. A four-digit run is not a postal code when
a founding or payment participle introduces it (`Opgericht in 2016`,
`Fondée en 2017`) or when it sits in a telephone parenthetical (`(toest. 3841)`,
`(ext. 2219)`). Postal codes that merely look like years are unaffected —
Antwerp's `2018` in `rustige ligging in 2018` is still a `POSTAL_CODE`.

A label never moves a span; it only decides the label. Which characters are
masked is unchanged either way.

For custom patterns registered via `add_custom_pattern()`, `entity_type` is a
plain string (e.g. `"EMPLOYEE_ID"`) rather than an `EntityType` enum member.

#### `DetectionSource`

String enum: `"rules"` or `"cloud"`.

## Batch processing and concurrency

Three entry points, and the difference between them is memory and ordering, not
detection. All three load the country configurations once instead of per
document, which is the whole reason to prefer them to a loop.

| Call | Shape | Use when |
|---|---|---|
| `redact_batch(texts)` | `list[str] -> list[RedactResult]` | you have a list and want it back in order |
| `aredact_batch(texts, max_concurrency=4)` | `list[str] -> list[RedactResult]`, awaited | you are in async code; work runs in a thread pool |
| `redact_iter(texts)` | `Iterator[str] -> Iterator[RedactResult]` | the input does not fit in memory |

```python
import asyncio
import euredact

docs = ["Mijn BSN is 111222333.", "IBAN NL91ABNA0417164300."]

# Synchronous, order preserved.
for result in euredact.redact_batch(docs, countries=["NL"]):
    print(result.redacted_text)

# Async, four at a time. The engine is CPU-bound, so this offloads to threads
# rather than pretending to be I/O.
results = asyncio.run(euredact.aredact_batch(docs, countries=["NL"], max_concurrency=8))

# Streaming: nothing accumulates.
with open("corpus.txt") as fh:
    for result in euredact.redact_iter(fh, countries=["NL"]):
        ...
```

`max_concurrency` defaults to 4. Raising it past your core count does not help —
the work is regex matching, not waiting.

### Reuse one instance

The module-level functions share a single hidden instance, which is usually what
you want. Construct your own when you need the configuration to be per-tenant:
custom patterns, an allowlist and the result cache all live on the instance.

```python
from euredact import EuRedact

engine = EuRedact(allowlist=["ACME Corporation"])
engine.add_custom_pattern("CASE_REF", r"CASE-\d{8}")

engine.redact_batch(
    ["ACME Corporation, ref CASE-20260401, BSN 111222333."], countries=["NL"]
)[0].redacted_text
# 'ACME Corporation, ref [CASE_REF], BSN [NATIONAL_ID].'
```

Reusing an instance is what makes the cache useful across documents. Instances
are safe to share across threads and across concurrent async tasks.

### Tokens and referential labels do not span a batch

`tokenize=True` mints tokens **per call**, so item 3 of a batch does not share a
token with item 1 even for the same value. If two documents must agree on a
label, redact them as one text, or use `referential_integrity=True`, whose
`TYPE_n` numbering is per instance rather than per call.

## Cloud tier

> **Status: private alpha.** The cloud tier is in closed testing — it is **not**
> in public beta and is not generally available. Keys are issued to alpha
> participants only, and the request/response surface may still change between
> releases. The rules engine below is unaffected and is the supported path:
> `mode="rules"` is the default and carries no alpha caveat.

The rule engine catches what has a shape: IBANs, national IDs, phone numbers,
anything with a checksum. It cannot catch what does not — a person's name, an
employer, a diagnosis, a job title. euRedact Cloud, the cloud tier, adds a fine-tuned model
asked only *what did the rules miss?*

```bash
pip install 'euredact[cloud]'
```

```python
import euredact

euredact.configure(api_key="erk_...")          # or set EUREDACT_API_KEY
result = euredact.redact(text, countries=["BE"], mode="cloud")

result.source                                   # "cloud"
[(d.entity_type, d.text) for d in result.detections]
# [(<EntityType.PERSON_NAME>, 'Bas Verhoeven'), (<EntityType.PHONE>, '+32 ...')]
```

`mode="cloud"` **raises** `NotConfiguredError` when the tier is not configured.
It never falls back to rules-only output: a caller who believes names and
diagnoses were checked, and ships a document that only had its phone numbers
masked, is the one failure this library must not have.

`aredact()` takes the same arguments from async code:

```python
result = await euredact.aredact(text, countries=["BE"], mode="cloud")
```

`CloudClient` and `AsyncCloudClient` in `euredact.cloud` are the transport
underneath. They send exactly the text they are given and return spans relative
to it: the local masking happens in `redact()`, not in the client. Use them
directly only with text you have already masked.

Retries carry an `Idempotency-Key`, so a retry after a timeout cannot bill
twice. `Retry-After` is obeyed. A document that outlives the service's sync
window is polled transparently — callers never write that branch. Oversized
input raises `TooLargeError` (413): the service refuses it rather than
chunking, because the model has never seen a chunk boundary.

### What leaves your machine

Be precise about this, because it is the question a security review asks first.

In `mode="cloud"` the rules engine runs **on your machine first**, and only
the text it leaves behind is sent to the service over TLS. Every value it finds
is already a `[TYPE]` placeholder by the time the request is built:

```
your text      IBAN NL91 ABNA 0417 1643 00 belongs to Nick Bols, tel +31 6 12345678
                 │  rules engine, on your machine
what is sent   IBAN [BANK_ACCOUNT] belongs to Nick Bols, tel [PHONE]
                 │  TLS ──▶ service (model) ──▶ spans relative to the masked text
what you get   IBAN [BANK_ACCOUNT] belongs to [PERSON_NAME], tel [PHONE]
```

The request body is that one `text` field plus the country. No list of types,
no offsets and no original values travel beside it. The service answers with
spans that index the masked text, and the SDK maps them back onto your
original, so `detections` always index the document you passed in.

**This is minimisation, not an exemption.** Three things still leave:

- what the model is there to find — names, employers, job titles, diagnoses;
- anything the rules engine missed, such as an identifier in a format it does
  not know;
- the surrounding prose.

So the service still processes personal data on your behalf, and a DPIA should
say so. What it no longer receives is every IBAN, national ID, phone number and
email address the local engine could place.

Three things behave differently here than in rules mode:

- `detect_dates` has no effect. The local pass always runs with dates on,
  because that is the rules output the model was trained against. It can only
  cause more to be masked, never less.
- `tokenize` and the allowlists are applied locally, after the response, and do
  not change what is sent: placeholders on the wire are always `[TYPE]`, and an
  allowlisted value is masked in the request and put back in your result.
- [Custom patterns](#custom-patterns) run in the local pass, so a value only
  you know how to recognise is masked before the request too.

Every span the service returns is checked against the text that was sent. One
that does not match raises `CloudError` rather than being placed by guesswork.

### Keeping identifiers local

`mode="cloud"` does this for you with our model. The same pattern works with
any other model — your own, or a general-purpose LLM — and that is what
`tokenize=True` is for:

```python
import euredact

prompt = "Stuur een mail naar Bas Verhoeven (bas@example.nl) over NL91ABNA0417164300."

# 1. Mask locally. Nothing has left the process.
local = euredact.redact(prompt, countries=["NL"], tokenize=True)
local.redacted_text
# 'Stuur een mail naar Bas Verhoeven (EMAIL_S5SH) over BANK_ACCOUNT_5XXR.'
local.tokens
# {'BANK_ACCOUNT_5XXR': 'NL91ABNA0417164300', 'EMAIL_S5SH': 'bas@example.nl'}

# 2. Send only the masked text to any model, ours or someone else's.
answer = call_your_model(local.redacted_text)

# 3. Put the real values back locally.
euredact.restore(answer, local.tokens)
```

The token suffixes are random per call, so yours will differ.

**Read that output carefully: `Bas Verhoeven` is still there.** That is the
whole trade-off and it is why the cloud tier exists. The local engine masks what
has a shape — the IBAN and the address — and cannot mask a name, an employer or
a diagnosis, because it cannot find them. So this pattern keeps every identifier
the engine detects inside your process and sends the prose, names included.
`mode="cloud"` sends that same masked prose to a model trained to find the
names in it.

Either way the prose leaves. If names must not leave your infrastructure
either, no hosted model can help; run one yourself against
`local.redacted_text`.


### What the service said: `result.cloud`

A cloud result carries `cloud`, what the service reported beside the spans;
it is `None` on a rules-only result.

```python
result = euredact.redact(text, countries=["BE"], mode="cloud",
                         idempotency_key="invoice-2291-v1")
result.cloud.job_id         # 'job-7f3a…': support, Jobs.retrieve, replays
result.cloud.model_version  # the model that answered; None when only rules ran
for item in result.cloud.unlocated:
    print(item.entity_type, item.text)   # found, but not placed: nothing masked for it
usage = result.cloud.usage
if usage:
    print(usage.tokens, usage.billing_rate, usage.credits)
    for factor in usage.factors:
        print(factor.code, factor.types or "", factor.detail)
```

- **`idempotency_key`** is sent as the request's `Idempotency-Key`: the same
  key returns the same job instead of running and billing the document again.
  One is generated per call when omitted. It applies to `mode="cloud"` only.
- **`usage`** is the total the request was charged and the reasons for it,
  never a cost per step. `credits` is `round(tokens × billing_rate)`, the
  formula the service debits by. The factor codes are `rules_only`,
  `document_length`, `instructions`, `dense_document`,
  `special_category_check` (with the `types` that triggered it),
  `long_document` and `batch_rate`; a code added later still arrives, as its
  string. `None` when the service does not report it or reported it
  malformed: a cost report never fails a redaction.
- **`unlocated`** lists what the model found but could not match to a
  position in the text it was sent. Nothing was masked for these.
- **Batch documents** carry `model_version` and `unlocated`; no `job_id` or
  `usage`, which belong to the batch (`Batch.tokens`, `Batch.credits_charged`,
  `Batch.billing_rate`).

All four types are frozen dataclasses, exported from `euredact`:

```python
@dataclass(frozen=True)
class CloudInfo:
    job_id: str | None = None          # None for a batch document
    model_version: str | None = None   # None when only rules ran
    usage: Usage | None = None         # None from older services, batch documents, or a malformed block
    unlocated: tuple[Unlocated, ...] = ()

@dataclass(frozen=True)
class Unlocated:
    text: str                          # what the model reported
    entity_type: EntityType | str      # e.g. EntityType.PERSON_NAME; unknown types stay strings

@dataclass(frozen=True)
class Usage:
    tokens: int                        # every model call the document triggered
    billing_rate: float                # 1.0, or e.g. 0.75 for a batch
    credits: int                       # round(tokens * billing_rate): what was debited
    factors: tuple[UsageFactor, ...] = ()

@dataclass(frozen=True)
class UsageFactor:
    code: str                          # see the list above; new codes arrive as strings
    detail: str = ""                   # one sentence from the service
    types: tuple[str, ...] | None = None  # special_category_check only
```

**Retrying safely.** Pass your own `idempotency_key` when your code may send the
same document twice: after a timeout of your own, from a retried queue message
or a restarted worker. Derive it from something stable (an invoice number plus
a version), not from a random value per attempt. A replay returns the same
`result.cloud.job_id` and is not billed again.

### Account, past jobs and the batch list

Everything an API key may do on the service is reachable from the SDK. What
needs a signed-in person (logging in, minting keys, settings, members, terms)
is not, because a key cannot do it.

| Call | Service endpoint | Returns |
|---|---|---|
| `Account().summary()` | `GET /v1/account` | `AccountSummary` |
| `Account().credits()` | `GET /v1/account/credits` | `Credits` |
| `Account().credit_history(limit=50)` | `GET /v1/account/credits/history` | `list[CreditEntry]`, newest first; `limit` 1–500 |
| `Account().usage(days=30)` | `GET /v1/account/usage` | `list[UsageDay]` |
| `Account().usage_by_key(days=30)` | `GET /v1/account/usage/by-key` | `list[KeyUsage]` |
| `Account().keys()` | `GET /v1/account/keys` | `list[ApiKey]` |
| `Account().revoke_key(key_id)` | `POST /v1/account/keys/{id}/revoke` | `KeyRevocation` |
| `Jobs().retrieve(job_id)` | `GET /v1/jobs/{id}` | `Job` |
| `Batches().list(limit=20)` | `GET /v1/batches` | `list[Batch]`, newest first; `limit` 1–100 |

```python
from euredact.cloud import Account, Batches, Jobs

with Account() as account:
    summary = account.summary()
    if summary.quota_remaining is not None and summary.quota_remaining < 100:
        print(f"{summary.quota_remaining} documents left today")
    print(account.credits().balance)
    for day in account.usage(days=7):
        print(day.day, day.documents, day.tokens, day.failures)
    for key in account.keys():
        print(key.id, key.name, key.active, key.last_used_at)

with Jobs() as jobs:
    job = jobs.retrieve("job-7f3a…")
    if job.result is not None:
        print(job.result.redacted_text)

with Batches() as batches:
    for batch in batches.list(limit=20):
        print(batch.id, batch.status, batch.documents, batch.credits_charged)
```

#### Creating the clients

`Account`, `Jobs`, `Batches`, `CloudClient` and `AsyncCloudClient` all read the
configuration `euredact.configure()` set (or `EUREDACT_API_KEY`). Each also
accepts:

- **`config=`**, a `CloudConfig` of its own, for a second key or another
  endpoint: `Account(config=euredact.cloud.CloudConfig(api_key="erk_…"))`.
- **`client=`**, an `httpx.Client` you own: for a proxy, custom TLS, or a mock
  transport in your tests. A client you pass in is not closed by the SDK.

Use them as context managers, or call `close()`, so the connection pool is
released. One instance can make any number of calls.

**From asyncio code** (FastAPI and the like): `Account` and `Jobs` are
synchronous. Each call is one short request, so run it on a thread:

```python
summary = await asyncio.to_thread(account.summary)
job = await asyncio.to_thread(jobs.retrieve, job_id)
```

#### What each call returns

Every value is a frozen dataclass. Each has `raw`, the service's JSON as
received, so a field the SDK does not type yet is still reachable; and every
field is `None` when the service left it out or sent the wrong type, rather
than the call failing.

```python
class AccountSummary:
    tenant_id: str | None; tenant_name: str | None; active: bool | None
    member_since: str | None          # ISO date
    role: str | None
    retention_mode: str | None        # "ttl", or "none": text discarded on delivery
    payload_ttl_hours: int | None     # 0 under "none"
    documents_today: int | None; documents_this_month: int | None
    tokens_this_month: int | None; failures_24h: int | None
    daily_quota: int | None           # None: no daily quota on this account
    quota_remaining: int | None
    active_keys: int | None
    batches_available: bool | None    # batches need retention ("ttl")

class Credits:
    balance: int | None; granted: int | None; spent: int | None
    unit: str | None                  # "tokens"

class CreditEntry:                    # rolled up per minute, reason and direction
    at: str | None                    # ISO timestamp
    delta: int | None                 # negative for a debit
    reason: str | None; count: int | None
    job_id: str | None; source_ref: str | None

class UsageDay:
    day: str | None                   # ISO date
    documents: int | None; tokens: int | None
    failures: int | None; rules_only: int | None   # None inside a KeyUsage series

class KeyUsage:                       # the busiest keys one by one; the rest
    key_id: int | None                # rolled into one entry with key_id None
    name: str | None
    documents: int | None; tokens: int | None; failures: int | None
    series: tuple[UsageDay, ...]

class ApiKey:                         # never the secret: shown once, when minted
    id: int | None; name: str | None; created_by: str | None
    created_at: str | None; last_used_at: str | None; revoked_at: str | None
    active: bool | None

class KeyRevocation:
    id: int | None
    keys_remaining: int | None
    locked_out: bool | None           # True: no active key left; mint one in the console
    self_revoked: bool | None         # True: you revoked the key this client uses

class Job:
    job_id: str
    status: str                       # "queued", "running" or "succeeded"
    created_at: str | None
    result: RedactResult | None       # set once succeeded
```

`Batch` gained four fields beside `id`, `status`, `counts` and the timestamps:
`documents` (int), `tokens` (`{"prompt": n, "completion": n}`),
`credits_charged` (int) and `billing_rate` (float, e.g. `0.75`).

#### Retrieving a job

`redact(mode="cloud")` already waits for its job, polling past the service's
synchronous window, so most code never needs `Jobs`. It is for:

- **Support and audits:** look up `result.cloud.job_id` later.
- **A caller that stopped waiting:** a job the client gave up polling on
  (`poll_timeout_s`) may still finish; retrieve it by id.

A failed or rejected job raises (`CloudError` with status 502, `TooLargeError`)
rather than returning a status. The result is over the text the service
received: under `mode="cloud"` that is the locally masked text, so its offsets
and `redacted_text` are not your original's. Results are kept for the
account's retention period (`AccountSummary.payload_ttl_hours`), and not at all
under `retention_mode == "none"`; after that, `ResultExpiredError`.

#### Revoking a key

`revoke_key` is the one write an API key may make, for a key that leaked. It
is sent once and never retried. Revoking the key the client itself uses
succeeds and reports `self_revoked=True`; every later call with it fails with
status 401. When `locked_out` is `True`, no active key is left and only a
person in the console can mint one.

#### Handling errors

All errors derive from `euredact.cloud.CloudError`, which carries `status`
(the HTTP status, or `None` for a network failure) and `detail` (the service's
JSON detail). Catch the specific ones first: `RateLimitedError` is a subclass
of `QuotaExceededError`.

```python
from euredact.cloud import (
    CloudError, NotFoundError, QuotaExceededError, RateLimitedError,
    ResultExpiredError, TooLargeError,
)

try:
    result = euredact.redact(text, countries=["BE"], mode="cloud",
                             idempotency_key=f"invoice-{invoice.id}-v{invoice.version}")
except RateLimitedError:
    ...  # rate limit still hit after every retry: back off, then retry the same key
except QuotaExceededError as exc:
    ...  # daily quota used up (exc.detail["used"], exc.detail["limit"]): wait for tomorrow
except TooLargeError:
    ...  # over the input cap: split the document and send the parts
except CloudError as exc:
    ...  # anything else; exc.status says what
```

| Error | Status | When | Retried by the SDK |
|---|---|---|---|
| `NotConfiguredError` | — | no API key configured, or `httpx` missing | no |
| `CloudError` | 401 | the key is wrong or revoked | no |
| `CloudError` | 403 | the service refused the call. Account calls are accepted with an API key while the service does not require a browser `Origin`; if that changes they answer 403 | no |
| `CloudError` | 403 | `Batches.create` on a workspace without retention: `detail["code"] == "retention_required"` (see [Batches](#batches)) | no |
| `NotFoundError` | 404 | no such job, batch or key — or another account's; also every batch call, `list()` included, while batches are not enabled on the service (`batches_not_enabled`) | no |
| `CloudError` | 409 | `revoke_key` on a key already revoked | no |
| `ResultExpiredError` | 410 | the job's result is past retention | no |
| `TooLargeError` | 413 | the document is over the input cap; also a rejected job | no |
| `QuotaExceededError` | 429 | the daily document quota is used up | **no**: it will not clear before the day does |
| `RateLimitedError` | 429 | the edge's rate limit, still hit after every retry | yes, `max_retries` times with backoff |
| `CloudError` | 5xx, `None` | the service or the network failed | yes for reads and `redact`; never for `revoke_key` |

**Availability:** like the rest of the cloud tier, these calls are in private
alpha, and the response shapes may still change between releases. The batch
calls, `Batches.list()` included, answer `NotFoundError` until batches are
enabled on the service (see [Batches](#batches)).

### `euredact.configure()`

```python
euredact.configure(
    api_key: str | None = None,
    *,
    base_url: str | None = None,
    timeout_s: float = 90.0,
    poll_timeout_s: float = 900.0,
    max_retries: int = 3,
    headers: dict[str, str] | None = None,
) -> CloudConfig
```

| Parameter | Default | Environment variable | Description |
|---|---|---|---|
| `api_key` | — | `EUREDACT_API_KEY` | Your alpha key. Required; `configure()` raises without one. |
| `base_url` | `https://api.euredact.dev` | `EUREDACT_BASE_URL` | Service endpoint. Must be `https://`; plain `http://` is refused except to `localhost`, `127.0.0.1` or `::1`, so the key and the text never travel unencrypted. |
| `timeout_s` | `90.0` | — | Per-request timeout. |
| `poll_timeout_s` | `900.0` | — | Ceiling for polling a document that outlives the synchronous window. |
| `max_retries` | `3` | — | Retries for `5xx`, timeouts and the edge's rate-limit `429`. Each carries an `Idempotency-Key`, so a retry after a timeout cannot bill twice, and `Retry-After` is obeyed. A `429` for the daily quota is not retried: it raises `QuotaExceededError` at once. |
| `headers` | `None` | — | Extra headers sent with every request. |

`configure()` reads the two environment variables itself, so it must still be
called — but with no arguments if the environment is set. Errors are
`NotConfiguredError`; `QuotaExceededError` (429, the daily quota) and its
subclass `RateLimitedError` (429 from the rate limit, after retries);
`TooLargeError` (413, the service refuses oversized input rather than chunking
it); `NotFoundError` (404); `ResultExpiredError` (410); and `CloudError` for
the rest, with `status` and `detail`.

Options the service cannot honour raise rather than being ignored: multiple
`countries`, `country_hint`, `context`/`chunk_offset`, `referential_integrity`
and `coref`. `tokenize`, `allowlist` and `allowlist_domains` are honoured: the SDK applies them to
the spans the service returns and rebuilds the text from those.

### Batches

> **Availability:** private alpha, like the rest of the cloud tier. Batches
> need **retention** on the workspace (see below); while batches are not
> enabled on the service, every batch call answers `NotFoundError`
> (`batches_not_enabled`).

A batch hands up to 5,000 documents to the cloud tier and collects the answers
within 24 hours, at a lower price and without your plan's concurrency limit.
It keeps the same promise as `mode="cloud"`: the rules engine masks each
document **on your machine**, and only the masked text is uploaded.

Because the answers arrive hours later, the SDK keeps what it needs to map
them back in a **local batch file**, `~/.euredact/batches/<batch_id>.json`
(directory `0700`, file `0600`). It holds each original, its local detections
and a SHA-256 of the masked text that was sent. None of it is uploaded.

```python
from euredact.cloud import Batches

batches = Batches()                     # uses euredact.configure(...)
batch = batches.create([
    {"custom_id": "invoice-2291", "text": invoice_text, "countries": ["BE"]},
    {"custom_id": "memo-17",      "text": memo_text,    "countries": ["NL"]},
])

# Later, possibly in another process on the same machine or share:
outcome = batches.results(batch.id)
if outcome.status == "resolved":
    for custom_id, doc in outcome.documents.items():
        if doc.ok:
            print(custom_id, doc.result.redacted_text)
        else:
            print(custom_id, "failed:", doc.error, doc.message)
```

**Batches need retention.** The service holds each document until it is
processed and each result until you fetch it, so a workspace that erases
documents on delivery (`retention_mode == "none"`, or 0 hours of retention)
cannot create one. `create()` then raises `CloudError` with status 403 and
`detail["code"] == "retention_required"`, before anything is uploaded or
written locally. The workspace owner turns retention on in the console, on the
**Account** tab. To check first:

```python
from euredact.cloud import Account, CloudError

if not Account().summary().batches_available:
    ...  # no retention on this workspace: send the documents with redact(mode="cloud")

try:
    batch = batches.create(documents)
except CloudError as exc:
    if exc.status == 403 and exc.detail.get("code") == "retention_required":
        ...  # ask the workspace owner to turn retention on
    raise
```

- **`results()`** returns `not_ended` while the batch runs, and changes
  nothing. Once the batch has ended it maps every document, wipes the originals
  and leaves a text-free receipt; a second call returns `already_resolved`. A
  batch is never mapped twice.
- **Integrity.** Each document's masked text is rebuilt from the local file
  and must match the hash the gateway reports before a span is placed. An
  edited file, a missing entry or no file at all (another machine, a purged
  batch) is an error for that document (`local_mismatch`,
  `missing_local_entry`) — never the masked text passed off as a result.
- **`pending()`** lists batches whose local file still waits to be mapped, so
  a restarted worker can resume. `retrieve(id)` and `cancel(id)` talk to the
  gateway; cancelling keeps the file until the finished documents are mapped.
- **Clean-up.** Receipts are deleted, and unmapped files expired and wiped,
  once the gateway's results expire; `purge(id)` deletes a file by hand.
- **Where the file lives.** `Batches(batch_dir=...)` for a directory several
  workers share, or `store=` for any object with `read`, `write`, `delete` and
  `list` (a key-value store, for example). `cipher=` takes an object with
  `encrypt(bytes)` and `decrypt(bytes)` to encrypt the file at rest with a key
  you hold; the SDK has no cryptography dependency and does not pick a scheme.
- **In an asyncio application** (FastAPI, say), call the methods through
  `asyncio.to_thread`, for example `batch = await asyncio.to_thread(batches.create,
  docs)`. Each call is short: it returns when its own upload, status request or
  download is done, never when the batch is. `create()` does the local masking
  for every document, though, so running it on a thread also keeps that work
  off the event loop.
- **Limits** checked before anything is masked or uploaded: at most 5,000
  documents, unique `custom_id`s of 1-64 characters from `A-Z a-z 0-9 _ . : -`
  (the gateway's rule, `euredact.cloud.batches.CUSTOM_ID`), one country per
  document, and a 128 MB body. The 5,000-token limit per document is **not** checked by the
  SDK: it is counted by the model's tokenizer on the gateway, which the SDK does
  not have, and no character count decides it reliably. A document over it
  comes back as an error with code `too_long`, whose message says so; split
  it and submit the parts.

## `NAME` is now `PERSON_NAME`

The canonical type name is `PERSON_NAME`; `NAME` is a legacy alias, exactly as
`IBAN` aliases `BANK_ACCOUNT`. The placeholder written into redacted text is
`[PERSON_NAME]`.

Nothing could have depended on the old value: the type is cloud-only and the
cloud tier was stubbed until this release, so it was never emitted. Code
matching the *string* `"NAME"` should be updated; `LEGACY_TYPE_ALIASES`
publishes the mapping, and `STREET_ADDRESS` → `ADDRESS` and
`NATIONALITY_ETHNICITY` → `SENSITIVE_ATTRIBUTE` are recognised the same way.

## What `countries` actually controls

This is the parameter most often misread, and reading it wrongly leads to
under-redaction in exactly the case you care about — a foreign identifier in a
domestic document. So, precisely:

> `countries` decides how a detection is **attributed and scored**.
> It does **not** decide what gets **found**.

Country-specific patterns for every supported country run on every document,
whatever you declare. Declaring `countries=["NL"]` does not switch the Belgian
patterns off; it says "this document is Dutch", and anything Belgian that turns
up is still detected and still masked — it is just flagged as sitting outside
what you declared.

The same Belgian national number, under four different calls:

```python
import euredact

text = "Rijksregisternummer 85.07.30-033.61 van onze klant."

for kwargs in ({"countries": ["NL"]},
               {"countries": ["BE"]},
               {},
               {"countries": ["NL"], "country_hint": ["BE"]}):
    result = euredact.redact(text, **kwargs)
    detection = result.detections[0]
    print(result.redacted_text, "|", result.detection_mode,
          "| country:", detection.country,
          "| country_confidence:", round(detection.country_confidence or 0, 2),
          "| out_of_scope:", detection.out_of_scope)
```

```
Rijksregisternummer [NATIONAL_ID] van onze klant. | declared | country: BE | country_confidence: 0 | out_of_scope: True
Rijksregisternummer [NATIONAL_ID] van onze klant. | declared | country: BE | country_confidence: 0.88 | out_of_scope: False
Rijksregisternummer [NATIONAL_ID] van onze klant. | inferred | country: BE | country_confidence: 0 | out_of_scope: False
Rijksregisternummer [NATIONAL_ID] van onze klant. | declared | country: BE | country_confidence: 0.88 | out_of_scope: True
```

**The masked output is byte-identical in all four.** What moved is the metadata.

| You pass | Effect on what is found | Effect on labelling |
|---|---|---|
| `countries=["NL"]` | None — every country's patterns still run | Attributed normally; a non-Dutch hit gets `out_of_scope=True` and `country_confidence=0.0` |
| `countries=["BE"]` | None | The Belgian hit is corroborated: `country_confidence=0.88`, `out_of_scope=False` |
| `countries=None` | None | `detection_mode="inferred"`; the country is inferred from the document and nothing is out of scope, because nothing was declared |
| `country_hint=["BE"]` | None | A *prior*: it corroborates Belgian attribution **without** widening or narrowing scope, so `out_of_scope` still reflects `countries` alone |

### How to use each one

- **`countries=[...]`** — you know where the document is from. Use it. It
  improves attribution and makes `out_of_scope` meaningful, and it is the
  cheapest path because country inference is skipped.
- **`countries=None`** — you do not know. The engine infers from the document
  and reports `inferred_countries` with confidences.
- **`country_hint=[...]`** — you have a weak signal (the customer's billing
  country, the mailbox a document arrived in) that should resolve ambiguity
  without declaring scope. Added in 0.3.2 precisely so that a hint could not be
  mistaken for a scope restriction.

### Why it works this way

Until 0.3.2, `countries` *did* gate detection, and 0.3.3 fixed the remaining
case where it "could change which spans were found, not just how they were
labelled". Both were changed deliberately: a redaction library that hides a
Belgian national number because you told it the document was Dutch has failed
at the only job it has. Scoping is a reporting concern; masking is not.

The practical consequence: **do not use `countries` as a filter.** If you only
want Dutch detections in your output, filter on
`detection.country == "NL"` or on `detection.out_of_scope` after the call —
the value was still masked in the text either way.

## Country codes

`countries=[...]` accepts **ISO 3166-1 alpha-2** codes. The two EU/VAT
spellings are accepted as equivalents:

| ISO 3166-1 | EU/VAT | |
|---|---|---|
| `GB` | `UK` | United Kingdom |
| `GR` | `EL` | Greece |

Codes are case-insensitive and whitespace-tolerant. An **unrecognised** code
does not raise — it emits an `UnknownCountryWarning` and detection continues
with the shared, country-independent patterns:

```python
euredact.redact(text, countries=["ZZ"])
# UnknownCountryWarning: Unknown country code: 'ZZ'. Continuing with shared
# country-independent patterns only (email, IBAN, international phone, ...).
```

This is deliberate. Raising on an unknown locale invites callers to wrap the
call in `try/except` and skip redaction entirely — failing open, with
unredacted PII in the output.

## Country-independent detection

**`countries=[...]` never gates detection.** Every pattern runs on every
document, whatever you pass. The country you declare decides *how a match is
labelled*, never *whether it is found*.

This is the engine's central invariant, enforced by
`tests/test_invariant_generation.py`: no value of `countries=` may change which
spans are detected. A wrong or missing country cannot cause a miss — silent
recall loss is invisible in testing and surfaces in a breach report, whereas a
false positive is recoverable.

It was not always so. `countries=["BE"]` used to make a valid Dutch BSN vanish
entirely, because the Dutch patterns were never run:

```python
euredact.redact("Werknemer met BSN 111222333", countries=["BE"])
# before: 'Werknemer met BSN 111222333'   <- leaked
# now:    'Werknemer met BSN [NATIONAL_ID]'
```

Entities found outside the countries you declared are **flagged, not dropped**:

```python
det = euredact.redact("BSN 111222333", countries=["BE"]).detections[0]
det.out_of_scope   # True — detected, masked, and marked as outside your scope
```

So a Belgian IBAN in a document processed with `countries=["AT"]` is still
detected — cross-border traffic (a foreign invoice in a local file, an employee
paid to a foreign account) does not leak:

```python
euredact.redact("Rekening: BE68 5390 0754 7034", countries=["AT"])
# -> 'Rekening: [BANK_ACCOUNT]'
```

### Which country a value belongs to

Because every pattern runs, the same digits often match several countries'
schemes. Of the national-ID values in our corpus that pass any country's checksum,
34.7% pass more than one country's (32,827 of 94,528), so the digits alone
cannot decide it — the *document* does.

The engine infers the document's countries from entities that carry their
country in the string, then uses that to resolve the ambiguity:

```python
r = euredact.redact("Bereikbaar op telefoon 0612345678, mail jan@test.nl")
r.inferred_countries          # (('NL', 0.98),)
r.detections[0].entity_type   # PHONE

r = euredact.redact("Kontakt: 0612345678, e-mail jens@test.dk")
r.inferred_countries          # (('DK', 0.98),)
r.detections[0].entity_type   # NATIONAL_ID
```

Identical digits, different answer — `0612345678` is both a valid Dutch mobile
number and a valid Danish CPR. Only the surrounding document distinguishes them.

Every inference is auditable: `result.evidence` lists each signal, its weight,
and the span that produced it.

| Signal | Weight (log-odds) | Measured reliability |
|---|---:|---|
| `e164_prefix` | 4.00 (capped) | 41,402 / 41,402 |
| `bic_country` | 4.00 (capped) | 2,588 / 2,588 |
| `email_tld` | 4.00 (capped) | 97,865 / 98,949 |
| `vat_prefix` | 2.84 | 19,022 / 20,136 |
| `iban_prefix` | 1.94 | 110,572 / 126,428 |

Weights are derived from the corpus, not chosen by hand. The IBAN prefix being
weakest is real and worth knowing: a Belgian IBAN in a Dutch invoice is
ordinary, so an account's country is only weak evidence about the document's.

Confidences are per-country and do **not** sum to 1 — document countries are
not mutually exclusive. A Belgian supplier invoicing a German customer is
genuinely both.

International phone numbers are matched by a generic E.164 pattern (`+`
followed by 8-15 digits, any grouping or separators, including `(0)` trunk
prefixes) that runs alongside the per-country patterns.

## BIC detection

BIC is the only bank identifier in the engine with **no check digit** — IBAN
has mod-97, VAT has country-specific checksums. ISO 9362 structure alone
cannot decide a match, because characters 5-6 of ordinary uppercase words are
frequently valid ISO 3166 country codes (`DRINGEND` → `GE`, `HOSPITAL` →
`IT`). Detection is therefore gated:

| Stage | Condition | Result |
|---|---|---|
| Gate 0 | the token also occurs as an ordinary lowercase word in the same document | never emitted |
| Tier 1 | registry hit on the BIC6 institution+country prefix (your registry, or the bundled seed of major banks) | emitted |
| Gate 2 | heading / shouted-word shape | never emitted |
| Tier 2 | `BIC`/`SWIFT` keyword, an IBAN, or a bank block in the enclosing line, record or paragraph | emitted — but a **letters-only** code (`BETALING`, `HOFFMANN`), unless the bundled GLEIF mapping knows the institution, also needs a `BIC`/`SWIFT` label touching it |
| — | none of the above | never emitted |

The context window is the enclosing **line, record or paragraph**, not a
character count — a banking cue often sits several fields away in the same
CSV row.

### Supplying your own BIC registry

The package bundles about **10,400 BIC6 prefixes** for the 31 supported countries,
derived from the [GLEIF BIC-to-LEI mapping](https://www.gleif.org/en/lei-data/lei-mapping/download-bic-to-lei-relationship-files),
which SWIFT develops and licenses for redistribution under the BIC/LEI Mapping
Table License Agreement; its required notice is in `NOTICE`. They tell a real
bank from an ordinary word next to an IBAN, but do not by themselves license a
code in bare prose: with that many real prefixes, some words begin with one
(`DERNIERS` → `DERN`+`IE`, an Irish institution). `scripts/refresh_bic_registry.py`
regenerates the list; GLEIF publishes the mapping monthly.

The complete SWIFTRef BIC Directory is a commercial product and is not
bundled. Deployments holding it install it at startup, where it works like the
seed — a hit is emitted without context:

```python
import euredact

# A path to a newline-delimited file of BICs...
euredact.set_bic_registry("/etc/euredact/swiftref-bics.txt")

# ...an iterable...
euredact.set_bic_registry({"ABNANL2A", "INGBNL2A", "BBRUBE"})

# ...or any membership callable.
euredact.set_bic_registry(lambda bic: bic in my_directory)

# Remove it again:
euredact.set_bic_registry(None)
```

Entries may be full BIC8/BIC11 codes or bare BIC6 prefixes; both are matched,
case-insensitively and ignoring spaces.

The registry is mostly an **accept** signal. A code missing from it falls
through to the context gate and is still detected when banking context is
present — except a letters-only code, the only shape an ordinary word can take,
which then also needs a `BIC`/`SWIFT` label touching it. Context alone used to
admit those, and surnames and words beside an IBAN were masked as `[BIC]`
(rules-engine#57).

## Custom Patterns

Register domain-specific PII patterns at runtime. Custom patterns are detected
alongside built-in patterns and participate in the same deduplication pipeline.

```python
import euredact

# Register patterns
euredact.add_custom_pattern("EMPLOYEE_ID", r"EMP-\d{6}")
euredact.add_custom_pattern("CASE_REF", r"CASE-\d{8}")

# They are detected alongside built-in PII
result = euredact.redact(
    "Employee EMP-123456, email jan@example.com, ref CASE-20260401"
)
print(result.redacted_text)
# "Employee [EMPLOYEE_ID], email [EMAIL], ref [CASE_REF]"

# Check detections
for d in result.detections:
    print(f"  {d.entity_type}: {d.text}")
# EMPLOYEE_ID: EMP-123456
# EMAIL: jan@example.com
# CASE_REF: CASE-20260401
```

### How Custom Patterns Work

- `name` becomes the entity type reported in detections and used in replacement
  tags (e.g. `[EMPLOYEE_ID]`)
- `pattern` is a Python regular expression (same syntax as `re` module)
- Custom patterns are always active regardless of the `countries` parameter
- Custom patterns have no validator (they are purely regex-based)
- In overlap resolution, custom patterns have higher priority than built-in
  regex-only patterns but lower priority than built-in patterns with a passing
  checksum validator

### Instance Isolation

Custom patterns registered on the module-level function apply to the shared
singleton. For isolated pattern registrations, use separate `EuRedact` instances:

```python
from euredact import EuRedact

# Instance A detects employee IDs
a = EuRedact()
a.add_custom_pattern("EMPLOYEE_ID", r"EMP-\d{6}")

# Instance B detects case references
b = EuRedact()
b.add_custom_pattern("CASE_REF", r"CASE-\d{8}")

# Each instance only detects its own custom patterns
result_a = a.redact("EMP-123456 CASE-20260401")
result_b = b.redact("EMP-123456 CASE-20260401")
```

## Secret and API Key Detection

EuRedact includes built-in detection for API keys, tokens, and passwords. This is
always active -- no configuration required.

### Known-Prefix Patterns

The following token formats are detected with high confidence based on their
distinctive prefixes:

| Pattern | Description |
|---|---|
| `AKIA...` | AWS Access Key ID |
| `ghp_`, `gho_`, `ghs_`, `github_pat_` | GitHub tokens (PAT, OAuth, app, server) |
| `sk_live_`, `pk_live_`, `sk_test_`, `pk_test_` | Stripe secret and publishable keys |
| `sk-`, `sk-ant-` | OpenAI and Anthropic API keys |
| `xoxb-`, `xoxp-`, `xoxa-`, `xoxs-` | Slack tokens |
| `eyJ...` (3-part base64url) | JWT tokens |
| `SG.` | SendGrid API keys |

```python
result = euredact.redact("My API key is sk-proj-abc123def456ghi789jkl0")
print(result.redacted_text)
# "My API key is [SECRET]"
```

### Entropy-Based Detection

For secrets that don't have a recognizable prefix, EuRedact uses Shannon entropy
analysis. A high-entropy string (32+ characters of alphanumeric/base64 content) is
flagged as `SECRET` when it appears near context keywords like `key`, `token`,
`secret`, `password`, `credential`, `auth`, or `bearer` (including translations in
12 European languages).

```python
result = euredact.redact("The api_key is xK9mPqR7vLnW2bFjY8cGhT4sDfAeU6iO")
print(result.redacted_text)
# "The api_key is [SECRET]"

# Without a context keyword, the same string is not flagged:
result = euredact.redact("identifier: xK9mPqR7vLnW2bFjY8cGhT4sDfAeU6iO")
print(result.redacted_text)
# "identifier: xK9mPqR7vLnW2bFjY8cGhT4sDfAeU6iO"  (unchanged)

# Low-entropy strings are also not flagged, even with context:
result = euredact.redact("The password is aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")
print(result.redacted_text)
# "The password is aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"  (unchanged)
```

## Country Hints

Two arguments tell the engine about country, and neither restricts what is
looked for:

| Argument | Meaning |
|---|---|
| `countries=[...]` | **Scope.** Resolves ambiguity, and flags anything attributed elsewhere as `out_of_scope`. |
| `country_hint=[...]` | **Prior only.** Resolves ambiguity without narrowing scope or flagging anything. |

```python
# You know this batch is Swedish, but do not want foreign PII marked out of scope:
euredact.redact(text, country_hint=["SE"])

# You want anything non-Swedish flagged for review:
euredact.redact(text, countries=["SE"])
```

Declaring a country helps where a value is genuinely ambiguous and the document
carries no other signal:

```python
euredact.redact("Telefon: 0708787668", country_hint=["SE"]).detections[0]
# PHONE / SE  — without the hint this is a valid Danish CPR and nothing says otherwise
```

`result.detection_mode` reports which happened: `"declared"` if you passed
`countries=`, `"inferred"` otherwise.

Custom patterns are always active regardless of either parameter.

Passing neither is safe, and is the right default for mixed-origin data: the
engine infers what it can and reports it in `result.inferred_countries`. On our
152,300-document corpus, blind detection scores 99.7% precision against 99.9%
with country hints — a 0.2-point gap, down from 3.4 points before inference.

## Chunked documents

A long document is usually redacted in pieces. Each piece is scanned
independently, so each infers its own country — and a chunk that happens to
contain no IBAN, no `+CC` number and no ccTLD infers nothing at all, even when
page 1 identified the document beyond doubt:

```python
euredact.redact("Telefoon 0612345678")
# -> NATIONAL_ID (DK)   <- a valid Danish CPR, and nothing here says otherwise
```

Pass a `DocumentContext` to carry evidence forward across chunks:

```python
from euredact import DocumentContext

ctx = DocumentContext()
offset = 0
for page in pages:
    result = euredact.redact(page, context=ctx, chunk_offset=offset)
    offset += len(page)

# page 1: "Factuur — IBAN NL91ABNA0417164300, info@example.nl"
# page 7: "Telefoon 0612345678"  -> PHONE (NL)
```

`chunk_offset` rebases spans recorded in the context so they point into the
whole document. Returned detections stay relative to the chunk you passed in.

A context is thread-safe — `aredact_batch` fans chunks across a thread pool.
Caching is disabled automatically while one is in use, because the result then
depends on evidence the text alone does not determine.

Reuse a context only for chunks of the **same** document. Sharing one across
unrelated documents mixes their countries together. That cannot hide anything
— the invariant still holds — but it will attribute values to the wrong
national scheme.

## Referential Integrity

When `referential_integrity=True`, each unique PII value is mapped to a consistent
label within the session. The same input always produces the same label:

```python
import euredact

text = "BSN 111222333 en later weer 111222333, IBAN NL91ABNA0417164300."
result = euredact.redact(text, referential_integrity=True)
print(result.redacted_text)
# "BSN NATIONAL_ID_1 en later weer NATIONAL_ID_1, IBAN BANK_ACCOUNT_1."
```

The mapping is scoped to the `EuRedact` instance. The module-level `redact()`
function uses a shared singleton, so labels are consistent across calls within
the same process.

## Reversible tokenization

`tokenize=True` is for text that has to come back. Each value is replaced by a
token that names its type and nothing else, and the result carries the mapping
that turns tokens back into values:

```python
import euredact

prompt = "Write an email to Joren at joren.janssens@euredact.be or call +32 475 12 34 56 about the invoice."
result = euredact.redact(prompt, countries=["BE"], tokenize=True)
print(result.redacted_text)
# "Write an email to Joren at EMAIL_K7Q2 or call PHONE_P4RT about the invoice."
print(result.tokens)
# {'EMAIL_K7Q2': 'joren.janssens@euredact.be', 'PHONE_P4RT': '+32 475 12 34 56'}

reply = call_your_llm(result.redacted_text)   # sees tokens, never the values
print(euredact.restore(reply, result.tokens))
# the reply, with the real address and number back in it
```

The suffixes are random; yours will differ. With the [cloud tier](#cloud-tier)
the name is tokenized too (`PERSON_NAME_W3NB`), since person names have no
shape for the rules tier to match on.

Within one call the same value gets the same token, so a prompt that names
someone twice still reads as one person. Across calls it gets a different
token: nothing is retained on the instance, `result.tokens` is the only copy,
and two tokenized documents never reveal that they share a value. That is the
opposite retention model from `referential_integrity`, which is why the two
cannot be combined. Batch and iterator variants tokenize each text on its own.

A token is `TYPE_` plus four characters from `BCDFGHJKLMNPQRSTVWXZ23456789`
(no vowels or `Y`, no `0`/`1`/`I`/`O`), about 615,000 suffixes. Tokens are kept clear of any token-shaped
string already in the document, so an LLM's reply to a tokenized prompt can
itself be redacted without `restore()` putting the wrong value back.
`restore()` replaces every occurrence, including a token an LLM glued to other
characters (`EMAIL_P4RTs`) — leaving a token behind is the worse failure.

Works in cloud mode: tokens are minted locally, after the response, over the
local spans and the service's. What is sent to the service always carries
`[TYPE]` placeholders, never tokens.

## Allowlist

Values a caller declares are not PII to them — their own organisation's name,
their own addresses. Set it per call, on the instance, or both; the two merge:

```python
import euredact

sdk = euredact.EuRedact(allowlist=["ACME NV", "info@acme.be"])

text = "ACME NV: mail info@acme.be or jan@acme.be about IBAN NL91 ABNA 0417 1643 00."
print(sdk.redact(text, countries=["NL"]).redacted_text)
# "ACME NV: mail info@acme.be or [EMAIL] about IBAN [BANK_ACCOUNT]."

print(sdk.redact(text, countries=["NL"], allowlist=["jan@acme.be"]).redacted_text)
# "ACME NV: mail info@acme.be or jan@acme.be about IBAN [BANK_ACCOUNT]."
```

Matching is whole-span and case-insensitive. A bare string
(`allowlist="ACME NV"`) raises `TypeError` rather than being iterated into
single letters that exempt nothing. Works in cloud mode — including on types
only the model finds, such as `ORGANISATION_NAME` — and together with
`tokenize`.

**Inside a longer cloud span.** The model's spans are not shaped like values:
it may return `ACME NV Brussel` or `Praktijk Dr. Peeters` as one
`ORGANISATION_NAME`. An allowlisted value is never masked, so it is cut out of
a cloud span that contains it as a whole word, and the rest of the span stays
masked under its type: `ACME NV [ORGANISATION_NAME] betaalt.` The cut is in
`exempted` with `rule_kind` `"value"`. Rules detections keep whole-span
matching, and `allowlist_domains` is unaffected.

### Spacing and punctuation

For **structured identifiers** separators are presentational, so an allowlisted
value matches however the document writes it:

```python
sdk = euredact.EuRedact(allowlist=["NL91ABNA0417164300"])
sdk.redact("Pay to NL91 ABNA 0417 1643 00.", countries=["NL"]).redacted_text
# 'Pay to NL91 ABNA 0417 1643 00.'   — exempt, despite the spacing
```

Applies to `BANK_ACCOUNT`, `BIC`, `CREDIT_CARD`, `PHONE`, `VAT`, `NATIONAL_ID`,
`SSN`, `TAX_ID`, `PASSPORT`, `DRIVERS_LICENSE`, `RESIDENCE_PERMIT`,
`HEALTH_INSURANCE`, `CHAMBER_OF_COMMERCE`, `IMEI` and `VIN`. It widens the
*spelling*, never the *scope*: a different account is still redacted.

Free-text types (`EMAIL`, `PERSON_NAME`, `ORGANISATION_NAME`, …) stay literal,
because folding would exempt values you never listed —
`jan.devries@acme.be` and `jandevries@acme.be` are different mailboxes at most
providers, and `ACME NV` folded would also match `ACMENV`. Case still does not
matter, so list a company's legal spellings if they vary:
`["ACME NV", "ACME N.V."]`.

### Exempting a whole domain

Enumerating every mailbox does not scale — a new hire's address is redacted in
your own documents until someone updates the list. `allowlist_domains` takes
the domain instead:

```python
sdk = euredact.EuRedact(allowlist_domains=["acme.be"])
sdk.redact("Mail jan@acme.be or piet@acme.be", countries=["NL"]).redacted_text
# 'Mail jan@acme.be or piet@acme.be'
```

It applies to `EMAIL` and `URL` only and covers subdomains (`mail.acme.be`).
The match is on a label boundary, so `acme.be` does **not** exempt
`evilacme.be`. Entries may be written `acme.be`, `@acme.be` or `.acme.be`.

There are deliberately **no wildcards**. The allowlist and `exclude_types` are
the only options that turn redaction *off*, so an over-broad entry fails toward under-redaction and
does so silently: `*@*` would disable email redaction entirely and `* Maes` a
family's names. A domain rule cannot be widened that way — it structurally
cannot match a person's name or an IBAN.

### What was exempted

Every exemption is reported, so a document that keeps a direct identifier can be
audited rather than taken on trust:

```python
r = euredact.redact("Mail jan@acme.be", countries=["NL"], allowlist_domains=["acme.be"])
for e in r.exempted:
    print(e.entity_type, e.text, e.rule, e.rule_kind)
# EntityType.EMAIL jan@acme.be acme.be domain
```

`Exemption` carries `entity_type`, `start`, `end`, `text`, the `rule` that
matched as you wrote it, and `rule_kind` (`"value"`, `"domain"` or `"type"`). An exempted
span is absent from `detections`, so `exempted` is the only record that it was
found at all.

## Excluding entity types

A deployment whose taxonomy is narrower than euRedact's can switch whole types
off instead of filtering every result itself. Excluded types are neither in
`detections` nor masked:

```python
r = euredact.redact("Mail anna@acme.be or call +32 470 12 34 56.",
                    countries=["BE"], exclude_types=["PHONE"])
r.redacted_text
# 'Mail [EMAIL] or call +32 470 12 34 56.'
for e in r.exempted:
    print(e.entity_type, e.text, e.rule, e.rule_kind)
# EntityType.PHONE +32 470 12 34 56 PHONE type
```

Set it once for a deployment with `EuRedact(exclude_types=[...])`; a per-call
list is merged with it.

- **Nothing is excluded by default.** Every type is detected and redacted
  unless you name it.
- **Names are checked.** They are case-insensitive, and legacy names resolve
  (`IBAN` is `BANK_ACCOUNT`, `CREDENTIAL` is `SECRET`). A registered custom
  pattern's name works too. An unknown name raises `ValueError`, so a typo
  cannot silently switch nothing off.
- **It applies to each span's final type.** In
  `NHS Number: 5556953627` the digits also pass the Austrian SVNr check.
  Excluding `HEALTH_INSURANCE` leaves them in the text; it does not let them
  be masked as an `SSN` instead.
- **Every excluded detection is recorded** in `result.exempted`, with
  `rule_kind="type"` and the type's name as `rule`, so an audit can see what
  was deliberately left in.
- **In `mode="cloud"`** an excluded rules type is still masked in the text
  sent to the service. The option decides what comes back, not what may leave
  your machine. Model types (`JOB_TITLE`, `FINANCIAL_AMOUNT`, …) are filtered
  from the merged result.
- **An excluded type never uncovers another type's text.** The cloud model
  can give one stretch of text two types: `Fedasil` is an organisation and, in
  context, a sensitive attribute; `Rotterdam` in `Geboorteplaats: Rotterdam`
  is a `LOCATION` inside a `SENSITIVE_ATTRIBUTE`. Excluding one type leaves the
  text masked by the other. Overlapping spans are masked as one stretch under
  one placeholder, and when a non-sensitive type covers the same text as
  `MEDICAL_CONDITION`, `SENSITIVE_ATTRIBUTE` or `BIOMETRIC_REF`, the
  placeholder names the non-sensitive type (`Imam` is `[JOB_TITLE]`), so the
  mask does not disclose the special category. `detections` still lists every
  span with its own type.
- **There is no include-list.** A list of types to keep would leave every type
  a later release adds unmasked for you, without notice.
- **A type is one switch.** `INTERNAL_ID` covers employee and customer numbers
  as well as ticket and case numbers; excluding it leaves all of them in.

## Architecture

```
Input text
    |
    v
[Normalizer] -- Unicode normalization, whitespace cleanup
    |
    v
[Pass 1: Pattern Matching] -- All country + shared + custom regexes
    |                          via MultiPatternMatcher (Aho-Corasick optional)
    v
[Pass 2a: Validation] -- Checksum validators (mod-97, Luhn, entropy, ...)
    |                     Failed spans are recorded, per entity type
    v
[Evidence]     -- Which countries does this document belong to? From IBAN
    |              prefixes, +CC codes, VAT prefixes, BIC, email ccTLDs
    v
[Pass 2b: Suppression] -- Remove false positives (currency amounts, units,
    |                      references). Failed checksums demote same-type
    |                      matches rather than deleting them
    v
[Deduplication] -- Priority-aware, country-evidence-weighted
    |               Longer span outranks declared country
    v
[Replacement] -- Right-to-left substitution with [ENTITY_TYPE] labels
    |               or labels
    v
RedactResult
```

The engine is **thread-safe**: a `threading.Lock` guards country loading and
custom-pattern registration, and all detection state is local to each `detect()`
call. `Detection` objects are frozen dataclasses and can be safely shared across
threads.

Registering a pattern while other threads are detecting is safe too:
`add_custom_pattern` builds a whole new scan plan and publishes it in one
assignment, so a scan in flight keeps using the plan it started with and sees
the pattern set as of either before or after the registration — never a mixture.
It will not observe the new pattern until it next calls `detect()`.

Two caveats that are about *sharing*, not locking:

- **Referential integrity is per instance.** Labels (`EMAIL_1`, …) come from a
  mapper owned by the `EuRedact` instance, so every caller of the module-level
  `euredact.redact()` shares one namespace. A label appearing in two documents
  tells you they contain the same underlying value. Give each tenant its own
  `EuRedact()` instance.
- **The mapper is never evicted**, because evicting would give a previously seen
  value a second label. It grows until you call `clear()`, which is the right
  thing to do between workloads in a long-running process.

### Suppression Zones

When a regex matches a pattern that has a checksum validator but the checksum
fails, the matched span becomes a "suppression zone." Any purely regex-based
detection fully contained within that zone is suppressed as a false positive.

For example, the text `BE71 0012 3456 7890` matches the Belgian IBAN regex but
fails mod-97 validation. Without suppression zones, sub-parts of this span might
be incorrectly detected as a license plate (`BE71`) or a phone number
(`0012 3456`). The suppression zone prevents these false positives while
correctly not reporting an invalid IBAN.

### Deduplication Priority

When multiple patterns match overlapping spans, the engine resolves conflicts
using a priority system:

| Tier | What |
|---:|---|
| 3 | **Validated** — a checksum validator passes *and* the document corroborates its country |
| 2 | **Custom patterns** registered via `add_custom_pattern()` |
| 1 | **Regex-only**, and validated patterns whose country the document does not corroborate |
| 0 | **Postal codes** — a bare digit run, the weakest evidence in the engine |
| -1 | **Demoted** — a validator-less match inside a failed checksum of *its own type* |

Within a tier, ranking is: longer span, then stronger country evidence, then
whether the country was declared. **Span length outranks country** deliberately:
preferring the declared country over the longest match truncates entities — with
`countries=["BE"]` the Belgian phone pattern claimed 11 of the 14 characters of
`06 12 34 56 78` and left three digits exposed. Country can change *which*
country is attributed, never *what* is masked.

Two tiers are less obvious than they look:

- **A passing checksum does not automatically win.** A weak checksum fits by
  luck — a mod-11 scheme accepts a random number about one time in eleven — so a
  validated candidate from a country the document shows no trace of drops to
  tier 1. Entities that carry their own country vouch for themselves (an IBAN
  emits evidence for its own country), so a foreign IBAN in a domestic invoice
  keeps tier 3.
- **Nothing is deleted for failing a checksum.** A failed checksum demotes
  rather than removes, and only candidates of *the same entity type*: it is
  evidence against that type, not against the span. Deleting instead removed 454
  detections across the corpus, of which 454 overlapped real labelled PII.

## Adding a New Country

Each country is a single Python file in `src/euredact/rules/countries/`. The
registry discovers new countries automatically -- no manual registration required.

1. Create a file, e.g. `src/euredact/rules/countries/gr.py`.
2. Define a `CountryConfig` subclass with patterns:

```python
"""Greece (GR) PII patterns."""

from euredact.rules.countries._base import CountryConfig, PatternDef
from euredact.types import EntityType


class GRConfig(CountryConfig):
    def __post_init__(self) -> None:
        self.code = "GR"
        self.name = "Greece"
        self.patterns = [
            PatternDef(
                entity_type=EntityType.NATIONAL_ID,
                pattern=r"\b[A-Z]{2}[0-9]{6}\b",
                validator=None,
                description="Greek national ID (example)",
            ),
        ]
```

That is all. The `CountryRegistry` scans the `countries/` package at startup and
picks up any module that defines a `CountryConfig` subclass with a non-empty
`code`. Files prefixed with `_` (like `_base.py` and `_shared.py`) receive
special treatment and are not treated as standalone countries.

Each `PatternDef` can specify:
- `entity_type` -- which `EntityType` this pattern detects
- `pattern` -- a regular expression
- `validator` -- an optional named validator (e.g. `"bsn"`, `"luhn"`, `"iban"`)
- `context_keywords` -- proximity keywords that boost confidence
- `requires_context` -- if `True`, the match is discarded without a nearby keyword

## Performance

Measured on one core (Apple Silicon M3 Pro, CPython 3.12.13), all 31 countries
loaded, `detect_dates=True`, cache off, over two cohorts sampled evenly from the
corpus: 3,000 distinct short records (~190 chars) and 611 distinct real
documents (~3,450 chars).

**Median** per document, 10th–90th percentile in brackets:

| Input | Pure Python | Aho-Corasick | With `[fast]` |
|---|---:|---:|---:|
| Short record (~190 chars) | 1,612 µs <br><sub>1,072 – 2,417</sub> | 1,507 µs <br><sub>956 – 2,293</sub> | **1,141 µs — 876/s** <br><sub>599 – 1,958</sub> |
| Real document (~3,450 chars) | 18.88 ms <br><sub>15.28 – 25.54</sub> | 15.95 ms <br><sub>12.51 – 22.70</sub> | **9.36 ms — 107/s** <br><sub>6.22 – 15.83</sub> |
| Memory per country | ~50 KB | ~50 KB | ~50 KB |

The spread matters more than the median: **cost tracks identifier density, not
length.** At an identical 3,424 characters, the cheapest document in the corpus
takes 4.8 ms on `[fast]` and the dearest 9.5 ms; across the whole long cohort
the range is 6.2–15.8 ms. A profile shows why — the dominant cost is not the
pattern scan but the cue and suppressor checks that run per candidate match, so
a document dense in identifiers pays for every one of them. Quote a figure for
*your* documents, not this table.

Making `\b` catch identifiers glued to a non-ASCII letter costs something, but
only where it must: next to a digit the ASCII reading alone is exactly
equivalent, so one lookaround replaces a three-way alternation, and plain `\b`
stays everywhere else. 0.3.3 applied the union to all 303 patterns and was
1.8×/2.8× slower for it.

```bash
pip install euredact[fast]
```

The `fast` extra installs two optional accelerators. Neither changes what is
detected — both are covered by `tests/test_scan_path_parity.py`, which runs
every available scan path against the plain-Python one and requires them to
agree:

- **`google-re2`** builds a prefilter over every pattern it can express (324 of
  347). One DFA pass per 1 KB window reports which patterns match anywhere in
  it — typically 42 of 314 for a real document — and only those are then run.
  A pattern RE2 rejects only for a lookaround is prefiltered by the same pattern
  with its lookarounds removed, a superset that can never skip a real match.
  The rest, chiefly the `SECRET` rules and patterns that can match further than
  the window overlap, always run.

  Asking the question per window matters: over a long document nearly every
  pattern matches *somewhere*, so a whole-document prefilter stops filtering
  (2.48× on a short record, decaying to 1.11× at 12 KB).

- **`pyahocorasick`** indexes patterns that begin with a literal and runs them
  only near a prefix hit. Used when `google-re2` is unavailable.

Detection cost is dominated by how much prose surrounds the PII, not by
document size alone: the same engine runs 1.4× faster on dense records and
~3× faster on prose-heavy documents where whole windows can be skipped.

The TypeScript SDK needs no such extra — V8's regex engine has literal
prefilters CPython lacks, and measured on the same documents it runs roughly 3×
faster than the accelerated Python path.

## License

Apache 2.0
