Metadata-Version: 2.4
Name: flexorch-audit
Version: 0.10.2
Summary: Zero-dependency PII + quality + noise audit for LLM datasets (TR/EU/US)
Project-URL: Homepage, https://flexorch.com
Project-URL: Issues, https://github.com/flexorch/flexorch-audit/issues
License: MIT
License-File: LICENSE
Keywords: audit,dataset,gdpr,kvkk,llm,pii,privacy,tckn
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing
Requires-Python: >=3.10
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == 'dev'
Description-Content-Type: text/markdown

# flexorch-audit

[![PyPI](https://img.shields.io/pypi/v/flexorch-audit)](https://pypi.org/project/flexorch-audit/)
[![Python](https://img.shields.io/pypi/pyversions/flexorch-audit)](https://pypi.org/project/flexorch-audit/)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)

Zero-dependency PII detection, quality grading, and noise audit for LLM datasets — in a single function call.

## Why

Before feeding documents into an LLM pipeline you need to answer three questions:

1. **Does this text contain personal data?** Sending PII to a language model is a compliance risk.
2. **Is the text quality high enough?** Short, noisy, or duplicate records hurt fine-tuning and RAG retrieval.
3. **How bad is the noise?** Garbled encodings and symbol clutter degrade model output silently.

Most tools that answer these questions require heavy NLP frameworks, model weights, or cloud APIs. `flexorch-audit` answers all three with one call — using only regex and Python's standard library. No model weights, no network calls, no external packages.

## Features

- **Quality grade** — A/B/C/D composite score: is this text LLM-ready at a glance?
- **Noise ratio** — line-level symbol clutter detection (`noise_ratio`); values above 0.20 indicate likely extraction artifacts
- **PII detection** — 45 types across 15 countries (TR, DE, AT, FR, BE, IT, NL, ES, UK, US, PL, PT, SE, DK, FI) + universal types; all regex-based with checksum validation where a checksum algorithm exists
- **Batch audit** — `audit_batch()` aggregates duplicate ratio and PII counts across an entire dataset in one call
- **Masking** — four strategies: redact, replace (synthetic), token, hash
- **Zero runtime dependencies** — pure Python stdlib, Python 3.10+

## Install

```bash
pip install flexorch-audit
```

## Quick start

```python
from flexorch_audit import audit, mask

text = open("contract.txt").read()  # extract from PDF/DOCX first

result = audit(text)               # "und" by default — all detectors active
# result = audit(text, locale="tr")  # restrict to TR-only detectors

result.quality_grade      # "B"
result.quality_score      # 0.73  (0.0–1.0 composite)
result.noise_ratio        # 0.04  (fraction of blank/garbage lines; >0.20 = low quality)
result.detected_language  # "und" (locale you passed in; caller controls language)
result.pii_summary        # [{"type": "email", "count": 2}, {"type": "national_id_tr", "count": 1}]

# Full findings and raw metrics — dict access also works:
result["pii"]    # [{"type": "email", "value": "ali@example.com", "start": 8, "end": 23}]
result["quality"]  # {"completeness": 1.0, "avg_length": 342, "duplicate_ratio": None}
result["noise"]    # {"garbage_ratio": 0.0, "encoding_ok": True}

clean = mask(text, result["pii"], strategy="redact")
# "Contact: [MASKED_EMAIL]"
```

![demo](assets/demo.svg)

## One-shot redaction

```python
from flexorch_audit import redact_for_llm

clean = redact_for_llm("TCKN: 12345678950, email: ali@example.com", locale="tr")
# "TCKN: [MASKED_NATIONAL_ID_TR], email: [MASKED_EMAIL]"

# Different masking strategies
redact_for_llm(text, locale="tr", strategy="token")   # <PII_NATIONAL_ID_TR_1>
redact_for_llm(text, locale="tr", strategy="hash")    # [3d4f9a1b2c8e7f0a]
redact_for_llm(text, locale="tr", strategy="replace") # static synthetic value
```

No PII found → original text returned unchanged.

## Token estimation

```python
from flexorch_audit import estimate_tokens

estimate_tokens("The quick brown fox jumps over the lazy dog.")  # → 16
estimate_tokens("")  # → 0
```

Heuristic: `words × 4/3` — no `tiktoken` required. Accuracy within ~15% of the real tokenizer for English and most European languages; treat as a planning estimate for context window sizing and cost forecasting.

## Batch audit

```python
from flexorch_audit import audit_batch

texts = [record["text"] for record in dataset]
batch = audit_batch(texts)           # locale="und" by default

batch["duplicate_ratio"]    # 0.12 — fraction of exact-duplicate records
batch["avg_quality_score"]  # 0.78
batch["pii_summary"]        # [{"type": "email", "count": 47}, ...]
batch["results"]            # list of AuditResult, one per text
```

## Streaming audit

```python
from flexorch_audit import audit_stream

async def lines():
    for line in open("data.txt"):
        yield line

async for result in audit_stream(lines()):
    print(result.quality_grade, result.pii_summary)
```

`audit_stream()` is an async generator over an async-iterable of texts — one `AuditResult` per input, yielded as soon as it's ready instead of waiting for the whole batch like `audit_batch()`. Useful for large document sets read incrementally (a file line-by-line, a paginated DB cursor). Processing is sequential per item; for CPU-bound parallelism, run each item through `asyncio.to_thread()` or a thread pool externally.

## Country coverage

| `locale` | Detectors activated |
|----------|---------------------|
| `"und"` **(default)** | All locales combined — use when document language is unknown |
| `"all"` | Alias for `"und"` |
| `"tr"` | TCKN · VKN · phone_tr · name · IBAN_TR · company_name_tr · MERSIS · postal_code_tr · province_tr · SGK sicil · emeklilik sicil · işyeri sicil · Bağkur sicil |
| `"de"` | Steueridentifikationsnummer · Sozialversicherungsnummer |
| `"fr"` | SIREN · SIRET · INSEE/NIR |
| `"it"` | Codice Fiscale · Partita IVA |
| `"nl"` | BSN · KvK |
| `"es"` | DNI/NIE · CIF |
| `"uk"` | NI number · UTR |
| `"us"` | SSN · EIN · ITIN · E.164 phone · company name |
| `"eu"` | E.164 phone · IBAN (EU+GB+CH+NO) · company name |
| `"pl"` | PESEL · NIP |
| `"pt"` | NIF |
| `"sv"` | Personnummer (Sweden) |
| `"da"` | CPR (Denmark) |
| `"fi"` | HETU (Finland) |
| `"at"` | Sozialversicherungsnummer (SVNr) — also active under `"de"` |
| `"be"` | Rijksregisternummer / Numéro de registre national — also active under `"fr"` and `"nl"` |

Universal detectors (always active regardless of locale): `email` · `iban` · `credit_card` · `ip` · `ip_v6`

> **Language detection:** `flexorch-audit` is zero-dependency — no language detection library is included.
> Pass the correct `locale` yourself, or use `"und"` (default) to activate all detectors.

## PII types

### Universal

| Type | Description |
|------|-------------|
| `email` | RFC-5321 email address |
| `iban` | ISO 13616 IBAN — mod-97 validated; suppressed when `iban_tr` or `iban_intl` fires on same span |
| `credit_card` | 16-digit groups, Luhn-validated |
| `ip` | IPv4 address |
| `ip_v6` | IPv6 — full, compressed `::`, loopback forms |

### Turkey (`locale="tr"`)

| Type | Description |
|------|-------------|
| `national_id_tr` | TCKN — 11-digit, modular arithmetic checksum |
| `tax_id_tr` | VKN — 10-digit, Luhn-variant checksum |
| `phone_tr` | Turkish mobile: `+90`/`0` prefix + 10 digits |
| `name` | Label-prefixed name: `Adı:`, `Full Name:`, `Customer Name:`, etc. |
| `iban_tr` | Turkish IBAN (`TR` + 24 chars), mod-97 validated |
| `company_name_tr` | Company with TR legal suffix: A.Ş. · Ltd.Şti. · Koll.Şti. · Koop. · T.A.Ş. |
| `mersis_no` | MERSIS — 16-digit company registry number |
| `postal_code_tr` | Turkish postal code (province plate 01–81) |
| `province_tr` | All 81 Turkish provinces |
| `sgk_no` | SGK Sicil Numarası — 10–11 digits, label-prefix gated |
| `emeklilik_no` | Emeklilik sicil numarası — 9–11 digits, label-prefix gated (`Emeklilik No`, `E Sicil` / `E. Sicil`, `Emekli Maaşı Nosu`) |
| `isyeri_sicil_no` | İşyeri SGK sicil kodu — 8–9 digits, label-prefix gated |
| `bagkur_no` | Bağkur (4/b) sicil numarası — 10–11 digits, label-prefix gated |

### Germany (`locale="de"`)

| Type | Description |
|------|-------------|
| `tax_id_de` | Steueridentifikationsnummer — 11 digits, ISO 7064 MOD 11,2 checksum |
| `social_id_de` | Sozialversicherungsnummer — area + DOB + letter + serial |

### France (`locale="fr"`)

| Type | Description |
|------|-------------|
| `siret_fr` | SIRET — 14 digits, label-prefix gated |
| `company_id_fr` | SIREN — 9 digits, label-prefix gated |
| `social_id_fr` | INSEE/NIR — 15 digits, starts with `1` or `2` |

### Italy (`locale="it"`)

| Type | Description |
|------|-------------|
| `national_id_it` | Codice Fiscale — 16 chars alphanumeric, uppercase normalized |
| `tax_id_it` | Partita IVA — 11 digits, Agenzia delle Entrate checksum |

### Netherlands (`locale="nl"`)

| Type | Description |
|------|-------------|
| `national_id_nl` | BSN — 9 digits, 11-check (weighted sum mod 11) |
| `company_id_nl` | KvK — 8 digits, label-prefix gated |

### Spain (`locale="es"`)

| Type | Description |
|------|-------------|
| `national_id_es` | DNI (8 digits + letter, mod-23) and NIE (X/Y/Z prefix, same check) |
| `tax_id_es` | CIF — letter prefix + 7 digits + control character |

### United Kingdom (`locale="uk"`)

| Type | Description |
|------|-------------|
| `social_id_uk` | NI number — 2 letters + 6 digits + A/B/C/D; HMRC forbidden prefixes excluded |
| `tax_id_uk` | UTR — 10 digits, label-prefix gated |

### United States (`locale="us"`)

| Type | Description |
|------|-------------|
| `ssn` | SSN — `###-##-####`, invalid prefixes (000/666/9xx) excluded |
| `tax_id_us` | EIN — `XX-XXXXXXX`, IRS invalid area prefixes excluded |
| `national_id_us` | ITIN — `9XX-7X/8X/9X-XXXX` middle group validated |

### EU / International (`locale="eu"`)

| Type | Description |
|------|-------------|
| `phone_intl` | E.164 international phone — 7–15 digits, TR (+90) excluded |
| `iban_intl` | IBAN for EU+GB+CH+NO — ISO 13616 country+length table + mod-97 |
| `company_name_intl` | Company with international suffix: GmbH · LLC · S.r.l. · B.V. · SAS · Inc. · Ltd. etc. |

### Austria (`locale="at"`, also active under `"de"`)

| Type | Description |
|------|-------------|
| `social_id_at` | Sozialversicherungsnummer (SVNr) — 10 digits, weighted checksum. No label required. |

### Belgium (`locale="be"`, also active under `"fr"` and `"nl"`)

| Type | Description |
|------|-------------|
| `national_id_be` | Rijksregisternummer / Numéro de registre national — accepts both the dotted display format (`YY.MM.DD-XXX.CC`) and bare 11 digits, mod-97 checksum |

### Poland (`locale="pl"`)

| Type | Description |
|------|-------------|
| `national_id_pl` | PESEL — 11 digits, weighted-sum checksum |
| `tax_id_pl` | NIP — 10 digits, label-prefix gated |

### Portugal (`locale="pt"`)

| Type | Description |
|------|-------------|
| `tax_id_pt` | NIF — 9 digits, weighted-sum mod-11 checksum |

### Sweden (`locale="sv"`)

| Type | Description |
|------|-------------|
| `national_id_se` | Personnummer — `YYMMDD-NNNN` / `YYYYMMDD-NNNN`, `+` separator marks pre-1900 birth |

### Denmark (`locale="da"`)

| Type | Description |
|------|-------------|
| `national_id_dk` | CPR — `DDMMYY-XXXX` |

### Finland (`locale="fi"`)

| Type | Description |
|------|-------------|
| `national_id_fi` | HETU — `DDMMYY[+\-A]XXX` + checksum character |

## Structured field masking

Sometimes a value doesn't match any PII pattern (empty cell, non-standard ID format) but the **column name itself** already tells you it's sensitive. `mask_record_fields()` masks by key name instead of scanning the value:

```python
from flexorch_audit import mask_record_fields

row = {"tckn": "12345678950", "customer_email": "ali@example.com", "city": "Istanbul"}
mask_record_fields(row)
# {"tckn": "[MASKED_NATIONAL_ID_TR]", "customer_email": "[MASKED_EMAIL]", "city": "Istanbul"}
```

Matching is case-insensitive and matches on exact key name or substring (`"customer_email"` matches because it contains `"email"`). `None` values and unmatched keys are left untouched. Returns a new dict — the input is not mutated. This does not replace `mask()` — use it alongside PII span-detection for tabular/JSON data where key names carry signal that regex on the value alone would miss.

## Compliance report

```python
from flexorch_audit import compliance_report

result = audit(text, locale="tr")
report = compliance_report(result)
# {
#     "has_pii": True,
#     "pii_types": ["email", "national_id_tr"],
#     "risk_level": "high",           # "none" | "low" | "medium" | "high"
#     "masking_required": True,
#     "recommendations": [
#         "Apply mask(strategy='redact') before storing or sharing this text.",
#         "Review applicable regulations (KVKK Art. 6, GDPR Art. 9) for special category data handling.",
#     ],
# }
```

A technical summary only — not a legal document or regulatory opinion. `risk_level` is `"high"` when any national-ID-class type is found (TCKN, SSN, credit card, etc.), `"medium"` for contact-info-class types (email, phone, IBAN, name), `"low"` otherwise, `"none"` when no PII was found.

## RAG helpers

Zero-dependency chunking and indexing-readiness metadata for RAG pipelines:

```python
from flexorch_audit import audit, chunk_text, rag_metadata

result = audit(text, locale="tr")
meta = rag_metadata(result)
# {"should_index": True, "pii_safe": False, "quality_gate": "pass",
#  "recommended_chunk_size": 512, "estimated_chunks": 4}

if meta["should_index"]:
    chunks = chunk_text(text, strategy="paragraph", max_tokens=meta["recommended_chunk_size"])
    # [{"text": "...", "token_count": 118, "chunk_index": 0}, ...]
```

`chunk_text()` supports three strategies: `"paragraph"` (default, splits on blank lines and merges short ones), `"sliding_window"` (fixed-size overlapping windows via `overlap`), and `"sentence"` (splits on sentence boundaries). `rag_metadata()` reads `quality_grade`/`pii`/`avg_length` off an `AuditResult` — `should_index` is `True` only for grade A/B, and `pii_safe` is `True` only when no PII was detected; combine both before indexing into a vector store.

## Noise detection

`noise_ratio` measures the fraction of lines that are blank or contain symbol clutter:

```python
result = audit("clean line\n@@@garbage\n\nclean")
result.noise_ratio   # 0.5  (2 noisy lines out of 4)
```

A line is "noisy" when it is blank (after strip) or contains 3+ consecutive characters from `@ # ! ~ * =`.

| `noise_ratio` | Signal |
|---------------|--------|
| `< 0.05` | Clean — likely well-extracted text |
| `0.05–0.20` | Acceptable — minor formatting artifacts |
| `> 0.20` | Low quality — likely OCR noise or extraction failure |

## Masking strategies

```python
clean = mask(text, result["pii"], strategy="redact")   # default
clean = mask(text, result["pii"], strategy="token")
clean = mask(text, result["pii"], strategy="hash")
clean = mask(text, result["pii"], strategy="replace")
```

| Strategy | Example output |
|----------|----------------|
| `redact` (default) | `[MASKED_EMAIL]` |
| `replace` | `user@example.com` (static synthetic) |
| `token` | `<PII_EMAIL_1>` (unique per type per call) |
| `hash` | `[3d4f9a1b2c8e7f0a]` (SHA-256 first 16 hex chars) |

## Quality grade

`quality_grade` (A–D) and `quality_score` (0.0–1.0) are composite signals:

| Grade | Score | Signal |
|-------|-------|--------|
| A | ≥ 0.85 | Ready for LLM training or RAG |
| B | ≥ 0.65 | Usable with minor cleanup |
| C | ≥ 0.40 | Review before use |
| D | < 0.40 | Not suitable — empty, too short, or high noise |

Score formula: `completeness × (0.4 × noise_score + 0.4 × length_score + 0.2)`  
`length_score = min(char_count / 500, 1.0)` · `noise_score = max(0, 1 − garbage_ratio × 10)`

## Limitations

- **No automatic language detection** — `flexorch-audit` has zero dependencies. Pass `locale` explicitly, or use the default `"und"` to activate all detectors. See [LIMITATIONS.md](LIMITATIONS.md).
- **Free-standing name detection** (without a label prefix) requires NLP/NER — not included.
- `replace` masking uses realistic pooled synthetic values for `national_id_tr`, `iban_tr`/`iban_intl`, and `name`; every other type gets a fixed static placeholder (e.g. `user@example.com`, `000-00-0000`) — not a locale-aware generator.
- `mask_record_fields()` matches by key name (exact or substring), not by inspecting the value — it will not catch a sensitive value sitting under an unrelated key name.
- The library audits plain text. PDF/DOCX parsing, e-invoice extraction, and pipeline orchestration are out of scope.

## Integrations

[![Works with LangChain](https://img.shields.io/badge/Works%20with-LangChain-blue)](examples/langchain_loader.py)
[![Works with LlamaIndex](https://img.shields.io/badge/Works%20with-LlamaIndex-purple)](examples/llamaindex_reader.py)

`flexorch-audit` slots into any LangChain or LlamaIndex pipeline as a pre-load filter — audit quality, detect PII, and optionally mask before your documents reach the LLM.

**LangChain** — [`examples/langchain_loader.py`](examples/langchain_loader.py)

```python
from examples.langchain_loader import AuditedLoader  # copy to your project

loader = AuditedLoader(
    texts=my_texts,
    locale="tr",       # or "de", "fr", "us", "und" (all)
    mask_pii=True,     # redact PII before loading
    min_grade="B",     # skip low-quality documents
)
docs = loader.load()
# doc.metadata → {"quality_grade": "A", "quality_score": 0.91, "pii_summary": [...], ...}
```

**LlamaIndex** — [`examples/llamaindex_reader.py`](examples/llamaindex_reader.py)

```python
from examples.llamaindex_reader import AuditedReader  # copy to your project

reader = AuditedReader(locale="tr", mask_pii=True)
docs = reader.load_data(my_texts, min_grade="B")
# doc.extra_info → {"quality_grade": "A", "quality_score": 0.91, "pii_summary": [...], ...}
```

Both loaders are thin wrappers (~60 lines) with no new dependencies beyond `langchain-core` or `llama-index-core`. Copy them into your project — no framework lock-in.

## Also available for JavaScript / TypeScript

```bash
npm install @flexorch/audit
```

## Contributing

See [CONTRIBUTING.md](CONTRIBUTING.md).

## License

MIT
