Metadata-Version: 2.4
Name: dlp-patterns
Version: 0.4.0
Summary: Fast, zero-dependency DLP pattern scanner — detects PII, secrets, and sensitive data in text
Project-URL: Homepage, https://spidercob.com
Project-URL: Repository, https://github.com/spidercob/dlp-patterns
Project-URL: Bug Tracker, https://github.com/spidercob/dlp-patterns/issues
License: Apache-2.0
License-File: LICENSE
Keywords: compliance,data-loss-prevention,dlp,gdpr,hipaa,pci-dss,pii,privacy,redaction,regex,secret-scanning,secrets,security,sensitive-data
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Information Technology
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Security
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Topic :: Text Processing
Requires-Python: >=3.9
Provides-Extra: dev
Requires-Dist: faker>=20.0; extra == 'dev'
Requires-Dist: pytest>=7; extra == 'dev'
Provides-Extra: fuzz
Requires-Dist: faker>=20.0; extra == 'fuzz'
Description-Content-Type: text/markdown

# dlp-patterns

**Fast, zero-dependency DLP pattern scanner for Python.**

Detects PII, secrets, and sensitive data in any text — documents, logs, source code, emails. Built from the scanning engine that powers [Spidercob](https://spidercob.com), an enterprise DLP platform.

```python
import dlp_patterns

result = dlp_patterns.scan("My SSN is 432-78-9012 and CC 4111 1111 1111 1111")
print(result.highest_severity)   # CRITICAL
print(result.critical[0].type)   # credit_card

clean = dlp_patterns.redact("Send to alice@corp.com with CC 4111 1111 1111 1111")
# "Send to [REDACTED: Email Address] with [REDACTED: Credit Card Number]"
```

## Install

```bash
pip install dlp-patterns
```

No external dependencies. Python 3.9+.

## What it detects

| Category | Patterns |
|---|---|
| **Financial** | Credit cards (Luhn + BIN), SSN, IBAN, bank account, routing number |
| **PII** | Email, US phone, passport, driver's license, date of birth |
| **Healthcare** | Medical record numbers, ICD-10 codes, NPI, DEA numbers, NDC codes |
| **Secrets** | AWS keys, GitHub PATs, Slack tokens, Google API keys, Bearer tokens, JWTs |
| **Cloud / SaaS** | Stripe, SendGrid, Mailgun, Twilio, HuggingFace, NPM, Cloudflare, Azure |
| **Infrastructure** | DB connection strings, hardcoded passwords, Docker registry auth |
| **Crypto** | RSA/EC/SSH/PGP private keys, X.509 certs |
| **Webhooks** | Slack webhooks, Discord webhooks, Telegram bot tokens |
| **Cryptocurrency** | Bitcoin addresses, Ethereum addresses |

50+ pattern categories total.

## Features

- **Validators** — Luhn check for credit cards, FICA rules for SSNs, JSON decode for JWTs. Reduces false positives before they reach you.
- **Entropy gating** — Shannon entropy + sliding-window analysis rejects low-entropy matches (e.g. `aaaaaaa...`) from generic secret patterns.
- **Context scoring** — Each finding gets a `context_score` (0–1) based on surrounding words. Proximity to `production`, `secret`, `deploy` boosts the score; proximity to `example`, `placeholder`, `test` lowers it.
- **Required context keywords** — Patterns like ICD-10 codes and Telegram tokens only fire when relevant keywords appear nearby.
- **`secrets_only` mode** — Scan just for API keys and credentials, skipping PII. Faster for CI/CD secret scanning.
- **`redact()`** — Replace all findings with `[REDACTED: <type>]`.
- **`fuzz()`** — Replace findings with realistic fake values (requires `faker`). Useful for building safe test datasets from production data.
- **Git history scanning** — `dlp-scan --history` walks every commit, not just the working tree, so a secret that was committed and later deleted still gets caught. See [Git history scanning](#git-history-scanning).
- **Live secret verification** — `dlp-scan --verify` checks whether a found secret is still active by making a real, read-only call to its own provider's API. See [Live secret verification](#live-secret-verification).
- **CLI** — `dlp-scan` command for shell pipelines and CI.

## Usage

### Python API

```python
import dlp_patterns

# Scan
result = dlp_patterns.scan(text)

result.has_findings          # bool
result.highest_severity      # "CRITICAL" | "HIGH" | "MEDIUM" | "LOW" | None
result.critical              # list[Finding]
result.all                   # all findings across severities
result.elapsed_ms            # scan time in milliseconds

# Each Finding:
f = result.critical[0]
f.type                       # "credit_card"
f.description                # "Credit Card Number"
f.value                      # masked: "4111...1111"
f.severity                   # "CRITICAL"
f.position                   # "char 10-29"
f.context                    # surrounding text (±100 chars)
f.context_score              # float 0.0–1.0
f.context_keywords_found     # ["payment", "billing"]
f.verification                # None until dlp_patterns.verify() is called

# Secrets only (faster for source code scanning)
result = dlp_patterns.scan(code, secrets_only=True)

# Redact
clean = dlp_patterns.redact(text)

# Fuzz (pip install dlp-patterns[fuzz])
safe = dlp_patterns.fuzz(text)

# JSON output
result.to_dict()
```

### CLI

```bash
# Scan a string
dlp-scan "My SSN is 432-78-9012"

# Scan a file
dlp-scan path/to/document.txt

# Scan a directory recursively
dlp-scan path/to/project/

# Pipe from stdin
cat logfile.txt | dlp-scan

# JSON output
dlp-scan --json document.txt

# Redact in place (files only, not directories)
dlp-scan --redact document.txt > clean.txt

# Secrets only (for source code)
dlp-scan --secrets-only src/config.py

# Exit code: 1 if CRITICAL findings, 0 otherwise — useful in CI
dlp-scan --secrets-only . && echo "clean"

# Only fail the exit code for high-confidence findings (context_score >= 0.5) —
# reduces false-positive CI failures from doc/test fixtures without hiding them
# from the output. Findings are always reported regardless of this flag.
dlp-scan --secrets-only --min-confidence 0.5 .
```

#### Exit code and `--min-confidence`

By default, exit code `1` means "at least one CRITICAL finding" — full
stop, regardless of how confident the match is. The engine already
computes a `context_score` per finding (0.0-1.0: proximity to words like
`production`/`deploy` raises it, proximity to `example`/`test`/`mock`/a
code fence lowers it — see [Features](#features)), but by default the
exit code ignores it entirely, same as it always has.

`--min-confidence <float>` changes what the exit code reacts to: a
CRITICAL finding only fails the build if its `context_score` is `>=` the
given value. This matters most for scanning a codebase whose job
description *includes* containing realistic-looking fake secrets — a
secret-scanner's own test suite, security training data, a docs site with
credential examples — where `--secrets-only` alone will always find
something. Findings are still fully reported either way; the flag only
changes what fails the build.

```bash
dlp-scan --secrets-only --min-confidence 0.5 src/     # a reasonable CI default
dlp-scan --secrets-only --min-confidence 0.5 --history .   # combine with history scanning
```

#### Directory scanning

`dlp-scan <directory>` walks recursively and scans every text file it finds.
It automatically skips:

- Version control and vendor directories: `.git`, `node_modules`, `__pycache__`, `.venv`, `venv`, `dist`, `build`, `.mypy_cache`, `.pytest_cache`, `.tox`
- Lockfiles (`package-lock.json`, `yarn.lock`, `pnpm-lock.yaml`, `Cargo.lock`, `poetry.lock`, `Pipfile.lock`, `go.sum`, `composer.lock`, `npm-shrinkwrap.json`) — these are auto-generated and full of base64 integrity hashes that false-positive against secret regexes
- Binary files (detected by a null-byte sniff on the first 8KB)

`--json` output for a directory scan has a different shape than a single-file scan — findings are grouped by file:

```json
{
  "mode": "directory",
  "path": "src/",
  "files_scanned": 42,
  "files_with_findings": 2,
  "highest_severity": "CRITICAL",
  "findings_by_file": {
    "config.py": { "CRITICAL": [...], "HIGH": [], "MEDIUM": [], "LOW": [], "INFO": [], "elapsed_ms": 1.2 }
  },
  "elapsed_ms": 38.4
}
```

Exit code is still `1` if any file has a CRITICAL finding, `0` otherwise.

### Git history scanning

Deleting a leaked key from HEAD does not un-leak it — anyone with a clone of
the repo (including one taken before the deletion) can still read it out of
`git log -p`. This is the single most common way real secrets leak, and
plain `dlp-scan <directory>` mode can't see it: it walks the *working tree*
and explicitly skips `.git`. `--history` walks every commit on every branch
instead:

```bash
# Scan full history of the repo at (or containing) the given path
dlp-scan --history path/to/repo

# Defaults to the current directory
dlp-scan --history

# Limit to the N most recent commits
dlp-scan --history --max-commits 500 .

# --history defaults to secrets only (PII noise across full history is
# high — every email/phone number ever committed, including test fixtures).
# Opt into PII scanning too:
dlp-scan --history --full-scan .

dlp-scan --history --json . | tee history-report.json
```

Only lines *added* in a commit's diff are scanned, once per commit that
introduced them — sufficient and non-redundant, since a line later removed
was necessarily added by some earlier commit. Requires the `git` binary on
PATH (an external tool dependency, not a pip package — this library still
ships with zero pip-installable dependencies).

Each finding carries the commit it was introduced in:

```json
{
  "type": "aws_access_key",
  "severity": "CRITICAL",
  "value": "AKIAIOSFOD...",
  "commit": "<full 40-char SHA>",
  "short_commit": "b874554",
  "author": "Jane Doe <jane@example.com>",
  "date": "2026-08-06T09:46:52+05:30",
  "subject": "add key",
  "file": "config.py"
}
```

This only *detects* — it intentionally does not try to rewrite history or
force-push a fix. Purging a secret from history for real (`git filter-repo`
+ a coordinated force-push + rotating the credential) is a separate,
higher-stakes operation you should do deliberately, not something a scanner
should do for you.

### Live secret verification

Regex matching alone can't tell a live production key from one that's
already been rotated. `--verify` closes that gap for a curated set of
secret types by making one real, read-only API call per distinct secret —
"who am I" / "list my scopes", never an action — to confirm whether it
still authenticates:

```bash
dlp-scan --verify --secrets-only src/config.py
dlp-scan --verify --history .          # combine with history scanning
dlp-scan --verify --verify-timeout 8 . # per-secret network timeout (default 4s)
```

```
[CRITICAL]
  github_token                   GitHub Personal Access Token
                                  value=ghp_9a...***  pos=char 15-55
                                  verify=INVALID (GitHub API rejected the token (401))
```

**Opt-in only** — never runs from a plain `dlp-scan`/`scan()` call, and the
CLI prints a warning the first time `--verify` fires. It's off by default
because it makes real network requests using the extracted secret value.

**What's checked:** GitHub PATs, Slack tokens, Stripe (secret/restricted)
keys, SendGrid, HuggingFace, npm, Google API keys, Telegram bot tokens,
Mailgun, and Cloudflare API tokens — plus two *paired* credential types:

- **AWS access/secret keys** — an access key ID alone can't authenticate
  anything; it needs its secret key signed alongside it. `--verify` finds
  an `aws_access_key` and `aws_secret_key` near each other in the same file
  (within ~2000 characters — comfortably a few lines of a config/env file)
  and verifies the pair with a single [AWS Signature Version 4][sigv4]
  -signed call to STS `GetCallerIdentity` — implemented with stdlib
  `hmac`/`hashlib`, no `boto3` dependency. Temporary (`ASIA`-prefixed)
  credentials additionally look for a paired `aws_session_token` nearby;
  without one, signing would fail even for a genuinely live credential, so
  it's reported `unverifiable` rather than a wrong `invalid`.
- **Twilio Account SID + Auth Token** — paired the same way, verified with
  a Basic-auth `GET` against the account's own Twilio API endpoint.

[sigv4]: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_sigv-signing.html

An access key (or SID/token) with no partner found nearby is reported
`unverifiable`, not silently skipped — pairing never guesses across file or
commit boundaries (an access key in one file is never checked against a
secret key from another).

**What's deliberately *not* checked, and why:**
- **Slack/Discord webhook URLs** — "verifying" a webhook means POSTing to
  it, which sends a real message to someone's channel. That's a side
  effect, not a check, so it's never attempted.
- A network error (timeout, DNS failure) is always reported as `error`,
  never collapsed into `invalid` — not knowing is not the same as revoked.

Same `verify=` line appears in directory and `--history` output. In
directory/history mode, an identical secret found in multiple files or
commits is checked against its provider **once**, not once per occurrence.

Python API:

```python
import dlp_patterns

result = dlp_patterns.scan(code, secrets_only=True)
dlp_patterns.verify(result)  # mutates findings in place

for f in result.all:
    if f.verification:
        print(f.type, f.verification.status, f.verification.detail)
```

### Pre-commit hook

Block commits containing secrets automatically using [dlp-pre-commit](https://github.com/SpiderCob/dlp-pre-commit):

```yaml
repos:
  - repo: https://github.com/SpiderCob/dlp-pre-commit
    rev: v1.0.0
    hooks:
      - id: dlp-scan-secrets-only
```

```bash
pip install pre-commit
pre-commit install
```

### Use in CI (GitHub Actions)

The easiest way is the official [dlp-scan-action](https://github.com/SpiderCob/dlp-scan-action):

```yaml
- name: DLP Secret Scan
  uses: spidercob/dlp-scan-action@v1
  with:
    secrets-only: 'true'
    fail-on: 'critical'
```

Or run the CLI directly:

```yaml
- name: DLP secret scan
  run: |
    pip install dlp-patterns
    dlp-scan --secrets-only --json src/ | tee dlp-report.json
```

## Advanced — use `Scanner` directly

```python
from dlp_patterns import Scanner

scanner = Scanner()

# Reuse the same instance (compiled patterns cached)
for text in documents:
    result = scanner.scan(text)
    if result.has_findings:
        print(result.to_dict())
```

## Enterprise

Need a full DLP platform with dashboards, audit logs, ICAP proxy integration, Gmail/Slack scanning, compliance reports, and AI-powered analysis?

→ **[Spidercob](https://spidercob.com)** — the enterprise DLP platform this library is extracted from.

## License

Apache 2.0 — free for commercial use.
