Metadata-Version: 2.4
Name: trueset
Version: 0.2.0
Summary: Write a data-quality check once, run it anywhere -- pandas, DuckDB, or your warehouse -- plus cross-system reconciliation.
Project-URL: Homepage, https://github.com/trueset-dev/trueset
Project-URL: Repository, https://github.com/trueset-dev/trueset
Project-URL: Issues, https://github.com/trueset-dev/trueset/issues
Project-URL: Changelog, https://github.com/trueset-dev/trueset/blob/main/CHANGELOG.md
Author: trueset contributors
License: Apache-2.0
License-File: LICENSE
Keywords: data-diff,data-engineering,data-quality,data-testing,dbt,duckdb,pyspark,reconciliation,validation
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.10
Requires-Dist: click>=8.1
Requires-Dist: pandas>=1.5
Requires-Dist: pyyaml>=6.0
Requires-Dist: rich>=13.0
Provides-Extra: dev
Requires-Dist: pyarrow>=12.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.6; extra == 'dev'
Provides-Extra: spark
Requires-Dist: pyspark>=3.4; extra == 'spark'
Provides-Extra: sql
Requires-Dist: duckdb>=0.10; extra == 'sql'
Requires-Dist: sqlalchemy>=2.0; extra == 'sql'
Description-Content-Type: text/markdown

# trueset

[![CI](https://github.com/trueset-dev/trueset/actions/workflows/ci.yml/badge.svg)](https://github.com/trueset-dev/trueset/actions/workflows/ci.yml)

**Write a data-quality check once, run it anywhere** — pandas today, PySpark and
your warehouse next. One declarative check language, one Python API, one CLI.

Data quality tools today force a trade-off: `dbt` tests are SQL-only, Great
Expectations is powerful but heavy, and each engine (pandas / Spark / SQL) tends
to want its own syntax. `trueset` bets on a single small **Backend protocol** — a
check is written once and runs unchanged on any engine that implements the
protocol. Adding Spark or Snowflake support means writing *one class*, not
re-authoring a single check.

## Quickstart

```bash
# From source (PyPI release coming — the name is being finalized):
git clone https://github.com/trueset-dev/trueset && cd trueset
pip install -e ".[sql]"          # [sql] adds the DuckDB backend; omit for pandas-only

trueset run --data examples/orders.csv --checks examples/checks.yml
trueset list-checks
```

Declarative checks (`checks.yml`):

```yaml
suite: orders_quality
checks:
  - type: not_null
    column: order_id
  - type: unique
    column: order_id
  - type: in_range
    column: amount
    min: 0
  - type: in_set
    column: status
    values: [pending, shipped, delivered, cancelled]
  - type: matches_regex
    column: email
    pattern: '[^@\s]+@[^@\s]+\.[^@\s]+'
    severity: warn        # warns, doesn't fail the run
```

Or the Python API:

```python
import pandas as pd
from trueset import Suite, validate_dataframe

df = pd.read_csv("orders.csv")
result = validate_dataframe(df, "checks.yml")

print(result.passed)                    # False
print(result.to_dict())                 # JSON-ready, ship to a warehouse/dashboard
```

The CLI exits non-zero when any `error`-severity check fails, so it drops
straight into CI, dbt, or Airflow.

## Use it anywhere in your pipeline (not just the warehouse)

trueset is a **library first**. Because checks only speak to the `Backend`
protocol and the pandas backend takes any DataFrame, you can validate data
*in flight* — at ingestion, before loading — and fail the pipeline the instant a
batch is bad, so garbage never reaches storage (the "shift-left" pattern):

```python
from trueset import validate_dataframe

df = extract_from_api()                        # your ingestion step
result = validate_dataframe(df, "checks.yml")  # same suite, in-memory
if not result.passed:
    raise ValueError(f"bad batch, not loading: {result.counts}")   # fail the task
load_to_warehouse(df)                          # only reached if data is clean
```

Drop that into an Airflow `PythonOperator`, a Dagster op, a Prefect task, a
Lambda, or a plain script. The *same* `checks.yml` then runs post-load against
the warehouse (`SQLAlchemyBackend`) and in CI (the Action) — write once, enforce
at ingestion, in transit, and at rest. See
[`examples/pipeline_demo.py`](examples/pipeline_demo.py) for a runnable gate.
(Batch / micro-batch oriented; native Spark validation is on the roadmap.)

### Quarantine bad rows, keep the clean ones flowing

Blocking the whole batch is one option; routing is better. `split()` partitions a
batch into clean and bad rows — with the reason each bad row failed — so you can
dead-letter the bad and keep moving:

```python
from trueset import split

result = split(df, "checks.yml")     # or a Suite / dict spec
load_to_warehouse(result.good)       # clean rows proceed
dead_letter(result.bad_annotated())  # bad rows + a `_trueset_reasons` column
```

Only `error`-severity checks divert a row by default (a `warn` surfaces the issue
without quarantining it). From the CLI: `trueset run --data batch.csv --checks
c.yml --quarantine bad.csv`. Under the hood, checks expose a per-row
`failure_spec()` and every backend can return the actual failing rows
(`backend.failing_rows(...)`) — identical across pandas, DuckDB, and SQL. trueset
*identifies* the bad rows; **you** decide where they go. See
[`examples/quarantine_demo.py`](examples/quarantine_demo.py).

## Gate your CI in three lines

trueset ships a GitHub Action, so a data-quality check becomes a required status
check on every pull request:

```yaml
# .github/workflows/data-quality.yml
name: data-quality
on: [pull_request]
jobs:
  trueset:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: trueset-dev/trueset@v0
        with:
          data: data/orders.csv
          checks: quality/orders.yml
```

The job fails the moment an `error`-severity check fails. Use `severity: warn` on
a check to surface a problem without blocking the merge.

## Profiling & AI-assisted authoring

Draft a check suite instead of writing one by hand:

```bash
trueset profile  --data orders.csv                 # stats + inferred semantic types
trueset suggest  --data orders.csv                 # deterministic draft suite (no AI)
trueset suggest  --data orders.csv --calibrate     # + data-derived ranges & volume band
trueset suggest  --data orders.csv --ai --out checks.yml           # AI copilot
trueset suggest  --data orders.csv --describe "amount can't be negative; \
    status is one of pending/shipped/delivered/cancelled"        # English -> checks
```

**Auto-calibrated thresholds** (`--calibrate`): rather than hand-picking limits,
trueset derives numeric `in_range` bounds (from the 1st/99th percentiles) and a
row-count volume band straight from your data — emitted as `warn` for you to
review and commit, never auto-enforced. Run it on known-good data: calibration
learns from the sample, so representative input in means sensible thresholds out.

**The trust rule that makes AI safe here:** the copilot only ever *authors*
checks. Every spec it returns — from profiling or natural language — is passed
through the same deterministic check registry (`build_check`). Anything the
model hallucinates or mis-configures is discarded before it can reach your
data. The AI is never in the runtime pass/fail path, so your validation stays
deterministic and auditable. The model is injected as a plain callable
(`Completer`), so it's provider-agnostic and fully testable without a key.

```python
from trueset import profile_dataframe, suggest_from_profile
from trueset.copilot import anthropic_completer, checks_from_profile

prof = profile_dataframe(df)
draft = suggest_from_profile(prof)                      # deterministic
draft = checks_from_profile(prof, anthropic_completer())  # AI (needs ANTHROPIC_API_KEY)
```

## Portability, proven (pandas ⇄ DuckDB)

The whole bet is that a check is written once and runs on any engine. It's not
a slogan -- it's tested. The identical suite, run on pandas (in-memory) and on
DuckDB (SQL pushed into the database), returns the same verdict:

```
check                   pandas      duckdb      match
------------------------------------------------------
row_count_parity        pass(0)     pass(0)     OK
referential_integrity   fail(1)     fail(1)     OK
value_parity            fail(3)     fail(3)     OK
------------------------------------------------------
identical verdict: True
```

```bash
pip install -e ".[sql]"                 # adds duckdb
python examples/portability_demo.py     # runs the comparison above
```

The DuckDB backend implements the same `Backend` protocol, but every check
becomes `SELECT count(*) ... WHERE ...` executed inside the database -- so full
tables never move. `tests/test_duckdb.py` asserts the two engines agree.

## Run against your warehouse (Postgres, Snowflake, BigQuery, …)

The `SQLAlchemyBackend` generalizes the DuckDB proof to **anything SQLAlchemy
speaks** — Postgres, MySQL/MariaDB, SQLite, Snowflake, BigQuery, Redshift. Same
checks, same verdicts, pushed down as SQL. Point the CLI straight at a table:

```bash
trueset run \
  --url "postgresql+psycopg://user:pw@host:5432/analytics" \
  --table public.orders \
  --checks quality/orders.yml
```

Or from Python:

```python
from sqlalchemy import create_engine
from trueset import Suite, SQLAlchemyBackend

engine = create_engine("snowflake://…")          # any SQLAlchemy URL
backend = SQLAlchemyBackend(engine, "ORDERS")
result = Suite.from_yaml("orders.yml").run(backend)
```

`tests/test_sqlalchemy.py` runs the identical example suite on pandas **and** on
a SQL database and asserts the verdicts match — the same guarantee we hold for
DuckDB, now for every warehouse. Reconciliation works across engines too: the
primary can be a pandas DataFrame while the reference is a Postgres table.

## Cross-system reconciliation (the wedge)

The thing no single-source tool (GE, Soda, Pandera, dbt tests) is architected
to do, and whose only open-source option (`data-diff`) was archived in 2024:
validate one dataset **against another system**. Because checks talk only to
the `Backend` protocol, a reconciliation check just holds a second backend --
which can be a totally different engine.

```bash
trueset reconcile \
  --data   warehouse_orders.csv \
  --checks reconcile.yml \
  --ref    source=source_orders.csv
```

```yaml
suite: warehouse_vs_source
checks:
  - type: row_count_parity        # counts agree within tolerance
    reference: source
    tolerance: 0.0
  - type: referential_integrity   # every key traces back to the source
    column: order_id
    reference: source
    ref_column: id
  - type: value_parity            # join on key, compare actual values
    key: order_id
    columns: [amount, status]
    reference: source
    ref_key: id
```

`value_parity` reports four failure modes at once: mismatched values on shared
keys, keys only in the primary, keys only in the reference, and **duplicate join
keys** on either side (an ambiguous join is itself a reconciliation defect).
Either side can be a file, DuckDB, or any SQLAlchemy warehouse — they need not
share an engine.

## Governance: enforcement + evidence

trueset isn't a catalog (that's DataHub/Collibra territory). It owns the half
catalogs are weak on — **enforcing** policy and **proving** compliance happened.
Governance is optional metadata on a check, nothing more:

```yaml
- type: matches_regex
  column: email
  pattern: '[^@\s]+@[^@\s]+\.[^@\s]+'
  owner: risk-team              # accountable party
  sensitivity: pii              # public|internal|confidential|pii|pci|phi
  regulation: [gdpr, ccpa]      # regime tags
  description: "Customer email is PII and must be well-formed"
```

These fields change nothing about how the check runs — they ride onto the result
as machine-readable, auditable evidence. `trueset report` then answers policy
questions directly:

```bash
trueset report --data orders.csv --checks governed.yml --by sensitivity
```

```
 governance report :: orders_quality_governed  (by sensitivity)
┏━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━┳━━━━━━┳━━━━━━━┳━━━━━━━━━━━┓
┃ sensitivity  ┃ checks ┃ pass ┃ fail ┃ error ┃ status    ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━╇━━━━━━╇━━━━━━━╇━━━━━━━━━━━┩
│ pii          │ 1      │ 0    │ 1    │ 0     │ VIOLATION │
│ confidential │ 2      │ 0    │ 2    │ 0     │ VIOLATION │
└──────────────┴────────┴──────┴──────┴───────┴───────────┘
⚠ 3 failing check(s) on sensitive (pii/pci/phi/confidential) data.
```

Group `--by owner` to route failures to a team, or `--by regulation` for a GDPR
/ SOX posture. `--json` emits the same as auditable evidence for a compliance
trail.

**Classification is suggested, never imposed.** `trueset profile` infers a
`sensitivity` for high-precision patterns (email, phone, SSN, credit card via
Luhn, IBAN), and `trueset suggest` pre-tags the drafted checks:

```
column   inferred   sensitivity
email    email      pii
ssn      ssn        pii
card     credit_card pci
```

Both the deterministic profiler and the AI copilot only *suggest* tags — like
every check, they are reviewed and committed by a human, never auto-applied.

## Monitoring: freshness & volume

Turn one-shot validation into continuous monitoring. Persist runs, then watch for
staleness and volume drift:

```bash
trueset run --url "postgresql://…" --table orders --checks orders.yml --save "postgresql://…/trueset_history"
trueset history --store "postgresql://…/trueset_history"      # the audit trail
trueset monitor --store "postgresql://…/trueset_history" --suite orders_quality --sigma 3
```

- **`freshness`** is an ordinary check — the newest timestamp must be recent:
  ```yaml
  - type: freshness
    column: created_at
    max_age_hours: 6        # fail if the table hasn't updated in 6 hours
  ```
- **`trueset monitor`** trends any metric against the baseline of past runs and
  exits non-zero on a >`sigma` anomaly — a silent 90% drop in rows is caught even
  when every row that *is* there passes every check:
  ```bash
  trueset monitor --store "$H" --suite orders --metric rows --method mad
  trueset monitor --store "$H" --suite orders --metric failing_rows --check not_null --column email
  ```
  `--metric` can be `rows`, `failing_rows`, or `total_rows` (with `--check`/
  `--column` to pick a check); `--method` is `zscore` (mean/std) or `mad`
  (median-absolute-deviation — robust when a few past runs were themselves
  outliers). Detection is deterministic and explainable — never a black box.

## Handling ambiguity (when the extreme value is the *truth*)

The hardest real-world data problem — especially in commodities, markets, and
sensors: an extreme value is often *correct* (a cold-snap demand spike, a
geopolitical event, a COVID price move), not an error, and the same statistical
signal can be either. You can't threshold your way out of that. trueset's job is
to **surface and quantify** the ambiguity, not pretend to resolve it.

- **Corroboration** — judge a suspicious value against *supporting signals*, not
  in isolation. A real price spike shows up in volume too; a silent bad tick
  doesn't:
  ```yaml
  - type: corroboration
    column: price
    corroborate_with: [volume]   # trust the spike only if volume backs it
    severity: warn               # surface it — real extremes happen, don't block
  ```
- **Annotate-and-flow** — instead of blocking rows, score them and let them pass
  with metadata (market data needs a *full view*):
  ```python
  from trueset import annotate
  scored = annotate(df, suite, key="day")   # adds _trueset_quality (0..1) + _trueset_flags
  ```
- **Adjudication** — when a human rules a flag "actually valid," record it so it's
  never re-flagged (the feedback loop that kills repeat false positives):
  ```python
  from trueset import Adjudications
  adj = Adjudications(); adj.mark_valid("in_range(price)", "2026-03-09")
  annotate(df, suite, key="day", adjudications=adj)   # suppressed going forward
  ```
- **Context-aware ranges** — one expected band *per segment* (region/season), via
  `segment_bounds(df, "price", "region")`, so a legitimate regime isn't judged by
  another's. All of it rests on robust statistics (`stats.robust_z` / MAD).

Runnable: [`examples/ambiguity_demo.py`](examples/ambiguity_demo.py). (In-memory
today; warehouse pushdown is on the [roadmap](ROADMAP.md).)

## Works with your stack (dbt today)

trueset composes with the tools you already run — it doesn't replace them. If you
have dbt tests, adopt trueset **without rewriting a single one**:

```bash
trueset import-dbt --schema models/schema.yml --model orders --out orders.yml
trueset run --url "$WAREHOUSE" --table analytics.orders --checks orders.yml
```

`not_null`/`unique` map directly, `accepted_values`→`in_set`,
`relationships`→`referential_integrity`, and dbt test severity is preserved.
Custom/singular dbt tests are reported (never silently dropped), and every
imported check is validated so the output always runs. Point trueset at the
tables dbt builds and you also get cross-engine portability, reconciliation
against the source, governance, and monitoring — on top of your existing tests.

## Built-in checks

`columns_exist`, `not_null`, `unique`, `in_set`, `in_range`, `matches_regex`,
`row_count`, `no_duplicate_rows`, `freshness`, `metric`, plus the reconciliation
checks `row_count_parity`, `referential_integrity`, `value_parity`. Each has a
`severity` of `error` (default) or `warn`.

## Architecture

```
YAML / Python API
        │
     Suite ── list of ── Check          checks speak ONLY to the Backend
        │                  │            protocol, never to an engine
        ▼                  ▼
   SuiteResult  ◀──  Backend protocol
                          ▲
        ┌────────────┬───────┴───────┬────────────┐
   PandasBackend  DuckDBBackend  SQLAlchemyBackend  (SparkBackend)
     [built]        [built]     [built: any warehouse]  [next]
```

The `Backend` protocol (`src/trueset/backends/base.py`) is deliberately tiny:
`row_count`, `null_count`, `distinct_count`, `count_out_of_range`, etc. A SQL
backend implements each as a pushed-down `SELECT count(*) ... WHERE ...` so data
never leaves the warehouse.

## Roadmap

See **[ROADMAP.md](ROADMAP.md)** for what's shipped, what's next, and the idea
inbox. Governance, results history, freshness/volume monitoring, and dbt import
all shipped in v0.1.0; auto-calibrated thresholds and AI failure diagnosis are
next. Have an idea? Drop it in the roadmap's idea inbox or open a
[feature request](../../issues/new?template=feature_request.md).

## Development

```bash
pip install -e ".[dev]"
pytest
```

## License

Apache-2.0 (recommended for broad corporate adoption).
