Metadata-Version: 2.5
Name: qavf
Version: 0.6.2
Summary: Configurable, plugin-based data QA/QC: column limits, SQL checks, temporal and spatial outliers, regression testing and HTML reports, over any source DuckDB can read.
Project-URL: Homepage, https://github.com/AhmedAredah/qavf
Project-URL: Documentation, https://github.com/AhmedAredah/qavf/tree/main/docs
Project-URL: Issues, https://github.com/AhmedAredah/qavf/issues
Project-URL: Changelog, https://github.com/AhmedAredah/qavf/blob/main/CHANGELOG.md
Author-email: Ahmed Aredah <Ahmed.Aredah@gmail.com>
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: data-quality,data-validation,duckdb,outlier-detection,polars,qa,qc,regression-testing,validation
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: Software Development :: Testing
Classifier: Typing :: Typed
Requires-Python: >=3.12
Requires-Dist: duckdb<3,>=1.3.1
Requires-Dist: jinja2>=3.1
Requires-Dist: markupsafe>=2.1
Requires-Dist: numpy>=1.26
Requires-Dist: pyarrow>=16
Requires-Dist: pydantic>=2.7
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: tzdata; sys_platform == 'win32'
Provides-Extra: all
Requires-Dist: pandas>=2.1.3; extra == 'all'
Requires-Dist: polars>=1.32; extra == 'all'
Requires-Dist: pyproj>=3.6.1; extra == 'all'
Requires-Dist: scipy>=1.11.3; extra == 'all'
Provides-Extra: pandas
Requires-Dist: pandas>=2.1.3; extra == 'pandas'
Provides-Extra: polars
Requires-Dist: polars>=1.32; extra == 'polars'
Provides-Extra: spatial
Requires-Dist: polars>=1.32; extra == 'spatial'
Requires-Dist: pyproj>=3.6.1; extra == 'spatial'
Requires-Dist: scipy>=1.11.3; extra == 'spatial'
Provides-Extra: stats
Requires-Dist: polars>=1.32; extra == 'stats'
Requires-Dist: scipy>=1.11.3; extra == 'stats'
Description-Content-Type: text/markdown

# QAVF

[![CI](https://github.com/AhmedAredah/qavf/actions/workflows/ci.yml/badge.svg)](https://github.com/AhmedAredah/qavf/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/qavf)](https://pypi.org/project/qavf/)
[![Python](https://img.shields.io/badge/python-3.12%20%7C%203.13%20%7C%203.14-blue)](https://github.com/AhmedAredah/qavf/blob/main/pyproject.toml)
[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue)](https://github.com/AhmedAredah/qavf/blob/main/LICENSE)

**Configurable data QA/QC for Python.** Validate files, dataframes and database
queries against declared limits, statistical expectations and stored baselines.
QAVF reports violation counts with bounded evidence. Counts are exact unless
explicitly labelled as lower bounds, and an insufficient lower bound cannot
establish a pass.

- **Any input.** CSV, Parquet, JSON, Excel, globs and URLs; DuckDB, SQLite, PostgreSQL
  and MySQL tables; SQL queries; Polars, pandas and Arrow frames; Python loaders.
- **Column limits.** Types, nullability, ranges, allowed values, patterns, lengths,
  uniqueness, keys, foreign keys and aggregate statistics, each with its own severity.
- **Statistical and temporal QA.** Trend, jump, seasonal and level-shift checks; entity
  outliers; coverage gaps; distribution drift; reconciliation of totals across datasets.
- **Regression testing.** Keyed, tolerance-aware comparison against accepted baselines,
  usable from pytest.
- **Dataset comparison.** Compare two versions of a dataset whose columns, types,
  units or codes differ: structure, key coverage, record-level changes, grouped
  measures and time series, distributions, relationships between columns and what
  explains the total change, a difference failing when it is both material (beyond a
  tolerance) and statistically significant.
- **Spatial data.** Coordinates inside a study area, points assigned to zone
  polygons or grid cells, per-zone comparisons with expected zones, rank and share
  changes, and Moran's I and LISA naming the clusters of change; features that moved
  or changed shape. Geometry uses DuckDB's `spatial` extension, and without it those
  checks are not assessed, never passed.
- **Reports and CI gates.** HTML and JSON reports, deterministic evidence and exit codes.
- **Extensible.** Rules, sources, comparison levels, measures, comparators and
  statistical tests are plain classes discovered from packages, files or entry points.

## Install

```bash
pip install "qavf[stats]"                                     # or: uv tool install "qavf[stats]"
brew install AhmedAredah/tap/qavf                             # macOS and Linux command line
docker run --rm -v "$PWD:/work" ghcr.io/ahmedaredah/qavf run -c qavf.yaml
```

Python 3.12 or newer on Linux, macOS or Windows. `pip install qavf` provides
DuckDB/Arrow validation without Polars. The `polars` extra installs the full
execution backend; `stats` adds that backend and SciPy-based tests; `pandas` adds
pandas adapters; `spatial` adds the full backend, indexed neighbour searches and
authoritative CRS resolution; `all` installs every extra. In GitHub Actions, use
[`AhmedAredah/qavf-action`](https://github.com/AhmedAredah/qavf-action).

## Quick start

```bash
qavf demo my-demo && qavf run -c my-demo/qavf.yaml   # open my-demo/qa_output/qa_report.html
```

For your own data, generate a configuration with inferred limits, review it, then run it:

```bash
qavf init -d data/counts.csv -d crashes=data/crashes.parquet --margin 0.1
qavf run -c qavf.yaml
```

```yaml
datasets:
  counts:
    path: data/counts.csv
    schema:
      primary_key: [site_id, year]
      columns:
        site_id: {dtype: str, regex: 'S\d{2}'}
        aadt:
          dtype: int
          ge: 0
          checks:
            - {le: 200000, severity: WARNING, name: implausibly high}
        truck_share: {dtype: float, ge: 0, le: 1}
    foreign_keys:
      - {columns: [site_id], ref_dataset: sites}
  sites:
    path: data/sites.csv
    schema: {primary_key: [site_id]}
```

Row expressions and SQL assertions run in DuckDB, where each dataset is a table:

```yaml
datasets:
  crashes:
    path: data/crashes.parquet
    expressions:
      - {expr: fatal <= crashes, severity: CRITICAL}
assertions:
  - name: every site has counts
    sql: SELECT site_id FROM sites WHERE site_id NOT IN (SELECT site_id FROM counts)
```

[`config/qavf.full.yaml`](https://github.com/AhmedAredah/qavf/blob/main/config/qavf.full.yaml)
documents every option. Paths are relative to the configuration file, and any string
may use `${ENV_VAR}` so credentials stay out of the file.

See [backends and installation](docs/backends.md), [key contracts](docs/key_contracts.md),
[not-applicable values](docs/applicability.md) and [value policies](docs/value_policies.md)
for nullable keys, N/A tokens, number grammar and what each backend supports.

## Cloud storage

Read `s3://`, `az://` and `abfss://` URLs with credentials configured once. AWS
roles, Azure identity and GitHub OIDC logins work without stored keys:

```yaml
secrets:
  lake: {type: s3, provider: credential_chain, region: us-east-1}
  blob: {type: azure, connection_string: "${AZURE_STORAGE_CONNECTION_STRING}"}
datasets:
  orders: {path: "s3://my-lake/orders/*.parquet"}
```

S3-compatible stores (Cloudflare R2, MinIO, Wasabi) take an `endpoint`. See
[cloud storage](https://github.com/AhmedAredah/qavf/blob/main/docs/cloud_storage.md).

## Python API

```python
from qavf import Dataset, ResourceLimits, validate

report = validate(
    "readings.parquet",
    schema={"columns": {"value": {"dtype": "float", "ge": 0}}},
)
print(report.verdict())

report = validate(
    {
        "counts": Dataset(counts_frame, schema={"columns": {"count": {"ge": 0}}}),
        "sites": Dataset("sites.parquet", schema={"primary_key": ["site_id"]}),
    }
)

# Validate the configured inputs in a supervised worker with a wall-time deadline.
report = validate(config="qavf.yaml", resources=ResourceLimits(wall_time=60))
```

Add `address_space="4GiB"` to limit each process's virtual address space on Linux
or committed memory on Windows; macOS rejects this control. See
[execution limits](docs/execution_limits.md) for the supported controls and their scopes.

## How a run is judged

Each result is **PASS**, **FAIL**, **SKIPPED** (nothing configured) or **ERROR**
(configured but not assessed, for example missing data or invalid SQL).

| Verdict | When | Exit code |
|---|---|---|
| passed | At least one check was assessed, none reports ERROR, and no failure meets `engine.fail_on` | 0 |
| failed | A failure at or above `engine.fail_on` (default `ERROR`) | 1 |
| incomplete | No qualifying failure, but a check could not be assessed or nothing was assessed | 1 by default |

A qualifying failure takes precedence over incompleteness. Setting
`engine.fail_on_incomplete: false` allows exit code `0` for an incomplete report;
its verdict remains `incomplete`.

Use `engine.require` to name checks that must be assessed for a run to pass.

### Reading evidence

A failed check lists the first offending rows, up to `engine.max_affected_records`, with
their values and why each fails (`violates`). `_row` is a row's 0-based position in
the dataset as read: the first data row of a file with a header is `_row: 0`. A
dataset read from several files adds `_file` and `_file_row`, the row's position
within that file. Duplicate keys list their positions in `rows`, and a line the
reader could not parse is named by its file and 1-based `line`.

A column check's failing rows show the columns the check reads (the checked column,
the columns of its `where` and its unit column) and the `primary_key`; other columns
are not read for evidence. Set `evidence_columns: all` in a dataset's `schema` to show
whole rows.

## Rules

| ID | Default | Checks |
|---|---|---|
| INP-001 | CRITICAL | Sources accessible and schemas inspectable |
| INP-002 | CRITICAL | Columns present with declared types; row-count limits |
| INP-003 | ERROR* | Column QA limits (*per-check severities) |
| INP-004 | ERROR | Primary keys unique |
| INP-005 | ERROR* | Foreign keys resolve |
| INP-006 | ERROR* | Row expressions hold |
| INP-007 | ERROR* | SQL assertions return no rows |
| INP-008 | ERROR | Newest date/time value is recent enough |
| INP-009 | ERROR | N/A tokens apply to all or none of a column group |
| TMP-001 | WARNING | Period-over-period change within a rate |
| TMP-002 | WARNING | Series trend in their declared direction |
| TMP-003 | WARNING | No period is a significant and material trend outlier |
| TMP-004 | ERROR | No group is missing a period |
| TMP-005 | WARNING | No value stuck for several periods |
| TMP-006 | ERROR | Cumulative series never decrease |
| TMP-007 | WARNING | No value is anomalous for its season |
| TMP-008 | WARNING | No series shifts to a sustained new level |
| SPA-001 | WARNING | No entity is an outlier among its peers |
| SPA-002 | WARNING | Each entity within a factor of its group's median |
| SPA-003 | ERROR | Every expected entity is covered in every period |
| GEO-001 | ERROR | Coordinates valid, present and inside a bounding box |
| STA-001 | WARNING | z-score, modified z-score or IQR outlier scans |
| STA-002 | WARNING | Distributions have not drifted from a reference |
| REC-001 | ERROR | Totals reconcile between datasets, key by key |
| REC-002 | ERROR | Shares sum to their total within each group |
| REC-003 | ERROR | The full expected key grid is present |
| REC-004 | ERROR | Two datasets have the same key set |
| REC-005 | ERROR | Related datasets carry one identical batch id |
| REG-001 | ERROR | Outputs match baselines within tolerance |
| CMP-001 | ERROR | Two datasets compare within tolerance after alignment |

Parameters for every rule: [docs/rules.md](https://github.com/AhmedAredah/qavf/blob/main/docs/rules.md)
or `qavf list-rules`.

## Regression testing

```python
from qavf.regression import assert_outputs_match


def test_output_matches_baseline(tmp_path):
    run_pipeline(output=tmp_path / "out.parquet")
    assert_outputs_match(
        "tests/baselines/out.parquet", tmp_path / "out.parquet", keys=["site_id", "year"], rtol=1e-6
    )
```

Rows match by key, numbers within tolerance, and changed values, added or removed rows
and schema changes are reported. Accept reviewed outputs with `qavf baseline`, which
records a manifest with the keys, column types and a SHA-256 of the file.

## Dataset comparison

Compare a new version of a dataset with the previous one, even when columns were
renamed, retyped, rescaled or recoded:

```python
from qavf import compare_datasets

result = compare_datasets(
    "v1/counts.parquet",
    "v2/counts.parquet",
    keys=["site_id", "year"],
    columns={"AADT": "aadt"},  # baseline name -> candidate name
    units={"aadt": {"factor": 1000}},  # v1 counted thousands of vehicles
    recode={"fclass": {"values": {"1": "interstate", "2": "arterial"}}},
    levels=[
        "structure",
        "coverage",
        "records",
        {"level": "groups", "group_by": ["county"], "measures": ["count", "sum(aadt)"]},
    ],
)
print(result.summary())
```

Each level reports graded checks with exact counts and bounded evidence: columns
added or removed (with suggested renames), keys on one side only, changed values per
column with their largest differences, and changes of grouped measures that are
material and statistically significant (with p-values, effects and intervals; the
`stats` extra, or `decision="material"` for tolerances alone). A lossy cast or a
missing mapped column makes the comparison not assessed, never a pass. In a configuration, `CMP-001` compares configured datasets. See
[dataset comparison](https://github.com/AhmedAredah/qavf/blob/main/docs/comparison.md).

Spatially, the `assign_zones` step gives points the zone they fall in (a polygon
layer from GeoParquet, GeoJSON or a shapefile, or grid cells), and the `spatial`
level compares zones: one-sided and missing zones, measures per zone, rank and share
changes, and whether the changes cluster (Moran's I, and LISA naming hot and cold
spots after a multiple-testing correction). The `locations` level reports keyed
features that moved (haversine metres) or changed area or shape. See
[spatial data](https://github.com/AhmedAredah/qavf/blob/main/docs/spatial.md).

The `distribution` level compares each column's distribution per group, zone or time
bucket (PSI, KS, Anderson-Darling, Wasserstein, Jensen-Shannon, quantile deltas, and
categories that appeared or disappeared), `relationships` compares correlations, ratio
invariants such as `fatal / crashes` and the agreement of matched records (Bland-Altman,
concordance, error metrics, the candidate-on-baseline line), and `contribution` says
which members explain the total change, with an exact mix-versus-rate decomposition.
HTML reports show each comparison with a verdict per level, a drill-down with tests
and charts; `result.write_html(path)` and `qavf compare-datasets` write the same report
for one comparison.

## CLI

| Command | Purpose |
|---|---|
| `qavf run -c CONFIG` | Run QA and write HTML/JSON reports |
| `qavf init -d [NAME=]PATH` | Profile data and write a starting configuration |
| `qavf profile PATH` | Print inferred QA limits for a file |
| `qavf schema` | Write the configuration JSON Schema for editor support |
| `qavf list-rules` | List supported or discovered rules |
| `qavf capabilities` | Show the backend capability catalog as JSON |
| `qavf compare BASELINE CANDIDATE -k KEY` | Compare an output with a baseline (`--map OLD=NEW`) |
| `qavf compare-datasets BASELINE CANDIDATE -c SPEC` | Compare two datasets at several levels; write `--html`/`--json` reports |
| `qavf baseline SOURCE DEST -k KEY` | Accept an output as a baseline |
| `qavf demo DIR` | Write a runnable demo project |

`--help`, `--version`, `schema`, `list-rules` and native `run` work with the core
installation. `run`, `schema` and `list-rules` accept `--backend auto|duckdb|polars`;
`auto` uses Polars when installed, otherwise DuckDB. Advanced profiling, comparison,
scaffolding and demo commands require `qavf[polars]`. Inspect the import-light
[capability catalog](docs/backend_capabilities.md) before selecting a backend.

Exit codes: `0` passed, `1` QA failed or outputs differ, `2` usage, configuration or
resource error.

## Documentation

- [Python API](https://github.com/AhmedAredah/qavf/blob/main/docs/api.md) and
  [sources](https://github.com/AhmedAredah/qavf/blob/main/docs/sources.md)
- [Dataset comparison](https://github.com/AhmedAredah/qavf/blob/main/docs/comparison.md),
  its [component reference](https://github.com/AhmedAredah/qavf/blob/main/docs/comparison_reference.md)
  and [statistical inference](https://github.com/AhmedAredah/qavf/blob/main/docs/inference.md)
- [Spatial data](https://github.com/AhmedAredah/qavf/blob/main/docs/spatial.md): zone
  layers, CRS, assignment policies, grids, crosswalks, weights and the spatial levels
- [Rules reference](https://github.com/AhmedAredah/qavf/blob/main/docs/rules.md) and
  [writing plugins](https://github.com/AhmedAredah/qavf/blob/main/docs/plugins.md)
- [Cloud storage](https://github.com/AhmedAredah/qavf/blob/main/docs/cloud_storage.md)
- [Resource limits](https://github.com/AhmedAredah/qavf/blob/main/docs/resources.md) and
  [isolated execution](https://github.com/AhmedAredah/qavf/blob/main/docs/execution_limits.md)
- [Numerical contracts](https://github.com/AhmedAredah/qavf/blob/main/docs/numerics.md) and
  [baseline acceptance](https://github.com/AhmedAredah/qavf/blob/main/docs/baseline_acceptance.md)

## Development

```bash
uv sync --all-extras
uv run pytest
uv run ruff check . && uv run mypy qavf scripts
```

See [releasing](https://github.com/AhmedAredah/qavf/blob/main/docs/releasing.md)
for the release checks and [CHANGELOG.md](CHANGELOG.md) for changes.

## License

[Apache License 2.0](LICENSE)
