Metadata-Version: 2.5
Name: aice-data-quality
Version: 0.1.1
Summary: Data quality SDK: checks, one result contract, one destination
Author-email: Adanian Labs <engineering@adanianlabs.io>
License: MIT
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Typing :: Typed
Requires-Python: <3.14,>=3.10
Requires-Dist: pydantic>=2
Requires-Dist: pyyaml>=6
Provides-Extra: dbt
Provides-Extra: dev
Requires-Dist: httpx>=0.27; extra == 'dev'
Requires-Dist: jsonschema[format]>=4.21; extra == 'dev'
Requires-Dist: openpyxl>=3.1; extra == 'dev'
Requires-Dist: opentelemetry-api>=1.20; extra == 'dev'
Requires-Dist: opentelemetry-sdk>=1.20; extra == 'dev'
Requires-Dist: pandas>=2; extra == 'dev'
Requires-Dist: psycopg[binary]>=3.1; extra == 'dev'
Requires-Dist: pytest>=8.3; extra == 'dev'
Requires-Dist: sqlalchemy>=2; extra == 'dev'
Provides-Extra: excel
Requires-Dist: openpyxl>=3.1; extra == 'excel'
Requires-Dist: pandas>=2; extra == 'excel'
Provides-Extra: http
Requires-Dist: httpx>=0.27; extra == 'http'
Provides-Extra: otel
Requires-Dist: opentelemetry-api>=1.20; extra == 'otel'
Provides-Extra: pandas
Requires-Dist: pandas>=2; extra == 'pandas'
Provides-Extra: sql
Requires-Dist: psycopg[binary]>=3.1; extra == 'sql'
Requires-Dist: sqlalchemy>=2; extra == 'sql'
Description-Content-Type: text/markdown

# aice-data-quality

Python SDK for data quality checks. Run checks on a DataFrame, a CSV or Excel file, a SQL table or
dbt test results, and record each outcome as one standard result record sent to one destination.
Checks never raise when data is bad: you get a report and decide what to do with it.

```bash
pip install aice-data-quality
```

Requires Python 3.10 to 3.13. The import name is `dq`.

## Quickstart

```python
import pandas as pd

from dq import QualityClient, Dataset, Severity, checks as c
from dq.sources import DataFrameSource

df = pd.DataFrame({"amount": ["12.50", "7", None]})

client = QualityClient(project="my-project", environment="production")

with client.run(pipeline="uploads") as run:
    report = run.validate(
        DataFrameSource(df),
        dataset=Dataset("finance_uploads", "statement", "bronze"),
        owner="finance",
        checks=[c.not_null("amount", name="amount_set", severity=Severity.BLOCK)],
    )

if report.blocked:
    raise ValueError(report.messages())
```

`report.messages()` names the columns, rules, counts and row numbers involved, never the values.
With no server configured, results are logged as JSON lines, which is enough to try the SDK locally.

## Install extras

The base install covers checks and suites. Add extras for the sources and destinations you use:

| Extra | Adds | Use it for |
|---|---|---|
| `pandas` | pandas | `DataFrameSource`, `CsvSource` |
| `excel` | pandas, openpyxl | `ExcelSource` |
| `sql` | SQLAlchemy, psycopg | `SqlSource` (PostgreSQL) |
| `http` | httpx | sending results to a server (`HttpSink`) |
| `otel` | opentelemetry-api | tracing spans |

```bash
pip install "aice-data-quality[excel,http]"
```

## Configuration

`QualityClient()` reads environment variables. Arguments you pass override them.

| Variable | Meaning | Default |
|---|---|---|
| `DQ_PROJECT` | Project name (required) | |
| `DQ_ENVIRONMENT` | `development`, `staging` or `production` (required) | |
| `DQ_SINK` | `log` or `http` | `log` |
| `DQ_SERVER_URL` | Server address, when the sink is `http` | |
| `DQ_TOKEN` | Write token for this project and environment | |
| `DQ_SPOOL_DIR` | Where undelivered results wait for retry | `.dq-spool` |
| `DQ_SAMPLE_POLICY` | `keys` or `none`: what row identifiers may appear in results | `keys` |
| `DQ_SENSITIVE_DATASETS` | Comma-separated `source.name` datasets whose keys are hashed | |
| `DQ_HASH_SALT` | Secret used for hashing. Without it, sensitive keys are dropped | |
| `DQ_CHECK_TIMEOUT_S` | Time limit per check, in seconds | `60` |

In containers, mount a volume at `DQ_SPOOL_DIR` so results survive a restart. If the server is
unreachable, results are kept in the spool and replayed in order.

## Checks

| Check | What it verifies |
|---|---|
| `schema` | Required columns exist, with the expected types |
| `not_null` | A column has no missing values |
| `parses_as` | Values parse as a date, number or other type, using the formats you give |
| `accepted_values` | Values are in an allowed set |
| `unique` | One column or a combination of columns has no duplicates |
| `row_count` | Row count is within bounds, or within a range of the previous run (`vs_previous`) |

Every check takes:

- `name`: required, lowercase snake case. Keep it stable, because it identifies the check and its history.
- `severity`: `block`, `flag` or `warn` (default `flag`).
- `dimension` and `key`: optional.

A report exposes `blocked`, `flagged` and `messages()`. A check that fails never raises or exits the
process. Only an unreadable source raises `SourceError` (an unreadable file, formulas with no cached
values, an oversized file or a broken query), and the run is recorded as `error`.

## Sources

| Source | Reads | Extra |
|---|---|---|
| `DataFrameSource(df)` | a pandas DataFrame | `pandas` |
| `CsvSource(path)` | a CSV file | `pandas` |
| `ExcelSource(path, sheet=...)` | an Excel sheet | `excel` |
| `SqlSource(...)` | a table or query in the database that holds the data being checked | `sql` |

To add your own, implement `columns() -> dict[str, str]` and
`evaluate(check, ctx) -> Measurement`, and raise `UnsupportedCheck` for checks it cannot run.

## YAML suites

Keep checks in a file instead of code:

```yaml
dataset: { source: finance_uploads, name: bank_statement, layer: bronze }
owner: finance
tags: [finance.cash]
checks:
  - { name: required_columns, type: schema, columns: [txn_date, amount], severity: block }
  - { name: date_parses, type: parses_as, column: txn_date, as: date, formats: ["%d/%m/%Y"], severity: block }
  - { name: has_rows, type: row_count, min: 1, severity: block }
```

```python
report = run.validate(ExcelSource(path, sheet="Transactions"),
                      suite=client.load_suite("quality/bank.yml"))
```

Validate suite files without running them, for example in CI:

```bash
dq suite validate quality/*.yml
```

## dbt

Publish dbt test results as result records after a build:

```bash
dbt build; status=$?
dq dbt publish --manifest target/manifest.json --run-results target/run_results.json
exit $status
```

Set owners and dimensions in dbt YAML with `meta: {dq_owner: finance}` on a model or test, and
`dq_dimension` on tests whose dimension cannot be inferred. Publishing the same invocation twice
replaces it.

## Privacy

Results never contain data values. Failing rows are identified by key only, according to
`DQ_SAMPLE_POLICY`. For sensitive datasets, set `DQ_SENSITIVE_DATASETS` and `DQ_HASH_SALT` so keys are
hashed with an HMAC. If no salt is set, those keys are dropped instead.

## Destinations

- `LogSink`: writes results as JSON lines to the log. This is the default.
- `HttpSink`: sends results to a server over HTTP, with a local spool for retry and ordered replay.
  Set `DQ_SINK=http`, `DQ_SERVER_URL` and `DQ_TOKEN`.

The SDK stores nothing itself. Results are kept and queried on the server.

## License

MIT
