Metadata-Version: 2.4
Name: metricproof
Version: 0.3.0
Summary: Executable checks for trustworthy business metrics, cohorts, denominators, and analytical populations.
Author: AtomicGlance
License-Expression: MIT
Project-URL: Repository, https://github.com/AtomicGlance/metricproof
Project-URL: Issues, https://github.com/AtomicGlance/metricproof/issues
Project-URL: PyPI, https://pypi.org/project/metricproof/
Keywords: analytics,data-quality,metrics,cohort-analysis,retention,reconciliation
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Information Analysis
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: jsonschema>=4.23; extra == "dev"
Requires-Dist: ruff>=0.12; extra == "dev"
Dynamic: license-file

# MetricProof

[![Tests](https://github.com/AtomicGlance/metricproof/actions/workflows/tests.yml/badge.svg)](https://github.com/AtomicGlance/metricproof/actions/workflows/tests.yml)
[![PyPI](https://img.shields.io/pypi/v/metricproof.svg)](https://pypi.org/project/metricproof/)
[![Python](https://img.shields.io/pypi/pyversions/metricproof.svg)](https://pypi.org/project/metricproof/)

**Executable checks for the assumptions behind business metrics.**

MetricProof is a small, dependency-free Python package and CLI for detecting
analytical failures that ordinary schema tests can miss: a KPI calculated from
the wrong denominator, customers silently lost before evaluation, segment
totals that do not reconcile, and malformed retention cohorts.

It was built from a practical SaaS retention problem: a dashboard value can
look plausible while still being misleading because its population or
denominator changed upstream.

## Why this project is different

Most data-quality tools are strong at generic checks such as nulls, types,
ranges, and duplicate rows. MetricProof has a narrower purpose: it turns
**metric definitions into executable evidence**.

- `ratio_consistency` recomputes a published rate from its numerator and
  denominator.
- `population_preserved` catches survivorship bias when inactive or unmatched
  entities disappear before analysis.
- `cohort_integrity` checks frozen denominators, period-zero retention, valid
  rates, and monotonic strict-retention curves.
- `reconciliation` proves that headline and segmented totals tie out.
- `unique_grain` and `numeric_range` protect the table structure and metric
  bounds supporting those business assertions.

The checks are deliberately narrow. They complement schema validation by
testing whether a published analytical result still matches its declared
population, grain, and calculation.

## Quick start

MetricProof requires Python 3.11 or newer and has no runtime dependencies.

```bash
python -m pip install metricproof
metricproof --help
metricproof --version
```

To run the included SaaS retention audit:

```bash
git clone https://github.com/AtomicGlance/metricproof.git
cd metricproof
metricproof audit examples/retention_contract.json
```

Expected result:

```text
MetricProof: SaaS retention metric audit
Result: PASS | 6 passed, 0 failed, 0 errors

[PASS ] one-headline-per-month (unique_grain)
        1 rows match the declared grain.
[PASS ] activation-rate-bounds (numeric_range)
        All 1 'activation_rate' values are within range.
[PASS ] activation-rate-recomputes (ratio_consistency)
        All 1 reported rates recompute correctly.
[PASS ] segment-mrr-ties-to-headline (reconciliation)
        The two sum aggregates reconcile.
[PASS ] eligible-population-survives (population_preserved)
        Evaluated population preserves 100.0% of baseline entities.
[PASS ] retention-cohorts-are-valid (cohort_integrity)
        All 2 cohort(s) preserve their analytical contract.
```

Run the intentionally broken example to see the audit expose four subtle
failures:

```bash
metricproof audit examples/broken_retention_contract.json
```

The command exits with `0` when critical checks pass, `1` when a critical check
fails, and `2` when the contract or input cannot be read. That makes it useful
as a lightweight CI quality gate.

## Analytical contract

Checks live in a readable JSON contract. Dataset paths are resolved relative to
the contract file.

```json
{
  "contract_version": "1.0",
  "title": "SaaS retention metric audit",
  "datasets": {
    "headline": "data/headline.csv",
    "baseline": "data/baseline_accounts.csv",
    "evaluated": "data/evaluated_accounts.csv"
  },
  "checks": [
    {
      "id": "activation-rate-recomputes",
      "type": "ratio_consistency",
      "dataset": "headline",
      "numerator": "activated_accounts",
      "denominator": "eligible_accounts",
      "rate": "activation_rate"
    },
    {
      "id": "eligible-population-survives",
      "type": "population_preserved",
      "baseline": "baseline",
      "evaluated": "evaluated",
      "key": "account_id"
    }
  ]
}
```

Reports can be emitted for people or automation:

```bash
metricproof audit examples/retention_contract.json --format json
metricproof audit examples/retention_contract.json \
  --format markdown --output audit-report.md
```

JSON and Markdown reports also record a dataset inventory: the
contract-relative source path, row count, file size, and SHA-256 fingerprint.
This makes an audit reproducible and lets a reviewer confirm which exact
extracts produced the reported result, even when the source files are
regenerated later.

MetricProof 0.2 reports use a versioned, domain-neutral evidence envelope. The
JSON output includes `schema_version`, `report_type`, producer identity,
artifact fingerprints, structured check results, and report-specific context.
Print the bundled schemas or validate a contract before loading its data:

```bash
metricproof schema contract
metricproof schema evidence
metricproof validate-contract examples/retention_contract.json
```

## Compare saved audits (0.3)

Save audit JSON before and after a data or contract change, then compare:

```bash
metricproof compare previous.json current.json
metricproof compare previous.json current.json --format markdown --output changes.md
metricproof compare previous.json current.json --format json
```

The Python API is `compare_reports(previous_report, current_report)`, accepting
decoded report dictionaries or MetricProof report objects. It reads no datasets
and executes no plugins. Results expose `changes`, `warnings`, `regressions`,
`current_failures`, `needs_review`, `exit_code`, and `to_dict()`.

Checks are matched by `check_id`, artifacts by `name`; duplicates are errors.
Timestamps and result/artifact ordering are ignored. Configuration arrays retain
their order because order can affect a plugin's meaning. Definition fingerprints
use sorted-key compact JSON with non-finite numbers prohibited. An explicit
default option and an omitted option may be flagged as different conservatively.

Audit reports now record each definition and its SHA-256 fingerprint under
`context.contract.checks`, together with runner identity/version. This is an
additive use of evidence schema 1.0's open context; existing consumers and older
reports remain supported. Registered custom runners without distribution metadata
have an unknown version. Entry-point plugins record their distribution version.

Comparison exit codes:

- `0`: no new blocking result or review requirement. An unchanged existing audit
  failure remains visible under `current_failures`; this is not a clean-audit gate.
- `1`: a new critical failure/error, escalation to an execution error, removed
  critical check, changed definition/type/severity/runner/producer/contract version,
  or unknown comparability requires attention.
- `2`: malformed input, contradictory summaries, duplicate identifiers, unsupported
  schema/report type, or a file I/O error.

Input fingerprints changing alone do not block. Removed checks are reported as
removed, never as improvements. Old reports missing definition provenance require
review. Changed observed/expected values are shown without automatically blocking.
Only `metric-audit` reports are supported; NWB evidence semantics need a separate
adapter. Fingerprints detect differences, not authenticity or scientific validity.

For CI, run the current audit and comparison as separate steps so existing audit
failures still block. Preserve the baseline from a trusted previous build, rather
than replacing it automatically with the new output:

```yaml
- run: metricproof audit contract.json --format json --output current.json
- if: always()
  run: metricproof compare baseline.json current.json --format markdown --output changes.md
```

Reproduce population loss followed by a relaxed contract:

```bash
python examples/compare_retention.py
metricproof compare examples/comparison-output/initial.json examples/comparison-output/dropped.json
metricproof compare examples/comparison-output/dropped.json examples/comparison-output/relaxed.json
```

Both comparisons exit 1: the first loses half the eligible accounts; the second
changes the minimum coverage to 50%, causing the audit to pass but requiring review
of the changed definition. The script writes three saved audits and rebuilds its
small synthetic inputs; it overwrites only its generated `comparison-output`
directory files. These demonstrate recorded evidence, not a claim about real
customer behavior or the appropriateness of a 50% threshold.

## Check plugins

External packages can add contract check types without modifying MetricProof.
A runner receives the check definition, loaded datasets, and severity, and
returns a `CheckResult`:

```python
from metricproof import CheckResult, register_check_type

def row_count(check, datasets, severity):
    observed = len(datasets[check["dataset"]])
    expected = int(check["expected"])
    return CheckResult(
        check_id=check["id"],
        check_type="row_count",
        status="pass" if observed == expected else "fail",
        severity=severity,
        message=f"Observed {observed} row(s); expected {expected}.",
        observed=observed,
        expected=expected,
    )

register_check_type("row_count", row_count)
```

Published plugins should expose the runner through the
`metricproof.checks` entry-point group. The entry-point name becomes the
contract check type:

```toml
[project.entry-points."metricproof.checks"]
row_count = "my_package.checks:row_count"
```

## Python API

```python
from metricproof import check_population_preserved

eligible = [{"account_id": "A01"}, {"account_id": "A02"}]
evaluated = [{"account_id": "A01"}]

result = check_population_preserved(
    eligible,
    evaluated,
    key="account_id",
)

assert result.status == "fail"
assert result.evidence == [
    {"key": "A02", "issue": "missing from evaluated population"}
]
```

Inputs are sequences of dictionaries, so the package works with standard CSV
and JSON data and can also accept records from a dataframe:

```python
rows = dataframe.to_dict(orient="records")
```

## Where it fits

MetricProof is useful after analytical tables or dashboard extracts have been
produced and before their metrics are published. A JSON contract can run
locally, in a scheduled pipeline, or in CI and return a non-zero exit code when
a critical analytical assumption fails.

It does not replace source-system validation. Instead, it checks the layer
between clean source data and a trustworthy reported metric.

## Design boundaries

MetricProof intentionally stays small:

- CSV and JSON are supported directly; warehouse connections are out of scope.
- Strict retention is expected to be monotonic. Set `"monotonic": false` for
  rolling or resurrection-style retention.
- The package validates supplied analytical outputs; it does not calculate
  product KPIs or replace source-system tests.
- Version `0.2.0` introduces versioned contracts, external check plugins, and
  a stable evidence-report schema shared with domain integrations.

## Development

```bash
$env:PYTHONPATH = (Resolve-Path src).Path
python -m unittest discover -s tests -v
python -m metricproof audit examples/retention_contract.json
python -m metricproof audit examples/broken_retention_contract.json
```

On macOS or Linux, use `export PYTHONPATH="$PWD/src"` instead.

## License

MIT
