Metadata-Version: 2.4
Name: decision-contract-audit
Version: 0.1.0
Summary: Audit LLM decision points against six reliability contracts — an Inspect AI extension that replays frozen decision logs keylessly
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/CTWalk/decision-contract-audit
Project-URL: Repository, https://github.com/CTWalk/decision-contract-audit
Project-URL: Issues, https://github.com/CTWalk/decision-contract-audit/issues
Keywords: inspect-ai,llm,llm-evaluation,reliability,contract-testing,metamorphic-testing,test-oracles
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Testing
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Operating System :: OS Independent
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
License-File: NOTICE
Requires-Dist: inspect-ai<0.4,>=0.3.248
Dynamic: license-file

# Decision Contract Audit

An [Inspect AI](https://github.com/UKGovernmentBEIS/inspect_ai) extension that
audits an LLM (or any) decision point against six reliability contracts by
replaying a frozen decision log. No model calls, no API keys: every run goes
through Inspect's `mockllm` and exits with a machine-checkable report.

The six contract families:

| Family | Question it answers |
|---|---|
| `validity` | Is every replayed decision one of the declared options? |
| `stability` | Does the same state get the same decision across repeats? |
| `invariant` | Does any decision break a pre-registered forbidden-action rule? |
| `monotonicity` | Does strictly improved evidence ever move the decision *later* in the hesitancy order? |
| `representation` | Do two renderings of the same state diverge? |
| `cage` | Post-hoc certification: which caller-selected cases your cage rules cover, would override, or leave **residual** |

Scope is deliberate: this is **post-hoc certification of caller-selected cases**,
not runtime prevention. A residual is a supplied case outside every cage rule;
it is not automatically unsafe unless the caller selected it on that basis.

## Install

```bash
pip install decision-contract-audit
```

## Quickstart

The runnable example lives in the repository, not the distribution:

```bash
git clone https://github.com/CTWalk/decision-contract-audit
cd decision-contract-audit
pip install .
python examples/custom_decision_point.py
```

The example replays a synthetic loan-approval triage log with planted
violations in five of the six families and exits nonzero unless every one is
detected — it doubles as the repository's keyless self-check (the same command
CI runs).

## Wire your own decision point

A decision point is: an ordered set of options (least → most hesitant), a set
of states, decisions for those states (a replay log, a dict, or a callable),
and the rules you want enforced.

```python
from decision_contracts import run_contracts

report = run_contracts(
    decision_order=["approve", "review", "decline"],  # least → most hesitant
    states={"low": {"score": 1}, "high": {"score": 3}},
    decisions={"low": "review", "high": "approve"},   # or a log, or a callable
    invariant_rules=[{
        "name": "score floor",
        "predicate": {"field": "score", "op": "<", "value": 2},
        "forbidden": ["approve"],
    }],
    monotonic_pairs=[("low", "high")],                # optional
    cage_cases=[{"state_id": "high", "decision": "approve"}],
    cage_rules=[{                                     # optional
        "name": "hard floor",
        "predicate": {"field": "score", "op": "<", "value": 1},
        "forbidden": ["approve"],
        "fallback": "review",
    }],
)

print(report["passed"], "—", report["reason"])
for family, row in report["contracts"].items():
    print(f"{family:15} {'PASS' if row['passed'] else 'FAIL'} — {row['reason']}")
```

Expected output: every configured family passes except `cage`, which fails with
`1 valid decision(s) were residual to all cage rules` — the caller selected the
`high` approval for certification, but the toy rule (`score < 1`) does not cover
it. Decide whether that gap is acceptable or tighten the rule set.

`report["contracts"]["cage"]` carries the certification counts first-class:
`covered_count`, `overridden_count`, and `residual_count`. Supplying
`cage_rules` requires an explicit `cage_cases` set, so ordinary benign decisions
are never silently relabeled as residuals.

Malformed inputs fail closed: an unknown state, an unresolvable pair, or an
unsupported rule operator produces a failing score with a reason, never a
silent pass.

For finer control (running inside your own Inspect pipeline), use
`build_contract_task(...)`, which returns a plain Inspect `Task`.

## Provenance

- Built and verified against `inspect-ai` 0.3.248 (2026-07-20); requires
  Python ≥ 3.10, matching Inspect AI's floor (CI exercises 3.10 and 3.12).
  The scorers replay pre-recorded data through a no-generation solver, and
  the example fails unless Inspect reports zero model usage.
- The contract semantics were transferred from a private research harness in
  which they reproduced 12/12 published experimental records and 9/9
  adversarial edge cases against a frozen expectations corpus, and the cage
  certification reproduced its reference log's covered/residual split exactly
  (audit executed 2026-07-20). Those frozen artifacts contain full
  experimental history and remain in the private provenance archive; this
  repository ships only the extension and a synthetic worked example.
- Development was AI-assisted under human review — see `NOTICE`.

## License

Apache-2.0 — see `LICENSE` and `NOTICE`.
