Metadata-Version: 2.4
Name: valichord-attestation
Version: 2.1.0
Summary: Canonical attestation format for AI evaluation runs
Author: ValiChord
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/ValiChord/valichord-attestation
Project-URL: Source, https://github.com/ValiChord/valichord-attestation
Project-URL: Issues, https://github.com/ValiChord/valichord-attestation/issues
Project-URL: ValiChord protocol, https://github.com/ValiChord/ValiChord
Keywords: ai-evaluation,attestation,reproducibility,merkle,rfc8785
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: jcs>=0.2.1
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: pytest-cov; extra == "dev"
Provides-Extra: inspect-ai
Requires-Dist: inspect-ai>=0.3; extra == "inspect-ai"
Provides-Extra: wandb
Requires-Dist: wandb>=0.13; extra == "wandb"
Dynamic: license-file

# valichord-attestation

> **What this is:** a Python library and an open format for attesting AI evaluation runs: a
> verifiable commitment to a run's results and per-sample outputs, with challenge-response checks.
>
> **What this is not:** ValiChord itself. [ValiChord](https://github.com/ValiChord/ValiChord) is a
> commit-reveal reproducibility protocol on Holochain, where independent validators seal their
> results before anyone reveals. This library is a separate project. It is intended to become one
> way to bring AI evaluation results into that protocol, and is not yet connected to it.
>
> Moved from `ValiChord/ValiChord/valichord_attestation` on 2026-10-19, with its full history.

```bash
pip install valichord-attestation
```


A lightweight verification layer for AI evaluation claims.

The protocol provides a **verifiable commitment over an entire evaluation trace** — its summary metrics, its per-sample outputs, and the harness configuration that produced them — together with a **probabilistic challenge-response protocol** that lets a verifier confirm faithfulness of reported results without transferring the full log.

The system enables:
- **Selective disclosure** — the holder of the log can prove individual samples on demand without revealing the rest
- **Bounded-confidence fraud detection** — the verifier picks random samples; the probability of catching a misreport grows with the number of samples requested
- **Deterministic cross-implementation comparison** — RFC 8785 (JCS) canonical encoding means two implementations in different languages produce byte-identical bundles for the same input

v1 ships the format spec, the Merkle commitment, and selective disclosure. v1.1 adds the probabilistic challenge-response. v1.2 adds `Metric.filter` for multi-filter harnesses and `Bundle.meta` for provenance-aware comparison, together with a `content_hash` function for scientific-equivalence checks. **v2 (current; package 2.0.0 on 2026-08-18, 2.1.0 adds `claim_hash` and verifier-side metric recomputation)** changes only the Merkle construction — RFC 6962 style with domain separation ([spec](spec/attestation_format_v2.md)); `build_bundle` writes v2, and v1.x bundles still verify. No format version carries a signature. Future work extends this with hardware-attested execution and zero-knowledge faithfulness proofs. The probabilistic approach is a deliberate choice given current constraints on cryptographic proof systems — full rationale in [§11 of the spec](spec/attestation_format_v1.md#11-why-probabilistic-verification-instead-of-full-cryptographic-proofs).

The format is harness-agnostic. Adapters for specific harnesses are thin converters kept outside the core format; the package ships several in `valichord_attestation/adapters/` (Inspect AI logs, lm-evaluation-harness, AILuminate, W&B, pi sessions, and an Inspect Evals stub).

Architectural context: this format was designed in response to Scott Simmons's review of [UKGovernmentBEIS/inspect_evals#1610](https://github.com/UKGovernmentBEIS/inspect_evals/pull/1610). The core feedback was that the canonical spec belongs in Valichord, not in each eval harness, and that the valuable attestation is not "I have the log file" but "this reported result is faithful to the run."

---

## Verifiable statement vs attested claim

A bundle in isolation is a **verifiable statement**: any reader can confirm the bundle's internal consistency (the Merkle root commits to the per-sample outputs; the canonical encoding is deterministic; the challenge-response succeeds against a holder of the log). But anyone could have produced the bundle — there is no built-in identity layer in the format itself.

When a bundle is committed to a signed, append-only log — for example, Valichord's Holochain DNAs (`validator_workspace`, `attestation`, `governance`) — it becomes an **attested claim**: the commit is signed by the validator's Ed25519 keypair, recorded in a tamper-evident source chain, and witnessed by independent peers. At this point the bundle carries cryptographic non-repudiation: the validator cannot later deny they made the claim.

The two layers are deliberately separable. The format is harness-agnostic and substrate-agnostic — useful in contexts beyond Valichord's protocol, and compatible with any signed-log infrastructure. Within Valichord's protocol, the signed-log layer adds the identity and witnessing properties that the format alone deliberately doesn't carry.

> **Status (2026-10-18).** This library does not write to the Valichord DHT itself; it is a
> separate, client-side on-ramp, and Valichord's protocol is the Holochain code in `valichord/`.
> The one existing link: a validator's `ValidationAttestation` has an optional
> `reproduction_bundle_hash` field for the bundle's `claim_hash`, sealed into its commitment
> (`valichord/shared_types/src/lib.rs`). Rules for any fuller integration are in
> [`docs/PROTOCOL_INTEGRATION_BOUNDARY.md`](https://github.com/ValiChord/ValiChord/blob/main/docs/PROTOCOL_INTEGRATION_BOUNDARY.md).

---

## Schema instability across the AI eval ecosystem

The AI evaluation ecosystem is in active schema evolution: inspect_evals' `evaluation_report` (merged April 2026) and the executable-report direction (`UKGovernmentBEIS/inspect_evals#910`), EveryEvalEver's aggregate schema (v0.2.2, March 2026), lm-evaluation-harness output formats across versions, METR's task-standard, HELM, and others. Each is a separate evolving target.

The protocol layer is designed to survive this churn:

- The bundle format commits only to its own minimal field set (`model_id`, `task_id`, `raw_metrics`, `outputs_merkle_root`, `samples_total`, `samples_completed`).
- Harness-specific field names are adopted in adapters, not in the core bundle. Adapters can mirror upstream field names without coupling the protocol to them.
- Adapters are versioned independently and can update without changing the bundle format.
- No runtime imports from harness libraries in the core; adapters consume harness output as data.

The strongest properties of the attestation (deterministic encoding, Merkle commitments, probabilistic challenge-response) are independent of any specific harness schema. Adapters absorb upstream churn so the core doesn't have to.

---

## Concrete example

A lab publishes a benchmark result for a frontier model — say, *"87.2% on SWE-bench Verified"* — and constructs a bundle with the canonical metric, the harness configuration, and a Merkle commitment over the per-sample outputs. The lab does not need to share the underlying 4 GB of log files publicly.

A third-party verifier (a journalist, a regulator, a competing lab) reads the bundle and wants to confirm the score is faithful. They generate a fresh challenge — *"reveal samples 17, 142, 391, 894, 1,205, ..."* — and the lab responds with those 50 samples plus their Merkle paths. The verifier checks each path against the bundle's commitment, and recomputes the headline metric from the disclosed samples to confirm it matches what the lab reported.

If the lab fabricated even 5% of their results, the verifier's 50-sample challenge catches the fabrication with probability ≈92%. The verifier has confirmed faithfulness without ever downloading the full log; the lab has demonstrated their result without exposing per-sample data their privacy or competitive position requires they not publish wholesale.

That tradeoff — *probabilistic faithfulness verification with selective disclosure* — is what the protocol is for.

---

## Installation

From the `valichord_attestation/` directory:

```bash
pip install -e .
```

For development (includes pytest and pytest-cov):

```bash
pip install -e ".[dev]"
```

---

## Quickstart

```python
from valichord_attestation import (
    build_bundle, hash_bundle, content_hash,
    merkle_proof, verify_faithfulness,
)

# 1. Build a bundle from raw harness output
samples = [{"index": i, "output": "...", "correct": True} for i in range(1319)]
bundle = build_bundle(
    model_id="gpt-4o-2024-08-06",
    task_id="gsm8k",
    raw_metrics=[{"key": "accuracy", "value": 0.847, "stderr": 0.025}],
    samples=samples,
    samples_total=1319,           # optional: assert intended run size (detects silent omission)
    meta={                        # optional: provenance metadata (v1.2)
        "repo_commit": "abc123",
        "harness_version": "inspect_ai/0.3.19",
        "command": "inspect eval gsm8k --model openai/gpt-4o-2024-08-06",
    },
)

# 2. Hash it — publish bundle_hash alongside the report
bh = hash_bundle(bundle)   # byte identity: changes if any field changes, including meta
ch = content_hash(bundle)  # scientific equivalence: stable across reruns with different meta
print(bh)   # 64 hex chars
print(bh == ch)  # False — meta block differs them; True for v1.1 bundles with no meta

# 3. Prove a specific sample (for selective disclosure to a verifier)
proof = merkle_proof(samples, index=42)

# 4. Verify the proof (verifier side — receives bundle_hash, sample, proof)
ok = verify_faithfulness(bundle.outputs_merkle_root, 42, samples[42], proof)
```

Multi-metric evals work the same way — pass multiple entries in `raw_metrics`:

```python
raw_metrics = [
    {"key": "benign_utility",  "value": 0.75},
    {"key": "targeted_asr",   "value": 0.0625},
    {"key": "untargeted_asr", "value": 0.0},
]
```

For harnesses that report multiple metrics with the same key (e.g. lm-evaluation-harness `strict-match` vs `flexible-extract` filter passes), add a `"filter"` key to each metric dict:

```python
raw_metrics = [
    {"key": "exact_match", "value": 0.73, "filter": "strict-match"},
    {"key": "exact_match", "value": 0.81, "filter": "flexible-extract"},
]
```

`build_bundle` raises `MalformedBundleError` if any `value` key is missing — absent metrics are never silently defaulted to `0.0`. Pass `samples_total` to explicitly declare the intended run size; if an adapter silently drops samples, `bundle.samples_total > bundle.samples_completed` will be directly visible. Raises `ValueError` if `samples_total < len(samples)`.

### `bundle_hash` vs `content_hash`

| | `bundle_hash` | `content_hash` |
|---|---|---|
| **What changes it** | Any field, including `meta` | Everything except `meta` |
| **Use for** | Archival, challenge-response, DHT entries | Comparing reruns, asserting reproducibility |
| **v1.1 bundles** | Unchanged — backward-compatible | Equals `bundle_hash` (no meta present) |

⚠️ `generated_at` is a top-level field, not part of `meta`, so it **is** inside `content_hash` (and so are top-level `repo_commit`, `harness_version` and `command` if you pass them). Two reruns only share a `content_hash` if you pass the same `generated_at` to `build_bundle` — by default it is the current time. Put run-specific provenance in `meta`. *(Added 2026-10-18.)*

---

## Writing a harness adapter

Subclass `AdapterBase` and implement `to_bundle`:

```python
from valichord_attestation.adapters.base import AdapterBase
from valichord_attestation import build_bundle, Bundle

class MyHarnessAdapter(AdapterBase):
    def to_bundle(self, report: dict, samples: list[dict]) -> Bundle:
        return build_bundle(
            model_id=report["model"],
            task_id=report["task"],
            raw_metrics=[{"key": m["key"], "value": m["value"]} for m in report["metrics"]],
            samples=samples,
            repo_commit=report.get("commit"),
            command=report.get("command"),
        )
```

The `InspectEvalsAdapter` stub in `valichord_attestation/adapters/inspect_evals_stub.py` shows the intended field mapping for Inspect AI once the upstream API stabilises.

---

## Probabilistic Challenge-Response

Selective disclosure (Section 5 of the spec) lets the log holder choose which samples to prove. The challenge-response protocol inverts control: the verifier picks which samples to inspect, and the holder must reveal those specific ones.

```python
from valichord_attestation import (
    build_bundle, hash_bundle, Challenge,
    build_response, verify_response,
)
import os

bundle = build_bundle(...)
challenge = Challenge(
    bundle_hash=hash_bundle(bundle),
    verifier_nonce=os.urandom(32),
    k=60,  # challenge 60 randomly-chosen samples
)
response = build_response(challenge, samples)  # holder's side
ok = verify_response(challenge, response, bundle)  # verifier's side
```

With `k=60` and a 5% fabrication rate, catch probability is ~95%. See [`spec/attestation_format_v1.md`](spec/attestation_format_v1.md) Section 6 for the full protocol (seed derivation, index algorithm, test vector) and the sensitivity table. Runnable walkthrough: [`examples/challenge_response_demo.py`](examples/challenge_response_demo.py).

---

## Running the examples

```bash
# Verify synthetic examples (no GPU required)
python examples/verify_examples.py
python examples/challenge_response_demo.py

# Real-data demo: verify the committed Mistral-7B-Instruct-v0.3 + GSM8K-100 bundle (no GPU required for verification)
python examples/mistral_7b_gsm8k_demo/challenge_response_demo.py
```

⚠️ *(2026-10-18)* The committed demo `bundle.json` files are format v1, and the demo script builds its response under the library default (v2), so the last command above is expected to fail as written; `bash examples/mistral_7b_gsm8k_demo/verify_demo.sh` rebuilds the bundle first. See the note in that demo's README.

The synthetic examples contain GSM8K-shaped and agentdojo-shaped bundles with pre-computed inclusion proofs. `verify_examples.py` recomputes bundle hashes and Merkle roots from scratch.

The real-data demo (`mistral_7b_gsm8k_demo/`) exercises the full v1.1 protocol — `samples_total=100` declared explicitly, k=20 challenge-response, tamper detection — against a committed `bundle.json`. The bundle ships with simulated fixture data so the demo runs without a GPU. Run `run_eval.sh` on a GPU and re-run `build_bundle.py --output-path ./eval_output` to replace it with real eval output. See [`examples/mistral_7b_gsm8k_demo/README.md`](examples/mistral_7b_gsm8k_demo/README.md) for full instructions.

---

## Spec

Full format specification: [`spec/attestation_format_v1.md`](spec/attestation_format_v1.md), with the current Merkle construction in [`spec/attestation_format_v2.md`](spec/attestation_format_v2.md) and what a conforming implementation must do in [`spec/conformance.md`](spec/conformance.md).

The v1 spec covers: bundle schema, canonical encoding rules, pre-rounding policy, Merkle tree construction, proof format and verifier algorithm, versioning policy, adapter interface, security considerations.

---

## Running tests

```bash
pip install -e ".[dev]"
pytest tests/ --cov=valichord_attestation
```

**608 tests, 97% line coverage** (1114 statements, 33 missed) — measured 2026-08-22, CI run 32598696410, identical on Python 3.10 and 3.13.

Both numbers come from a CI run, not from anyone's local machine. The workflow measures
coverage on every push touching this package, so the figure above can be re-derived
rather than believed: Actions → *Attestation Format* → the `Tests` step → the `TOTAL`
line.

---

## Acknowledgements

Merkle conformance vectors and independent verification of the
RFC 6962 §2.1 construction: Cüneyt Öztürk (Falsify OÜ)

---

## License

Apache 2.0 — see [`LICENSE`](LICENSE) at the repository root.
