EvidenceBench v0.1 — groundedness, not prediction

Can your model answer questions about real SEC disclosures with verifiable grounding — and abstain when it cannot? Items regenerate weekly from the newest post-cutoff evidence, so no static answer key can have been memorized from a training corpus: the answers did not exist at training time. That property is mechanical, not aspirational — keys ship openly.

Current release: 750 items · generated 2026-09-18 · canonical item-set hash e96ea109ea55c228… · generation spec registered as protocol 6adb3a06f578 · window: trailing 7 days (post-cutoff, regenerated weekly)

How to run (current: the scorer in the installed package)

The scorer ships inside the package (since 7.0.1); nothing has to be cloned. It is standard-library only and reads exactly the item file you give it:

pip install yuclaw
curl -sO https://yuclaw.ca/evidencebench/items.jsonl
curl -sO https://yuclaw.ca/evidencebench/meta.json
yuclaw evidencebench score predictions.json "your-model-name" --items items.jsonl --rubric v1 \
       --expect-item-set-hash e96ea109ea55c228e8066080e8fb72f6d0e967b4ed32f9152f255fc44ae9a01e

predictions.json maps item_id → answer string; the literal abstention string is "cannot verify from the evidence provided". With meta.json beside the items the identity is checked automatically; a mismatch is a refusal (exit 3), never a score.

Historical reproduction only: release 7.0.0 did not ship the scorer; its numbers reproduce from a pinned checkout (git clone --branch v7.0.0 --depth 1 https://github.com/YuClawLab/yuclaw-brain.git, then python3 -m tools.yuclaw_evidencebench score /absolute/path/predictions.json "name" from its root, against the item file of that checkout). That recipe is kept for the record and is not the current instruction.

What exists, stated separately

Negative control on this release (computed at page build with the packaged scorer, on the 750 items identified below; every answer is the question itself): rubric v1 gives T1 = 1.0 and aggregate 0.4; rubric v2 gives T1 = 0.0 and aggregate 0.0. It shows v1's disclosed flaw and that v2 rejects a question echo; it is one control, not a validation of v2.

Item-set identity and lineage

This release: 750 items, generated 2026-09-18, window: trailing 7 days (post-cutoff, regenerated weekly). Two different digests identify it — do not compare one with the other:

The item set is regenerated every week from the evidence of the trailing window, so each release is a different set and the count moves with that week's evidence volume; nothing is migrated from one release to the next. Published releases (lineage.json; earlier entries come from the repository history of meta.json and name their commit):

Leaderboard

The only current row is our own extraction stack scored against its own corpus — a format demonstration, loudly labeled self-evaluation; nothing is claimed by it.

Dataset

Items + keys: items.jsonl (JSONL — one item per line: {item_id, template, question, key}) · meta.json. Under the export rule: derived events, verified excerpts, and keys only — no raw vendor OHLCV is published. Weekly snapshots are tagged in the repository (dataset citability: see CITATION.cff at the repo root and the replication page).