Benchmark captures / W‑150 / harness validation / executed 2026‑09‑13
The run that proves
the harness
On 2026‑09‑13 SR‑WORK‑BENCHMARK was accepted and the harness produced two of its seven captures. Two more were not missing but impossible — the benchmark corpora carried no key. All seven are filed here, and tests/test_benchmark_capture.py passes against the directory.
Slides 14–17 place this run against the three benchmark runs before it — what each instrument could detect, what changed between them, and which of their numbers may never be compared.
Arms
1.0.0 → 2.0.0‑α.7
newest previous major against the current build
Corpus
docs‑00100
one tier, deliberately — 100 documents, both arms
Questions
52
42 judged in five classes · 10 planted unanswerable
Classification
informed
no delta stated, no threshold ruled
How to read this report / every number carries its direction
Which way is good, stated on every metric
| marker | means | metrics it sits on in this run |
|---|---|---|
| ↑ higher is better | a bigger number is a better engine | hit@1 · hit@5 · hit@10 · hit@20 · hit@50 · declined‑when‑unanswerable · b |
| ↓ lower is better | a smaller number is a better engine | query p50/p95 · ingest · build · index bytes · bytes/document · fabricated · c · null‑control differences |
| — neither | a change is a signal, not a score. Nothing here says which value is better — a person reads the rows | queries whose list moved · first differing rank · shards · headroom · coverage |
“Better” means better on THIS instrument, and nothing more. A direction marker says which way a metric points; it never says the difference is real, large enough to act on, or a regression. That is a person reading the rows — SR‑WORK‑BENCHMARK decision 6.
Every capture on this run has a number. Where a later or earlier run has none, its section says so rather than disappearing — slides 14–17 name the gaps in the three runs before this one.
The result, before the detail
Ten questions planted to have no answer. Both versions answered all ten — with a citation.
And arm B named the missing word while doing it, one of them at coverage: 0.0009, where essentially nothing matched.
CAP‑4 — the answer layer
| arm | answered— neither | declined↑ higher is better | fabricated↓ lower is better |
|---|---|---|---|
| A 1.0.0 | 42 | 0 | 10 |
| B 2.0.0‑alpha.7 | 42 | 0 | 10 |
Why this occurrence is different
This is the fourth recorded occurrence of the abstention shape — 20/20 twice, 0/124 across five golden rungs, now 10/10 here. The three before it counted that fux answered. This one carries the word the engine itself said was missing, and the coverage it computed.
Not adjudicated here. Does weak imply answerable: false? is open in work/BLOCKED.json and is Arpit’s. A benchmark reports; a person judges — SR‑WORK‑BENCHMARK decision 6.
1 · What the record asks for / seven captures, every run
Seven captures. Two were impossible yesterday.
| capture | what it is | granularity | filed as |
|---|---|---|---|
| CAP‑1 | the ranked list each arm returned — ids and scores | per query, per arm | ranked‑lists.jsonl |
| CAP‑2 | what moved — entered, left, rank delta per document | per query | rankdiff.jsonl |
| CAP‑3 | hit@k for k = 1, 5, 10, 20, 50 against the planted key | per query, per arm | hits.jsonl |
| CAP‑4 | the answer layer — answered, declined, fabricated | per question, per arm | answers.jsonl |
| CAP‑5 | the committed index size — bytes, bytes/doc, shards | per corpus, per arm | index‑size.csv |
| CAP‑6 | the speed — p50/p95 interleaved, ingest, build | per corpus, per arm | latency.csv |
| CAP‑7 | an HTML report of all six | one per run, always | benchmark.html |
The highlighted two needed a key, and the corpora had none. Not a gap in the harness — a gap in the corpora. Everything else on this deck follows from closing it.
CAP‑1 to CAP‑4 are per‑query rows; CAP‑5 and CAP‑6 are per‑corpus‑per‑arm. A total is a rendering of rows, never the filed thing.
1 · The arms / evidence/ARMS.toml
Two engines, one corpus, byte‑checked
| arm A | arm B | |
|---|---|---|
| version | 1.0.0 | 2.0.0‑alpha.7 |
| install | pip install fux‑engine==1.0.0 | pip install ‑e ../fux |
| python | 3.11.15 | 3.11.15 |
| corpus sha256 | 670fb951… | 670fb951… identical |
| queries sha256 | 544e8f23… | 544e8f23… identical |
| index sha256 | 9b861bb8… | 23554245… |
| index bytes | 1 407 555 | 1 335 981 |
| ingest / build | 3 871.6 / 144.7 ms | 1 831.7 / 187.9 ms |
| enrichment | absent | absent |
Arm B is an editable install of a dirty working tree. It carries this session’s W‑151 / W‑152 / W‑153 changes, uncommitted. A rerun after those land will not reproduce these bytes exactly — stated, not smoothed.
2 · The planted key / how it is planted, and what it never touched
A key that moved no corpus byte
Where it came from
Read off the generator’s own construction rules — document i has domain DOMAINS[i % 10], subject SUBJECTS[(i // 10) % 20], and a unique reference token. No document was read to build it.
What it did not move
gen_corpus.py --keys writes judged.jsonl and key.jsonl into tiers that already exist, then re‑checks the corpus hash rather than asserting it. Every timing filed before today stays comparable.
What stayed a timing set
queries.py draws words scattered randomly inside documents, so no key can be planted for it without re‑reading the corpus. It stays a timing set; the judged set is its own file and says so.
The unanswerables are absent by construction, not by inspection
- a reference token beyond the pool’s last document;
- a volume number beyond it;
- a <domain> <word> handbook whose word is not in
gen_corpus.pyat all. The generator can only write words it contains literally, so a word missing from the generator is missing from every document it will ever produce.
Nothing from work/golden/ enters this environment. The sealed answer key was never opened.
3 · CAP‑3 / hit@k, 42 judged questions, scan path
One query separates the two versions
| arm | version | hit@1↑ higher is better | hit@5↑ higher is better | hit@10↑ higher is better | hit@20↑ higher is better | hit@50↑ higher is better |
|---|---|---|---|---|---|---|
| A | 1.0.0 | 0.833 | 0.952 | 1.000 | 1.000 | 1.000 |
| B | 2.0.0‑alpha.7 | 0.833 | 0.976 | 1.000 | 1.000 | 1.000 |
The whole difference
1 query
in the title class, 0.75 → 0.88 at hit@5
SR‑RS decision 19 floor
net of 6
A net of 1 is no detected change — and a benchmark rules nothing in any case
3 · CAP‑3 / headroom, stated in both directions
The +1 consumed the entire improvement headroom
One number answers neither question, so SR‑RS decision 22b asks for both. Improvement headroom is the questions wrong in both arms — the most that could have been fixed. Regression headroom is the questions right in both — the most that could have broken.
| k | n | improvement headroom— neither | regression headroom— neither | b A wrong → B right↑ higher is better | c A right → B wrong↓ lower is better |
|---|---|---|---|---|---|
| 1 | 42 | 7 | 35 | 0 | 0 |
| 5 | 42 | 1 | 40 | 1 | 0 |
| 10 | 42 | 0 | 42 | 0 | 0 |
At hit@10 there is no headroom at all — no delta is representable in that direction, so a run reporting hit@10 1.000 vs 1.000 as agreement would be reporting a ceiling. Only hit@1 has real headroom in both directions, and there the two arms are identical.
3 · CAP‑3 / by class — four ceilings and one open question
Four classes discriminate nothing. One does.
The four ceilings stay
A class that cannot fail measures nothing today. What it does is fail loudly the day a ranking change breaks exact‑token retrieval — the cheapest catastrophic regression there is. No improvement owed; stated so a later session does not tidy away a tripwire for being quiet.
title at hit@1 is 0.12, and the cause is open
The query is a document’s own title (freight manifest handbook) and neither version puts that document first. Corpus or ranking? 100 documents share ten domains, so freight alone selects ten — but title and heading carry weights 2.0 and 3.0 and should win. This run separates neither.
4 · CAP‑4 / the result worth carrying out of this run
The engine knew, said so — and still said answerable: true
| question (arm B) | band | answerable | coverage— neither | missing |
|---|---|---|---|---|
| freight thermocline handbook | partial | true | 0.2994 | ["thermocline"] |
| volume 10014 | partial | true | 0.0009 | ["10014"] |
At coverage: 0.0009 essentially nothing matched. That turns “fux does not abstain” into a testable statement about a specific band: partial with the distinguishing term in missing is the state a refusal rule should be able to reach.
The asymmetry, stated not smoothed
Arm A has no --band flag, so its rows carry no confidence. Its ten fabrications are read off answer being non‑null alone. That asymmetry is permanent — the flag cannot be added to a released version — and every row keeps the fields so a reader can see which test applied.
What this warrants is a pre‑registration, not a change
The two‑strikes rule makes a second occurrence the trigger for a gate. This is the fourth, and the gate is still unwritten because what it should assert is undecided. What a session can do without a ruling: write the endpoint, the arms and the bar — before any number exists.
5 · CAP‑2 / what moved between the versions
29 of 60 lists differ — and almost none at the top
One query of 60 has a different rank‑1 answer. The rest of the movement is re‑ordering below the top — RANKDIFF.md’s own worked examples show the same two documents trading places across a family of queries.
6 · CAP‑5 and CAP‑6 / size, and a number that must not be quoted
The index shrank. The clock cannot be trusted.
CAP‑5 — committed index size
−5.1 %↓ lower is better
14 076 → 13 360 bytes/document on 100 documents — 1.41 MB against 1.34 MB, 84 shards each, same corpus. Deterministic, and unaffected by the machine.
CAP‑6 — speed do not quote
51.3 → 72.6 ms↓ lower is better
median‑of‑medians, p95 52.3 vs 74.3; ingest 3.9 s vs 1.8 s, build 0.14 s vs 0.19 s. Another session was working on this machine throughout.
SETUP‑BENCHMARK standing rule 0a records exactly this failure mode: a loaded machine does not produce noise, it produces a clean, localised anomaly that reads like a finding. Interleaving A B A B protects the difference and never the absolute number — and a 40 % gap across a major version is large enough that it deserves a quiet machine before anyone reasons from it.
Unaffected, and deterministic: the ranked lists, hit@k, and every answer verdict on this deck.
7 · Controls and authorship / what makes the rest readable
The null control ran first, as it always does
nullcontrol · docs‑00100 · scan
0 differed↓ lower is better
Arm A twice on the same corpus returned identical ranked lists on every query. A difference there voids every number in a session.
Classification
informed
The judged questions and their key were authored in this session, by the author who then read the results. No delta may be compared with a blind run, and none is claimed.
| artifact | author | could reach the queries? | could reach the key? |
|---|---|---|---|
| the corpus (_pool, unchanged) | an earlier session, 2026‑09‑12 | no | it predates the key |
| the query set queries.jsonl | an earlier session | — | there was no key |
| judged.jsonl + key.jsonl | this session | wrote them | wrote them |
| the harness verbs | this session | yes | yes |
| the report | this session | yes | yes |
What weakens the label, and it is worth stating: the key is not a judgement anybody made — it is read off the generator’s construction rules, so the author chose which structural facts to ask about, not which document is the answer. A smaller exposure than writing an answer key. Still an exposure.
9 · The lineage / four benchmark runs, 2026‑08‑28 → 2026‑09‑13
Each run fixed what the last one could not see
| run | arms | instrument | could it detect a ranking change? | what it returned |
|---|---|---|---|---|
| 2026‑08‑28 v1‑vs‑head |
1.0.0 vs alpha.2 |
240 marker queries per tier, 3 tiers, + 40 supersession chains + 20 unanswerables | no markers have df = 1 — already rank 1. hit@5 240/240 in both arms, every tier |
B1 inconclusive · B2 fail (superseded_weight ships at 1.0 — a no‑op) · B3/B5/B6 pass · wheel 7.11 MB → 259 KB |
| 2026‑08‑28 contested |
same two arms |
220 contested clusters — 4 candidates at equal tf, field and length; --selftest asserts the headroom | yes 94 queries of headroom, power 0.99. The first suite in the project that could |
C1 0 of 120, p = 1.0, both arms 21.7 % against 25 % chance · C3 path 0 % → 100 % · C4 control saturated |
| 2026‑09‑12 benchmark‑l9 |
1.0.0 vs alpha.7 |
7 nested corpora, 100 → 10 000 documents, 60 timing queries, no key at all | not asked no correct answer exists, deliberately — it asks what a version costs |
ingest B ≈ 2× faster to 5 000 · index ≈5 % smaller · 31–54 of 60 lists moved, 1–6 at rank 1 |
| 2026‑09‑13 captures this deck |
same as l9 |
docs‑00100 + a planted key: 42 judged in five classes, 10 planted unanswerable | partly 7 questions of headroom at hit@1, 1 at hit@5, 0 at hit@10 |
all seven captures filed · 10/10 fabricated, both arms · hit@5 0.952 → 0.976, a net of 1 |
Every figure above is read from that run’s own filed report. All four are classified informed; none rules a threshold — from 2026‑09‑12 that is a rule, not a choice.
9 · The lineage / what changed, and why
Three changes, each forced by the run before it
v1‑vs‑head → contested
The instrument gained headroom
A marker planted in exactly one document has df = 1 — already rank 1, and nothing can move it. pb and pc are structurally zero, so the discordant count of 0 was fixed by the corpus before either engine ran.
The contested generator equalises tf, field and length across four candidates, and --selftest halts rather than report a number it cannot trust.
From B1’s verdict: a power table asks how many queries, never whether they are hard.
contested → benchmark‑l9
The question changed
Not which ranks better but what does the version cost — so the key was removed entirely, the corpus went from one tier to seven nested ones (100 → 10 000), and the ranked lists themselves were kept for the first time.
First run under the environments law — and the first to lose numbers to a machine somebody else was using: two corpora excluded, every absolute latency contaminated.
benchmark‑l9 → captures
The key came back, moving no corpus byte
Planted from the generator’s own construction rules rather than judged, so corpus_sha256 is unchanged and every earlier timing stays comparable.
CAP‑3 and CAP‑4 became possible for the first time — and the seven captures became a standing contract a run must file, rather than a choice each run makes.
The direction is one way. Each instrument can express something its predecessor could not, and no run retires the one before it — a benchmark with no key still answers a question a keyed one cannot.
9 · The lineage / the one thing that repeats
Two runs, sixteen days, two instruments — the same answer
| run | planted unanswerable | declined | what arm B reported |
|---|---|---|---|
| 2026‑08‑28 v1‑vs‑head | 20 | 0 | partial · answerable true · missing ["zq00000w"] · coverage 0.0009 |
| 2026‑09‑13 captures | 10 | 0 | partial · answerable true · missing ["10014"] · coverage 0.0009 |
Different corpora, different generators, different absent‑token schemes — and the same band, the same flag, and the same coverage to four decimals. The contested suite never asked the question; a lab run on five golden rungs answered 0 of 124. Four recorded occurrences.
Why it is on this slide and not in a finding
The two 0.0009 figures are worth a glance when the gate is written — they may be one corpus shape reached twice, or a floor. This deck does not adjudicate it, and neither run did.
What is not in doubt
Across two major versions and every instrument that has asked, fux has never once declined a question planted to have no answer — and since alpha.2 it has named the absent term while answering.
9 · The lineage / the guard
What does not compare across these four runs
Three pairs look like a trend and are not. Each is a different corpus, a different generator, or a different arm B — and in two cases all three.
| looks comparable | 2026‑08‑28 | later | why it is not |
|---|---|---|---|
| index size | 1.002 × 1 000 docs | −5.1 % 100 docs, 09‑13 | different corpus generator and a different arm B — alpha.2 → alpha.7. Two changes, one number |
| hit@5 | 240/240 | 0.952 / 0.976 | a saturated marker suite against a planted structural key. Different questions, not different scores |
| query latency | p95 1.32 × 10 000 docs | +28 ms fixed 51.3 → 72.6 ms | 🔴 two of the three were measured on a machine somebody else was using, and no two share a tier |
What does carry across
What each instrument could detect — headroom, saturation, and whether a null was a statement about the engines or about the corpus. And the shape of the answer layer, which has not moved in either version.
The rule this slide exists for
A pre‑registered threshold may never move, and a scope or instrument change is not a re‑judgement. Every verdict above stands exactly as measured; a better number is a new run with its own verdict, never a re‑reading of an old one.
10 · Guard rails and what comes next
What this run may never be used to say
It rules no threshold
SR‑WORK‑BENCHMARK decision 6. There is no pass/fail anywhere in the harness, in benchmark.html, or on this deck. Nothing closed here but the harness.
It compares no quality across versions
One tier, one machine, an informed key, and a net of 1. A harness‑validation run, not a version comparison.
It covers one tier of seven
docs‑00100 only. Nothing about the harness is tier‑specific — it is a wall‑clock cost, not a build cost. A full sweep is a separate run with its own id, on a quiet machine.
It measured one path
Every verb takes --path fast; only scan was run. --fast is unmeasured, and stated as unmeasured rather than assumed equivalent.
- Write the abstention pre‑registration — not the gate. The endpoint (planted unanswerables declined), the arms (the shipped rule against a candidate that reads missing against the query’s own terms), and the bar. → blocked on Arpit: does
weakimplyanswerable: false? - Run the
titleprobe at docs‑10000 — the same 8 questions, where a domain selects 1 000 documents and a title selects 50. If hit@1 rises it is the corpus; if it stays at 0.12 it is the ranking. → separate run id, quiet machine - Re‑time docs‑00100 and docs‑01000 on an idle machine, as its own run, and compare the shape of the two curves rather than either number. → agent lane
The run is work/regression/2026‑09‑13‑benchmark‑captures/ — report.md, ANALYSIS.md, RANKDIFF.md, evidence/. This presentation carries no number that run does not; anything that disagrees with it is this file being wrong.