Version benchmark / W-93 / executed 2026-08-28
fux‑engine 1.0.0
vs working‑tree HEAD
Two engines resident at once, over byte‑identical corpora, at 100 / 1 000 / 10 000 documents. Seven pre‑registered thresholds. 3 000 per‑query rows.
Arm A
1.0.0
fux.index.v1
Arm B
2.0.0‑a2
75ade57, fux.index.v2
Paired tests
7
exact McNemar, α = 0.05
Classification
informed
no delta stated
The result, before the detail
Every pre‑registered paired test returned a discordant count of zero.
Not a small delta. No query changed its outcome in either direction — on any tier, on any suite.
Finding 1 — the instrument
The primary endpoint was saturated before it ran. hit@5 came back 240/240 in both arms at every tier. A marker term with df = 1 is already rank 1; nothing can move it up and nothing moved it down.
So B1's null is a property of the queries, not a measurement of the engines.
Finding 2 — a default
superseded_weight ships at 1.0. HEAD parses supersedes:, builds the edge, resolves the flag onto the retired document — then multiplies its score by one.
Both arms invert identically. The machinery works and is switched off.
Everything that could be cleanly compared — bytes, wheel size, query latency, ingest latency — passed its fence with room.
1 · What was tested / the arms
Two engines, not two configurations
The committed record shape changed between them, so there is no shared index and there cannot be one. Each arm ingests the same corpus bytes and builds its own. Every result is end‑to‑end — ingest → index → rank → answer — never ranker‑only.
| Arm | What it is | Record shape | Lanes it has that the other does not |
|---|---|---|---|
| A | fux-engine==1.0.0the first major release | fux.index.v1 2 tf fields | the dense lane behind ask --hybrid — deleted 2026‑08‑25 after measuring 0 fixed / 2 broken. Not run here: benchmarking against a lane removed for being broken would flatter B. |
| B‑core | pip install -e at 75ade57shipped defaults only | fux.index.v2 5 tf fields, v2 analyzer | proximity reranking · supersedes: priors + commit recency · tune.toml · enrich · mcp · confidence bands · .fuxignore · ask --why |
| A′ | arm A again, second corpus seed | v1 | the null control — a halt gate, not a finding |
| B‑tuned | arm B + one tune.toml key | v2 | post‑hoc, carries no verdict and no version claim |
Both arms: CPython 3.11.15, own venv, archived_weight at its 1.0 default, no .fux/enrich, same machine, same session.
1 · What was tested / the thresholds
Seven frozen thresholds across four planes
| ID | Plane | What it measures | N | Bar, frozen before any number existed |
|---|---|---|---|---|
| B1 | quality | hit@5, A vs B‑core, tier 1 000 — the primary endpoint | 240 | improvement needs p < 0.05 and b > c |
| B2 | quality | supersession inversions — does a retired document outrank its successor? | 30–40 | as B1 |
| B3 | size | committed index bytes, shards, published wheel | — | B‑core bytes/doc ≤ 1.25 × A |
| B5 | latency | warm ask p50/p95, scan path, tier 10 000 | 1 200 | B p95 ≤ 1.5 × A p95 |
| B6 | latency | cold ingest and build wall‑clock | 3×2 | B ingest ≤ 2.0 × A |
| B7 | answer | honest decline on planted unanswerables | 20 | as B1 |
| B9 | control | A vs A′ — a halt gate, run first | 240 | discordant count 0, or halt |
B4 and B8 do not exist. The queue item and the register both said “thresholds B1–B9”. The frozen document defines seven. Nothing was skipped — two ids were never written, and that is worth knowing before someone hunts for them.
2 · How it was tested / the pipeline
Gates first, on purpose
2 · How it was tested / the corpus
Four structures, planted deliberately
Generated by the lab's own generator — seeded, byte‑identical for the same seed, and extended for this run with three structures it did not previously emit. The pairs are emitted mechanically from the planted facts, which is what makes 720 graded queries feasible at all.
Markers → B1
240
a rare term planted in exactly one document, per tier
Chains → B2
40
two documents, same marker; the newer declares supersedes:
Decoys → control
50
topically adjacent, factually silent — no marker anywhere
Unanswerable → B7
20
half absent‑entity, half compositional
The one design detail that decides whether B2 means anything
A chain's two halves are given unrelated document numbers and their old/new roles are drawn from the seeded stream. Without that, an engine breaking ties by path would score 0 % or 100 % inversions — and the number would be measuring sorted().
The limit, stated up front rather than discovered later
On a generated corpus “unanswerable” means no document holds the queried marker — not no true answer exists. The base documents state no facts at all. This instrument can show an answer layer declining when nothing matches; it cannot show it declining when something matches and does not support the claim.
2 · How it was tested / the statistics
Paired arms means the bar tracks the flips
Two arms graded on the same queries is a paired comparison, so only the queries that change outcome carry information. The set size does not set the bar; the discordant count does.
Exact two‑sided McNemar
A binomial on the discordant pairs only, computed from the filed per‑query rows and from nothing else. The harness reproduces the project's own resolution floor as a self‑check: net 6 → p = 0.031, net 5 → p = 0.063, 20 flips at net 10 → p = 0.041.
Why 240 queries per tier
Simulated power at α = 0.05: at N = 50 a 10 %‑fixed / 3 %‑broken effect is detected 14 % of the time. At N = 240, 83 %. Four previously filed runs were not unlucky — at their set size they could not have detected anything.
That table answers only half the question, and this run is how we found out. See slide 12.
2 · How it was tested / the discipline
Five rules that were followed, not intended
The threshold was written first, and committed
The pre‑registration was frozen by an earlier session. The HEAD sha was written into it and committed in its own commit, before the first command ran.
The null control ran first
Arm A twice on one corpus: 300/300 identical rows. Arm A vs A′ on a second seed: 0 discordant of 240. No A‑vs‑B number existed when this was ruled.
Arms interleaved, never sequenced
A B A B, 1 200 timed queries per arm, 20 warm‑up queries discarded. Thermal drift on a laptop is real, and AAAA BBBB hands the second arm a hotter machine.
Per‑query rows written as the run went
One row per query per arm per tier — 10 files, 3 000 rows. The first run in this project's evidence store to have them. Every aggregate is derivable from the rows and from nothing else.
The run is classified informed, and that was ruled by the pre‑registration rather than by judgement after the fact. Its §3 says: “If the same session writes the generator and reads a score, the run is informed and no delta may be stated from it.” That is exactly what happened, so no delta is stated. The specific exposure: the “decline” observable for B7 was chosen after seeing that both arms return three passages for an absent term.
3 · Results / the scoreboard
Seven verdicts
| ID | Verdict | Measured | Reading |
|---|---|---|---|
| B1 | Inconclusive | 0 discordant of 240 · p = 1.0 | hit@5 240/240 in both arms at every tier — the endpoint was saturated, so the null was determined by the corpus. |
| B2 | Fail | 0 discordant of 40 · 21/40 inversions in both arms | Predicted PASS with B better. B fixes nothing — superseded_weight ships as a no‑op. This endpoint did have power. |
| B3 | Pass | 1.002 × bytes · wheel 0.036 × | Bar was 1.25 ×. Wheel goes 7.11 MB → 259 KB. No int8 vectors committed. |
| B5 | Pass | p95 85.6 → 112.6 ms · 1.32 × | Bar was 1.5 ×. Inside the fence, and 1.32 × is not nothing. |
| B6 | Pass | ingest 25.68 → 26.78 s · 1.04 × | Bar was 2.0 ×. One 38.8 s outlier in arm B is filed, not dropped. |
| B7 | Inconclusive | 0 declines of 20, both arms | As predicted — but this corpus can only test the weak form of the question. |
| B9 | Pass | 0 discordant · 300/300 identical rows | The halt gate. Run first, passed, and everything downstream depends on it. |
Also collected and clean: the differential law holds in both arms — ask --fast and ask --scan byte‑identical across all 240 queries, 0 mismatches. Decoys reached the top 5 for 1 of 50 shadowed queries at tier 1 000 and 1 of 208 at tier 10 000 — identically in both arms.
3 · Results / quality
The retrieval numbers, all of them
| Tier | Arm | hit@5 | hit@10 | MRR@10 | rank‑1 | chain inversions | declines |
|---|---|---|---|---|---|---|---|
| 100 | A | 240/240 | 240/240 | 1.0000 | 240/240 | 5/10 | 0/20 |
| 100 | B | 240/240 | 240/240 | 1.0000 | 240/240 | 5/10 | 0/20 |
| 1 000 | A | 240/240 | 240/240 | 1.0000 | 240/240 | 21/40 | 0/20 |
| 1 000 | B | 240/240 | 240/240 | 1.0000 | 240/240 | 21/40 | 0/20 |
| 10 000 | A | 240/240 | 240/240 | 1.0000 | 240/240 | 17/40 | 0/20 |
| 10 000 | B | 240/240 | 240/240 | 1.0000 | 240/240 | 17/40 | 0/20 |
Read the columns, not the rows: every retrieval column is identical between the arms, and three of them are at their ceiling. The inversion column is the only one carrying variance — and it is the same variance in both arms, because it comes from the corpus.
Tier 1 000 is the pre‑registered primary. Tiers 100 and 10 000 are secondary and descriptive — named once, in advance, rather than carrying a multiplicity correction.
3 · Results / size and speed
Everything cleanly comparable passed
Committed index, 10k docs
0.998 ×
14.15 MB → 14.12 MB. At 1 000 docs, 1.002 ×. Bar was 1.25 ×.
Published wheel
0.036 ×
7 113 352 → 258 901 bytes. The bundled model left in alpha.1.
Cold ingest, 10k docs
1.04 ×
25.68 s → 26.78 s median. Bar was 2.0 ×.
Accelerator build
1.77 ×
612 ms → 1 086 ms. Not a pre‑registered bar; recorded for the record.
B3's one named check is answered: no. HEAD does not commit per‑chunk int8 vectors. Read from a document record in each arm's index — arm A carries a code key, arm B does not, and neither carries vectors. Five committed tf fields replace two for 0.2 % more bytes.
3 · Results / finding 1
A power table says how many queries. It never says whether the queries are hard.
The pre‑registration sized the query set correctly and rigorously — and the set still could not detect anything, because of a property the sizing never looked at.
A marker term planted in exactly one document has df = 1. It is the easiest retrieval problem BM25 has. It is already at rank 1, so a five‑field BM25F, a v2 analyzer and a proximity reranker cannot improve it — and cannot break it either.
In McNemar's terms: pb and pc are structurally zero. The discordant count of 0 was fixed by the corpus design before either engine ran.
The two suites that did discriminate both had contested answers: chains, where two documents compete and the inversion rate is a coin flip, and decoys, where a topically adjacent document occasionally reaches the top 5.
3 · Results / finding 2
The supersession prior works. It ships switched off.
tune.toml separates the middle bar from the right one.superseded_weight defaults to 1.0 and recency_half_life_days to 0.0. On shipped defaults both priors are multiplicative no‑ops.
The queue item predicted that a B2 failure would mean “the priors shipped and do not do the job they were built for.” They do the job. Nobody turned them on.
This is not an argument for changing the default, and reading it that way would be a real error.
A previous verdict — P‑SUPERSEDE, 2026‑08‑25 — already ruled exactly that change FAIL on the hand‑graded playground: at 0.5 it fixed two (q015, q049) and broke two (q022, q033), and every broken query had the superseded document as its correct answer.
This corpus cannot see that failure mode by construction — every planted chain query's right answer is the successor. So 0 broken here is a property of the generator. The two results are consistent, and the older one is more informative because its corpus contains the case that breaks.
3 · Results / finding 3
Neither engine ever declines
For a query whose term appears in no document anywhere, both arms returned three passages. 0 declines out of 20, at every tier, in both arms.
Arm B is not blind to it
It has the information and does not act on it. On every absent‑entity probe:
band: partial answerable: true coverage: 0.0009 doc_coverage: 0.0009 missing: ["zq00000w"]
Arm A emits no confidence block at all, so this half is a capability delta, not a comparison. Arm B returned partial on 20 of 20.
Why the verdict is inconclusive rather than a clean null
The comparison resolved exactly as predicted. But the question the threshold was written to answer — is the answer layer more honest? — is not what this corpus can ask.
It can test declining when nothing matches. It cannot test declining when something matches and does not support the claim, which is the failure that motivated the threshold in the first place.
Do not read this null as reassurance. A separate blind run on the hand‑authored playground found the same thing at severity: the engine reported answerable: true on 20 of 20 genuinely unanswerable questions.
4 · Recommendations
What to do about it
- Build a contested‑answer suite, and retire the marker suite as a quality endpoint. A marker query has one plausible answer and measures whether the index is wired up — keep it as a null‑control instrument, where it is excellent. A quality endpoint needs several documents plausibly matching and one correct. Until that suite exists, no run in this project can detect a ranking improvement. → filed as W‑95, agent lane
- Size every future paired run against power and headroom. Add one line to the pre‑registration template: what is the current score on this suite, and how many queries could change? A suite already at 100 % has a maximum detectable effect of zero, whatever the sample size says.
-
Close the supersession disclosure gap — but do not touch the default.
A corpus that declares
supersedes:today gets nothing from it and is told nothing.fux doctorcan say the prior is disabled; that is a disclosure, not a ranking change. Changing the default is the option P‑SUPERSEDE already failed, and its diagnosis still stands: supersession belongs to the query's intent, and a per‑document multiplier cannot express it. → filed as W‑94, Arpit's call — and doing nothing is a legitimate choice - Make a blind benchmark structurally possible. One session authors and freezes the generator, the query sets and the harness, then stops. A second session, which never reads them, executes and analyses. One session cannot do it — the pre‑registration said so in advance and was right. → filed as W‑96, agent lane
-
Teach the lab and playground harnesses to emit per‑query rows. The benchmark harness now does, and a gate now enforces it from 2026‑08‑29.
fux-lab/shared/regress/run.pystill emits totals only — the next lab or playground run filed will hit the gate with nothing to give it. One emitter, not a rule. -
Re‑run against the next release, and expect it to be cheap. The harness, corpora and arms are standing. A new
HEADis a new run citing the same frozen pre‑registration — never an edit to it. Total wall‑clock for the quality half of this run was under four minutes; the latency half was eight.
Worth saying / the guard rails
What this run may never be used to say
It cannot say the two engines retrieve equally well
The primary suite was saturated, so the observed equality is a property of the queries. The only suites that could have discriminated were the chains — equal because a default is off — and the decoys, where both arms are near‑perfect.
It cannot attribute anything to a feature
Two engines differ in a dozen places at once. A win would say the latest engine is better end‑to‑end on this corpus, never the proximity reranker works. Attribution needs an ablation, which is a different run.
It cannot state a delta at all
It is classified informed. It may be filed, listed, cited and used to inform the corpus. It is not a generalisation estimate, and the post‑hoc superseded_weight result carries no verdict.
It cannot compare wall‑clock to anything elsewhere
B5 and B6 were measured on one machine in one session, macOS arm64, CPython 3.11.15. A number measured on another surface is not comparable, and one arm's third ingest repeat was a 38.8 s outlier — filed, not dropped.
A recorded negative is a successful outcome. Four of the seven thresholds resolved to “no detected change” or worse, and the two that failed are the two most useful results in the run. The alternative — a suite tuned until it produced a pleasing direction — is the failure this whole apparatus exists to prevent.
Worth saying / side findings
Three things found on the way
Harness
fux setup anchors on the git root
Not on cwd. The first run wrote .fux/, .claude/, .github/ and .kiro/ into the harness repository and left the corpus with no index — and the next step failed with a FileNotFoundError, which reads like a broken harness rather than a working directory in the wrong place. Both arms do it, so it is not a version difference.
Suite
The test suite was red, not green
tests/test_setup_docs.py was failing when this session started: the benchmark setup document had been added the day before without its index row, and the previous session's handoff note claimed a fully green suite. Fixed in the first commit.
Contract
Seven verdicts were guarded by nothing
The per‑run contract test globbed VERDICT.md only. A run ruling on seven thresholds files seven verdict files, and six of them would have been filed, cited and checked by nothing — the worst state a verdict can be in. The glob is now VERDICT*.md.
And one honest note about this session's own bookkeeping: a concurrent session was editing the same repository, and a git add -A swept 152 lines of its in‑progress work into this run's commit. The content is preserved and committed; the commit message does not describe it. Left as it is rather than unpicked, because rewriting under a live session is the worse risk.
Reproduce
Everything here is re‑runnable
The evidence
work/regression/2026-08-28-benchmark-v1-vs-head/
report.md classification: informed
ANALYSIS.md findings + repro per finding
VERDICT.md B1 · VERDICT-B2.md B2
VERDICT-B3.md · -B5 · -B6 · -B7 · -B9
evidence/
rows/ 10 files · 3 000 per-query rows
eval/ every query, all four tiers
ARMS.toml the frozen arm manifest
harness/ bench.py · latency.py · make_corpus.py
latency-summary-t10000.json
The commands
python3 shared/generate/make_corpus.py \ --out corpora/t1000 --docs 1000 --seed 12 \ --bench --pairs 240 --chains 40 \ --decoys 50 --unanswerable 20 python3 bin/bench.py prepare --run R --arm A --tier t1000 python3 bin/bench.py quality --run R --arm A --tier t1000 --label A python3 bin/bench.py mcnemar \ --a rows/A-t1000.jsonl --b rows/B-t1000.jsonl \ --suite pairs --key hit@5 python3 bin/latency.py --run R --tier t10000 \ --queries 240 --repeats 5
The generator ships a --selftest that asserts determinism, that no “unanswerable” marker was quietly planted, that decoys carry no marker, and that every chain's newer half declares its older one.
Harness: ~/my_programs/fux-benchmark · corpora sha256‑recorded in ARMS.toml · frozen thresholds: work/benchmark/PRE-REGISTRATION-V1-VS-HEAD.md, HEAD = 75ade57.