Handbook · index-and-refer

Fux

A search index for your written knowledge — decisions, runbooks, specs, wiki pages — committed to git as text you can read, diff and review. Nothing to run, nothing to call, and no second copy of your documents.

Two ways to read this. Scroll it, or hit ▶ Present to step through it as slides from the top — what, why, how, then the detail. → / ← to move, Esc to leave, ? for the keys; p presents from wherever you happen to be reading.

New to this? Nothing here assumes you have built a search system before. Every slide opens with a plain-language line, and the vocabulary slide defines every term the rest of the deck uses. Version: this describes main — 2.0.0-alpha.2 plus the unreleased confidence-floor change of 2026-08-28.

00 · Start here

What Fux is

Rank from a small committed index · fetch content from the systems that own it · verify at answer time.

00 · Start here

Why Fux exists

The usual way to make documents searchable for an agent is a vector database. A vector database is an index, and that is not the objection. The objection is what the surrounding stack has to be true for it to work.

The bet

• For a project's own documents, lexical ranking is enough. ⚠ Held narrowly: the one thing measured here was a bundled static-embedding lane, gated at 0 fixed / 2 broken and deleted (the wheel went 6.84 MB → 233 KB). That retires that lane. It is not a result about dense retrieval in general.
• Freshness and provenance are the part that actually hurts — and a derived copy makes both harder, because nothing in it can notice the source moved on. So leave the documents where they live, keep only statistics, and re-read before quoting.
• An index plain enough to commit turns retrieval into something a human reviews in a diff and CI regression-tests byte for byte.

00 · Start here

Twelve words, before anything else

00 · Start here

How it works — the whole thing in five beats

Every slide after this one is detail on one of these five.

1. Ingest. Two committed text files — .fux/sources/dirs and .fux/sources/urls, one entry per line — say what to read. Each document is fetched, decoded, analysed into five fields, written out as statistics, and the content is thrown away.
2. The committed plane. .fux/index/ — 256 JSONL shards, one canonical line per document, full postings. Sorted and content-sharded, so git itself diffs and merges it line by line.
3. The derived plane. .fux/runtime/ — an accelerator rebuilt from those shards, never committed. Bound by a differential law: its results are byte-identical to the reference scan's, asserted over thousands of comparisons rather than spot-checked.
4. Query. ask · find · answer rank with BM25F over the five fields and hand back documents and locations. A confidence band travels with the result in --json and over MCP always; on a plain terminal it needs --band.
5. Refer. The words come from the live document today: fetch it, compare the sha, quote path:L12-L40, attach a freshness verdict. A differing sha marks the document dirty, and a narrowed re-ingest repairs the ranking.

Next: those same five beats as one diagram — then each of them in turn.

01 · How Fux works

The mechanism, end to end

• Blue — committed to git. Plain — derived and disposable. Green — the freshness loop back into ingest.
• Read it left to right, top to bottom: ingest → the committed plane → the derived plane → query → refer.

01 · How Fux works

Ingest — five steps, and the content is thrown away

• Walk — entries are deduped and sorted, so file order can never change a committed byte.
• Fetch — a thread pool bounded by min(declared, configured); network is opt-in and fenced.
• Decode — 19 built-in decoders; a consumer file overrides any of them.
• Analyse — the same analyzer the query uses. Two copies would drift, and a drifted analyzer misses silently.
• Write — same sources → byte-identical shards, on any machine, on any OS.

01 · How Fux works

The analyzer, and the five weighted fields

01 · How Fux works

Two planes — one holds truth, one holds speed

01 · How Fux works

Line ranges come from answer, never from ask

The single most confusing thing about the surface, so it goes first.

01 · How Fux works

What a citation actually is

• Not a document with a wish of luck — a span.
• A heading-delimited passage, a line range, and the sha of the bytes just read.

$ fux answer "why did we choose helix"

# Service mesh

We adopted a service mesh to stop a slow dependency taking down checkout.

## Why we chose Helix

Helix gave us per-route timeouts and circuit breaking without touching
application code. The alternative was a library in every service.

  -- docs/mesh.md:L1-L8 (sha 516bef067812, current)

"citation": {
  "id":        "file:docs/mesh.md",
  "loc":       "docs/mesh.md:L10-L13",
  "sha":       "516bef06781262a0a64b10027316c18d668eeccb",
  "freshness": "current"
}

01 · How Fux works

Four freshness verdicts

The fourth exists precisely so the other three stay honest.

01 · How Fux works

Using it

pip install fux-engine
fux setup
fux ingest && fux build

fux add docs/
fux add README.md
fux add https://wiki.corp/page
fux remove docs/legacy/
fux update

fux ask "…" --json
fux find "…"
fux answer "…"
fux doctor

• fux setup writes files you own — fux.toml, the source lists, the fetcher and decoder templates, the agent files. Commit them, edit them; fux will not overwrite them.
• fux add <URL> both records the URL and fetches it.
• fux update --check is read-only: it reports drift, offline, without fetching.

02 · Confidence

How much the index believes its own answer

• The second case is where an agent invents an answer and cites a real file while doing it.
• Every query verb therefore returns a confidence block that makes the difference machine-readable.
• It is a pure function of what ranking already produced — the query's term hashes, the df statistics BM25F needed anyway, the scored list, and (on answer only) the refer plane's verdict.
• Nothing in it fetches, samples, calls a model, or reads a clock. L1, L3 and L4 are untouched.

02 · Confidence

Five signals in, one band out

02 · Confidence

The six fields, and the one to read first

• missing — read this first. It is the difference between hedging vaguely and saying "nothing here mentions mTLS".
• coverage is idf-weighted — missing the is nothing; missing mTLS is the question.
• doc_coverage is the same idf mass measured over the top-ranked document instead of the corpus. A question whose four words sit in four different documents scores coverage: 1.0 — correctly — and it is doc_coverage that says no single document contains the question. ⚠ It reports; it does not gate (see the floors).
• separation is 1.0 when exactly one document scored — the strongest separation there is, not the weakest.
• ask and find always report unverified — "we did not look", never "it was fine".

02 · Confidence

Every number, written out

There is no one confidence formula. There are the four named signals plus doc_coverage, one shared weight underneath them, and a five-clause ladder that stops at the first clause that fires.

idf(t) = ln( (N − df(t) + 0.5) / (df(t) + 0.5) + 1 )

The signals

Q = the query's distinct term hashes        D = the top-ranked document's term hashes
N = documents in the corpus                 S = scores > 0, sorted descending

              Σ  idf(t)  for t ∈ Q where df(t) > 0
coverage    = ────────────────────────────────────          # clamped [0,1], 0 if Σ idf = 0
                       Σ  idf(t)  for t ∈ Q

              Σ  idf(t)  for t ∈ Q ∩ D
doc_coverage= ─────────────────────────                     # 1.0 when no results — never
                 Σ  idf(t)  for t ∈ Q                       # demote for "not computed"

              ⎧ (S₀ − S₁) / S₀    if |S| ≥ 2 and S₀ > 0
separation  = ⎨ 1.0               if |S| = 1               # nothing competes: the STRONGEST
              ⎩ 0.0               if |S| = 0               # separation, not the weakest

support     = |S|                                          # bounded by --top, deliberately
missing     = ⟨ surface(t) : t ∈ Q, df(t) = 0 ⟩            # the user's spelling, in order
verified    = current | stale | cached | unverified        # from the refer plane, or "we did not look"

The ladder — first true clause wins

support == 0                            → none        abstain; answerable = false
missing ≠ ∅  or  verified == "stale"    → partial     answer, and name what is missing
doc_coverage < doc_coverage_floor       → partial     ⚠ floor is 0.0 by default: OFF
separation   < separation_floor         → weak        do not answer; say what was searched
otherwise                               → grounded    use it and cite it

answerable = (band ≠ none)

• All four numbers are rounded to 4 dp and clamped to [0,1] before they are stored, so the band is computed from exactly what the caller is shown.
• stale lands in partial, not weak — stale bytes are a nameable defect, which is what partial means. A weak result has nothing identifiably wrong; the ranking simply cannot choose.
• Nothing above fetches, samples, calls a model or reads a clock. Every input is something ranking already produced, which is why L1, L3 and L4 are untouched.

02 · Confidence

Getting it out

"confidence": {
  "band":         "partial",
  "answerable":   true,
  "coverage":     0.6127,
  "separation":   0.3402,
  "separation_floor":   0.1,
  "doc_coverage": 0.4413,
  "doc_coverage_floor": 0.0,
  "support":      3,
  "verified":     "unverified",
  "missing":      ["mTLS"]
}

confidence: partial - answer, but
  say what is missing.
  Not in this corpus: mTLS.

confidence: weak - the ranking cannot
  separate the top results
  (separation 0.04, floor 0.10).

confidence: none - nothing in the
  index scored for this query.

• band and answerable are written out rather than left derivable — a consumer re-implementing the rules would be a second copy of this policy, drifting from the day it was written.
• The two _floor fields are there for the opposite reason — not so you can re-derive the band, but so you can see it is not comparable across repos that tuned differently. See Tuning the floors.
• On the CLI the block needs --band (or band = true in .fux/output.toml); the fux_search MCP result carries it always, because a tool call has no flags to pass. Absent means "not asked for" — it never means "not confident".
• Silent at grounded, on purpose. A note that fires on every healthy query is a note nobody reads by the second day.
• Never on stdout: find pipes bare paths and --json is a contract. ASCII only — a Windows console's default codepage crashes on a fancy dash rather than degrading.
• answerable is a boolean: an agent handed 0.3 uses it anyway and hedges in prose; an agent handed false has nothing to hedge with.

02 · Confidence

Which parts are facts, and which is a guess

• ⚠ It WAS deliberately not a tune.toml key — that was reversed on 2026-08-28. The argument for the lock is still true: a consumer who lowers the floor until their answers read grounded is tuning away the signal rather than the ranking, and the honest fix for a wrong floor is still to measure it once, for everyone. What changed is who decides — see Tuning the floors for the price of that.
• ⚠ separation is ordinal, and Chow's rule — the theory behind an abstention threshold — assumes a calibrated probability. The record states the gap rather than closing it.
• Confidence can never reach a score or an ordering. It is computed from rank()'s output and handed to the caller; nothing downstream feeds back.
• SR-CONFIDENCE is accepted (ratified 2026-08-27, amended 2026-08-28). What is still owed is R10 — the measurement, not the ratification.

02 · Confidence

Moving the two floors — and what that costs

Both cutoffs live in .fux/tune.toml. Neither can move a score or an ordering: confidence is computed from the ranking and nothing feeds back.

[confidence]
separation_floor   = 0.1   # the `grounded`/`weak` cutoff
doc_coverage_floor = 0.0   # 0.0 = the clause is OFF

• 🔴 The band now travels with the floor that judged it. Every block carries separation_floor and doc_coverage_floor, so a grounded at 0.02 is distinguishable from one at 0.10. Without that, exposing the knob would have made the band silently mean a different thing in every repo — worse than either the lock or the knob.
• Read the floor before comparing two bands from different repositories. Comparing them without it is a comparison fux told you not to make.
• fux ask --no-tune recomputes the band at the engine defaults — the same "is it me or the config?" switch it already was for ranking.
• ⚠ A measured run may never compare two arms with different floors. That is a pre-registered threshold moving inside a comparison, and it is the reopen trigger for this decision.
• fux setup is write-if-missing, so an existing repo never gains this table on upgrade. An absent [confidence] means the engine floors — nothing breaks; what is lost is discoverability.

03 · How it is tested

Two questions, not one

• Does the code do what it says? → the suites.
• Is what it says any good? → the quality contract.
• A run can pass the first and fail the second. It has happened twice, and a human caught both.

uv sync --extra dev
uv run pytest -q tests          # fast unit — hermetic, no network, no subprocess
uv run pytest -q tests_e2e      # the package as a user: the real CLI via subprocess

03 · How it is tested

Determinism is asserted through the CLI, on three operating systems

• Prediction R1 — double-ingest must produce byte-identical shards.
• tests_e2e/test_determinism.py runs the real fux ingest twice in a subprocess and compares SHA-256 per shard.
• It runs on the CI matrix's Linux, macOS and Windows runners — so a green run is a genuine cross-platform assertion, not a same-machine self-comparison.

03 · How it is tested

The guards that are not about code at all

A surprising share of the suite tests the project rather than the engine — the failure modes that hurt most here are documentary.

03 · How it is tested

What a fux quality number means

• Fux measured carefully for months and never wrote down what it was measuring.
• Every number therefore carried an undeclared query distribution and an implicit cost model in which a fabricated citation and an honest decline score identically.

03 · How it is tested

The six calls that make a number mean something

• recall@k is the headline — retrieval bounds the whole system, and it is the half fux fully controls. Compared at equal byte budget or not at all: recall bought with a larger window is not recall earned.
• nDCG and MRR are diagnostics — a reranker follows retrieval, and LLM attention over long context is U-shaped, so a decaying discount describes a consumer that does not exist.
• The query prior is declared and versioned, and starts uniform. The fork was never uniform-vs-weighted — it was declared vs undeclared.
• unanswerable queries are inside the gate — accuracy-only scoring does not merely fail to notice a guess, it rewards one over an abstention.
• Every verdict publishes a weight-stability interval — the range of c over which the verdict does not change.
• The judged series is pinned and never fused — it may inform a prediction, never adjudicate one.

03 · How it is tested

Measurement discipline

• An informed run is reclassified, not banned: file it, cite it — but never compare it with a blind run and never use it to state a delta.
• An informed number is not an "upper bound"; label it "not a generalisation estimate."

03 · How it is tested

What is not verified

04 · Reference

Merge — machine planes should never conflict on adjacency

• A shard is a header line plus one JSON line per document, sorted by id.
• Two branches that each added documents produce two line sets whose union is the correct answer — a textual three-way merge cannot see that.
• fux hooks --install wires a custom git merge driver: fux-merge-index %O %A %B.

04 · Reference

One boundary, used three times

• Fux refuses to own network I/O, model calls, and third-party parsing libraries.
• Each is a file the consumer writes, committed in their repo, that fux loads by path and never rewrites.
• That is how src/fux/ stays stdlib-only (L1) and offline by default (L4).

04 · Reference

Fetcher — the contract

# .fux/fetchers/<name>.py

def fetch(url: str) -> tuple[bytes, str]:
    """The bytes the server sent, plus the Content-Type it declared.
    May raise — that records the URL as skipped, never a crash."""

# optional
MAX_PARALLEL = 4                     # how many threads fux may run against you
def connect() -> None: ...           # called once before this fetcher's batch
def close() -> None: ...             # called once after
def configure(config: dict) -> None: # [sources.url.config], verbatim
    ...                              # fux never reads a key inside this table

• Routing is declared, never detected. Nothing escalates from one fetcher to another: a plain GET that returns a rendered shell returns a rendered shell, and a human writes fetch=cdp on that line.
• Network only inside two fenced paths — fux add <URL> (scoped to that one URL) and fux update. A plain fux ingest never imports a fetcher.
• Verify uses the ingest fetcher. A document fetched two ways is two documents.
• A bare str return still works — read as already-prose, so older fetchers keep running.

04 · Reference

Decoder — bytes become Markdown, or become a queue item

# .fux/decoders/<name>.py    (or a built-in of the same module name)

EXTENSIONS = (".html", ".htm")          # required; lowercase, with the dot

def decode(raw: bytes, rel_path: str) -> str | None:
    """Markdown, or None when a model is needed to read it."""

WANTS_PATH = True                       # opt-in, for a library that insists on a file
def decode(path: Path, rel_path: str) -> str | None: ...

• Markdown, not flat text — headings are their own weighted field, so the heading syntax is the interface.
• None is not an error — it means this needs a model to read it, and it is the signal the enrichment queue is built on.
• A decoder never raises for malformed input — one corrupt file in a 10 000-document corpus must not stop the other 9 999.
• Adding a decoder must not change what is indexed — the default type allowlist is derived from built-ins only.

The nineteen built-ins, all stdlib-only: csvdoc · docxdoc · drawiodoc · htmldoc · imagedoc · inidoc · ipynbdoc · jsondoc · jsonldoc · maildoc · odtdoc · pdfdoc · pptxdoc · rtfdoc · svgdoc · tomldoc · xlsxdoc · xmldoc · yamldoc.

04 · Reference

CLI — setup and corpus

04 · Reference

CLI — read, maintain, serve

• --scan is the default — the reference path, no build step needed. Kept as an explicit flag because it is what a bug report reproduces against.
• --fast uses the accelerator when one exists: same results, only faster.
• Exit codes — 0 ok · 1 error · 2 blocking (strict) · 130 interrupted.
• Errors are rendered only at the boundary; one FuxError, no subclass hierarchy.

05 · Context

Running it as an agent

05 · Context

The eight laws

Not guidelines. Every record cites them by number and none restates them — a paraphrase drifts, and a drifted paraphrase in an accepted record reads as authority.

05 · Context

Decisions — made, and still open

• The pruned index is dead, not deferred — a pre-registered gate closed FAIL. Full postings, permanently.
• The dense lane and its model were deleted — 0 fixed / 2 broken at every setting that fires. The model mean-pooled static vectors, so the lane was as order-blind as BM25F.
• A cross-encoder reranker was refused — onnxruntime is not byte-identical across architectures, which would break clone it and run the query.
• The post-commit hook defers — an inline re-index cost 44 s on a 20-document commit, because cost tracks corpus size, not delta size.
• Answer-time fetching is unconditional — and that single call withdrew two proposed flags outright.
• No local content store — copying the corpus blunts the wedge for two narrow wins.

05 · Context

Where things live

Fux 2.0.0-alpha.2 · rank from a small committed index · fetch from the systems that own it · verify at answer time.