RE-call Setup

Optional, and after the install works

Measure the number that decides when to refuse.

Nobody needs this page to get RE-call working. You need it when you want the answers to be trustworthy rather than merely plausible, and when you are ready to stop running in development mode.

01

What calibration measures

Every search produces a best match and a similarity score. Calibration decides where the line sits: above it, RE-call answers from the match; below it, RE-call says it does not know. Calibration is the measurement that puts the line somewhere defensible instead of somewhere convenient.

You give it questions you already know the answers to, and questions you already know your memory cannot answer. It runs both sets against your corpus, looks at how the two score distributions sit relative to each other, and picks the threshold that separates them best. It also reports how well they separate at all, which is the part that tells you whether the number is worth having.

This is optional, and it comes last

If you followed the install path, you already have a working memory that indexes and searches. Nothing here is required to keep that working. Come back when the corpus holds material you would be unhappy to see answered wrongly.

Why a fixed default cannot do this job

The threshold is an absolute cosine similarity, and cosine levels are a property of how an embedding model was trained, not a quantity that means the same thing across models. A number that is a sensible floor for one embedder sits below every score another embedder ever produces, where it can never fire, or above a sixth of them, where it discards real answers as though the corpus were empty.

That is why the shipped default is labelled a demonstration value rather than a setting, and why RE-call would rather refuse than quietly assume it applies to you.

02

What you get for doing it

Three things change, and the third is the one that matters.

BeforeAfter
Every result carries a DEGRADED tag, INDEX_NOT_READY on a fresh install and CALIBRATION_MISSING once a generation exists, and the CLI prints that it is using an uncertified demonstration threshold bound to no tenant, no generation and no corpus. The results carry a threshold measured on your own material, with your own embedder, and the degraded flag is gone.
RECALL_TRUST_MODE=development has to be set or searching stops. You are telling RE-call to answer despite having no evidence that it should. You can take that line out. Strict is the default, and it is the mode the product was designed for.
An abstention means the guessed threshold was not cleared. It is not evidence that your corpus lacks the answer. An abstention becomes a measured decision, with a false confident rate you have seen a number for.

The refusal is the product

An uncalibrated abstention tells you nothing, because you cannot tell a real gap in your memory from a threshold that was never right for your data. Calibration is what turns "it refused" into information you can act on.

03

Write the labelled queries

One JSON file, one array, one object per question. This is the only part that takes real thought, and it is worth the hour.

queries.json 
[
  {"query": "What is our current API rate limit?", "answerable": true},
  {"query": "What is our refund policy?", "answerable": false}
]
KeyRequiredWhat it does
query Yes The question, phrased the way someone would actually ask it. Not the words in your note.
answerable Yes true when your corpus really does contain the answer, false when it really does not. This is the label the measurement is built on, so a wrong one poisons the threshold.
relevant_ids No The chunk ids that hold the answer. The command in section 05 does not need them. The generation bound flow in section 08 canonicalises them into the query set digest.

You need both kinds, and the command enforces it

A file with only answerable questions is refused, and so is a file with only unanswerable ones. Calibration will not fit a threshold to one sided evidence, because a line drawn from examples that are all on the same side of it is not a measurement of anything. Every entry also needs both keys, or the file is refused before any work starts.

The unanswerable questions are not padding to satisfy that rule. They are the half of the file that teaches the threshold where to refuse, and refusing correctly is the whole point of the product. Write them the way your own users will stumble into them: plausible questions about your project, in the same vocabulary as the rest, that your memory simply has no answer for.

How many questions, and what makes a bad set

More is better, and balance between the two classes matters more than the total. The report you get back names how many of each it measured, and gives a separability interval rather than a single number, because the certification decision is applied to the lower bound of that interval. A thin or overlapping set produces an artifact that records why it was not certified rather than a threshold that pretends to be one.

The commonest mistake is writing the answerable questions by looking at the notes. That measures whether the embedder can match a paraphrase you wrote thirty seconds ago, not whether it can find the note from a question asked cold. Write the questions first, then check which ones your corpus can answer.

The second commonest is making the unanswerable questions obviously foreign, about the weather or a different industry. Those are easy to refuse and they flatter the threshold. Keep them close to home.

04

Practise on the sample

The package ships a small corpus and a matching labelled query file, so you can see the shape of both and prove the command runs before you build data of your own. First, find where they live.

python -c "import recall.eval, pathlib; print(pathlib.Path(recall.eval.__file__).parent)"

That prints one directory. Inside it are queries.json and corpus/. Open the query file: it is the same format as section 03, at the size a real one should be. Now run the calibration against the pair, writing the result somewhere you will not confuse with your own.

python -m recall.cli calibrate PASTE_PATH\queries.json --corpus PASTE_PATH\corpus --out sample-calibration.json

Replace PASTE_PATH with the directory the first command printed.

The threshold this produces is not yours

It belongs to the sample corpus and the sample questions. It is a demonstration that the mechanism works, not a measurement of your memory, and using it to judge your own answers would be worse than having no calibration at all, because the degraded flag would be gone and the number would still be wrong. Write it to sample-calibration.json as above, look at it, and then leave it alone.

What the command does to your database while it runs

It creates a throwaway table with a random name, indexes the corpus directory you pointed it at into that table, measures the top score for every labelled query, then drops the table again. Your real index is not read and not written. That is also why the corpus you pass does not have to be indexed beforehand: calibration indexes it itself, so the measurement is taken against exactly the material you named.

05

Calibrate your own corpus

Same command, your two paths. Run it from the folder that holds your .env, so the connection string and the embedder come from the answers you already gave the wizard.

python -m recall.cli calibrate queries.json --corpus memory/ --out calibration.json
PartWhat it is
queries.json Positional, and required. The labelled file from section 03. It is not a flag, so it takes no --queries in front of it.
--corpus The directory to measure against. Pass it. Left out, it falls back to the corpus bundled with the package, and you would be calibrating the sample without meaning to.
--out Where to write the artifact. The default is calibration.json in the current folder, and if you write it elsewhere the command prints the name of the environment variable to set. Do not read that as the search path picking it up: it does not, which is what section 06 is about. The file is read by lint --semantic and by the enterprise readiness check.

You should see, in this shape

Output
embedder:  BAAI/bge-small-en-v1.5
threshold: 0.615 (scale 0.05)
separability (AUC): 0.941 [0.857, 1.000] over 24 answerable / 21 unanswerable
FCR at default 0.50: 0.36 -> at calibrated: 0.00
saved: calibration.json

Your numbers will be different, because they are a measurement of your corpus and not of anyone else's.

Read the middle two lines rather than the threshold. Separability is how far apart the two groups of questions scored, reported as an interval because the certification decision is applied to its lower bound. The false confident rate is how often an unanswerable question would have been answered anyway, first at the shipped default and then at the threshold just measured.

If it says NOT CERTIFIED

The file is still written, and the command exits non zero. That is deliberate: the artifact records certified: false along with the reason, because deleting the evidence of why a measurement failed would leave you with nothing to act on. What it means in practice is that abstention on this corpus is not yet trustworthy, and you should not read a refusal as proof that the answer is absent. The usual cause is too few questions, or two groups that overlap. Add more of both kinds and measure again.

Command exits with a usage error, or cannot reach the database? Search and calibration problems.

06

Leave development mode

Step 5 of the install told you to add RECALL_TRUST_MODE=development to .env. This is where you take it back out. Delete that line, and delete it from the env block of .mcp.json too if you connected Claude Code.

Strict is what you get by leaving it out

There is no RECALL_TRUST_MODE=strict to type. Anything that is not the word development is strict, including a typo and including the variable being absent, so the gate cannot be opened by accident. Surrounding spaces and capitalisation are ignored, so Development does open it.

What changes: the banner about an uncertified demonstration threshold stops appearing, results stop carrying the degraded flag, and a search that cannot be backed by a calibration is refused before retrieval runs rather than answered with a warning attached. The refusal happens above the retrieval call, so a refused search never fetched any chunk text to leak in the first place.

A strict refusal names its reason with a stable code. These are an interface, meant to be automated on, so they are worth recognising.

CodeMeaningWhat to do
INDEX_NOT_READY No active generation for this tenant. Build and promote a generation.
CALIBRATION_MISSING No artifact bound to this tenant and generation. Calibrate against a labelled query set and publish it.
CALIBRATION_UNCERTIFIED An artifact exists but was never certified. A draft counts here: it is statistically certified but is not the artifact anyone chose to serve. Certify and publish a replacement. The rejected evidence is kept.
CALIBRATION_STALE A certified artifact no longer binds to the current lineage. Recalibrate. A rebuild or a privacy erasure changed the corpus fingerprint.
LINEAGE_MISMATCH The generation's pipeline or corpus fingerprint is not what the artifact bound to. Recalibrate against the current generation.
DEPENDENCY_UNAVAILABLE Something the gate needed was unreachable. The gate could not run, which is not the same as the gate finding nothing. Retry once the dependency is healthy.

If strict refuses with INDEX_NOT_READY even though you just calibrated

That is expected on a plain local install, and it is not a mistake you made. The strict gate resolves calibration through the binding between a tenant and an active index generation, and the calibration.json written in section 05 has no such binding: it cannot prove which corpus, which pipeline or which labelled question set it was measured on. The file is a real measurement of your data and the numbers in it are yours to act on. What makes strict mode pass is publishing an artifact bound to a generation, which is section 08.

Why an unbound file is never loaded silently

Because the alternative is worse than refusing. If a threshold measured on last month's corpus were picked up automatically after a rebuild, every search would look certified and none of them would be. The binding exists so that "this result was calibrated" is a claim about an exact tenant, generation, pipeline fingerprint, corpus fingerprint and query set, all of which have to agree before the claim is made.

An old unbound file can still be brought in for audit continuity. It is stored as legacy_unbound, and no search path will ever select it on its own.

07

When to do it again

A calibration is a measurement of one embedder against one corpus. Change either, and the number you measured is describing something that no longer exists.

What changedRecalibrate?Why
A new embedding model Always Cosine levels belong to the model. A threshold measured for one embedder says nothing about another, and matching vector widths do not make it transferable.
The corpus changed substantially Always New material shifts what the closest match to a question looks like, on both sides of the line. The old threshold was fitted to a distribution you no longer have.
A generation was rebuilt Always The rebuild changes the corpus fingerprint, so any artifact bound to the previous one goes stale and cannot be republished against the new one.
A privacy erasure ran Always An erasure changes the effective corpus fingerprint of every affected generation, and carries that change into replacements built from the old manifest.
You edited or added a handful of notes No Ordinary growth. Revisit when the corpus looks materially different from the one you measured, or when abstentions start feeling wrong.

Keep the labelled query file under version control beside your corpus. The human labels are the expensive part and they stay reusable; it is the retrieval measurement that has to be taken again.

08

Advanced: bound to a generation

Everything above this line is the single machine path. What follows is a different subcommand for a different job: measuring a calibration that is bound to one immutable index generation, and deciding explicitly when it starts being served. If you are calibrating your own notes on your own laptop, you do not need it.

Two commands with the same name, and one of them is documented wrongly

The top level calibrate in section 05 takes a positional queries file with --corpus and --out. It has no --generation and no --queries. The generation bound command is calibration calibrate, and that one does take --generation and --queries. The repository's docs/CALIBRATION.md prints the generation bound flags against the top level command, which fails with a usage error. Use the forms on this page.

The measurement requires a generation in state ready, active or retired, and the embedder you select has to match the model and dimension that generation stored.

python -m recall.cli --tenant acme calibration calibrate --generation gen_0a1b2c --queries queries.json

That measures and stores. It does not put anything into service. Adding --publish attempts publication after the measurement is stored, so a failed certification still leaves the evidence behind to inspect.

python -m recall.cli --tenant acme calibration calibrate --generation gen_0a1b2c --queries queries.json --publish

Publication is a separate decision from measurement, and I would keep it that way even where one person does both. A measured artifact is evidence; a published artifact is the one a tenant is serving, and at most one artifact is published per tenant and generation. Superseded artifacts and their audit events stay available, and uncertified evidence is retained as rejected rather than deleted.

The rest of the group inspects and moves those artifacts.

python -m recall.cli --tenant acme calibration listpython -m recall.cli --tenant acme calibration show cal_0a1b2cpython -m recall.cli --tenant acme calibration export cal_0a1b2c --output calibration-v2.jsonpython -m recall.cli --tenant acme calibration import calibration-v2.json

Exports are corpus data

An export carries the questions and the raw retrieval scores, plus checksums over the artifact and over the whole bundle. Treat the file with the same care as the memory it was measured against. An imported artifact comes back as a draft or as rejected; import never publishes anything by itself.

The full treatment, including states, search resolution and the grants a serving role needs: docs/CALIBRATION.md.

Where to go from here