How RNSR DocDB Works

1. What DocDB is, in one paragraph

DocDB is the heart of RNSR. It reads your documents once — PDFs, Word files, spreadsheets, emails, plain text — and packs everything it finds into a single database file called corpus.db. That file contains the complete original text of every page, plus useful extras layered on top: every table converted into a real, queryable database table; a full-text search index; and a machine-built "table of contents" (the manifest). When you ask a question, an AI model doesn't just read the documents — it writes small Python programs against that database to look things up, count, add, and cross-reference. The final answer must come with supporting quotes that the system checks, character by character, against the original text.

2. The problem it solves

Suppose a lawyer has a matter file: a master services agreement, two amendments that override parts of it, forty invoices, a chain of breach letters, and a superseded draft with the wrong numbers in it. Now they ask: "What is the total value of all unpaid invoices?"

The two standard AI approaches both struggle here:

DocDB's answer: don't make the model read everything, and don't gamble on fetching the right snippets. Put the documents in a database and let the model query it. Summing forty invoices becomes one SQL statement that is exact by construction. In RNSR's benchmarks, this is precisely where DocDB wins: it went 24/24 on multi-document matter files while every retrieval variant missed all of the aggregation questions.

3. The big picture: two phases

Everything in DocDB happens in one of two phases. Ingestion runs once per set of documents and produces the database file. Querying runs every time you ask a question, and only needs that file.

A key design commitment: the database only ever adds views on top of the original text — it never replaces or throws any of it away. The full text of every page is retained inside corpus.db, and every index entry, table row, and search hit can be traced back to an exact character range on an exact page of an exact document.

4. Phase 1 — Ingestion: from files to a database

You run ingestion with one command:

rnsr ingest report.pdf exhibits.docx ledger.xlsx -o corpus.db

Behind that command is a pipeline. Here is what happens to each file, step by step:

The steps in plain English

Pick a parser. PDFs go through a high-quality PDF parser (Docling), or a faster one when speed matters. Office documents (Word, Excel, PowerPoint and friends) go through an office-format converter. Markdown, plain text, and email files are handled directly. The choice is made purely by file extension.

Extract the text. The parser produces the full text of every page, recording exactly where each piece came from. If a page is a scan with no digital text, a vision-capable AI model reads the image and transcribes it.

Cut it into chunks. Search works better on passages than on whole documents, so the text is cut into pieces — along section headings when the document has structure, or into ~1,500-character windows with a 200-character overlap when it doesn't. Every chunk remembers its document, page, and exact character positions.

Convert tables. This is DocDB's signature move. A table spotted in a PDF isn't kept as a picture or a blob of text — it becomes an actual SQL table with properly typed columns, so "$1,250.00" is stored as the number 1250.00 that SQL can sum. The original cell text is kept alongside in shadow columns, and every row records the page and position it came from.

Validate the tables. Table extraction from PDFs is error-prone, so DocDB checks its own work: if a table has a "Total" row, do the extracted numbers actually add up to it? Tables that pass are marked trusted. Tables that fail get a second chance through a vision-model re-extraction; if they still fail, they are honestly labelled untrusted in the manifest so the query phase knows to fall back to the raw text instead.

Index, describe, and freeze. Finally, a full-text search index (SQLite FTS5, with BM25 ranking) is built over the chunks; a manifest is written — a machine-generated inventory listing every document, every table, its columns, and its trust status; and database triggers are installed that make all source data immutable. From this moment, nothing — not even the AI at query time — can modify or delete the original text and tables.

5. Inside corpus.db

Everything lives in one ordinary SQLite file. You can open it with any SQLite tool. Here is its anatomy:

Two properties are worth underlining:

Provenance everywhere. A chunk knows its document, page, and character range. A table row carries the page and bounding box it was extracted from. This is what makes verified quotes (section 9) possible — any claim can be walked back to the exact original characters.

Frozen source, writable margins. After ingestion, triggers reject any insert, delete, or update against the source columns. The only things that can ever be written later are additions in the working layer: annotation columns the AI adds (section 8), the log recording how they were made, and cached embeddings. The originals are tamper-proof.

6. Phase 2 — Answering a question

Now the interesting part. You ask:

rnsr query corpus.db "What was FY2023 segment revenue?"

RNSR does not paste your documents into an AI prompt. Instead, it runs a loop — called the RLM loop (recursive language model) — in which a "root" AI model is given the manifest (the map of what's in the database) and a Python workspace connected to corpus.db. The model investigates by writing code; the system runs the code and shows the model what came back; the model writes more code; and so on until it is confident enough to commit to an answer.

The model's toolbox

Inside the sandbox, the model finds these ready-made tools:

Why a sandbox?

The model's code runs in a separate process with no network access. It can touch the database and nothing else. When its code needs an AI call (for annotation or search), the request is passed to the parent process, which makes the call on its behalf under strict concurrency and budget limits. If a code cell runs away, the sandbox is restarted. Every code cell, result, and model call is written to a trajectory file — a complete, replayable record of how the answer was reached.

7. The search ladder: cheap first, expensive only if needed

"Find where the documents talk about X" is the bread-and-butter operation, and DocDB refuses to pay for an expensive search when a cheap one will do. search() is a ladder of six rungs, tried from cheapest to priciest. It stops at the first rung that finds hits.

Three details matter:

Every rung resolves to real text. No matter which rung produced a hit, the result carries provenance — document, page, character range — pointing into the retained source text. A hit is never just a score; it's a place you can look.

Embeddings are lazy. Vector embeddings (the technology behind "semantic" search) are not built at ingestion. The first time a question actually needs rung 4, the chunks are embedded and the vectors are cached inside corpus.db itself — so the cost is paid once, and only if ever needed.

The sweep asks permission. If rungs 0–4 all come up empty, the ladder does not silently read every chunk with an AI. It returns a cost estimate, and the root model must explicitly decide the question is worth it. This is how DocDB can prove absence — "there is no such clause" — by actually enumerating everything, but only deliberately.

8. Semantic annotation: AI judgments become database columns

Some questions need a judgment applied to every row: "How many of these 200 contract clauses are termination clauses?" A naive approach asks the AI to read all 200 and count — slow, expensive, and error-prone.

semantic_annotate does it differently: it sends each row's text to a cheap sub-model in parallel batches, collects the yes/no (or label) for each, and writes the results back as a new real column on the table. Now counting is just SQL:

# One batched AI pass adds the column...
semantic_annotate("t_contract_003", "is_termination",
                  "Is this clause a termination clause? Answer yes or no.")

# ...and the count is exact, forever after:
db.execute("SELECT COUNT(*) FROM t_contract_003 WHERE is_termination = 'yes'")

Three guarantees come with it:

In complexity terms: reasoning that would take the model O(N²) reading becomes O(N) cheap calls plus one exact SQL query.

9. Verified quotes: the answer must show its receipts

In DocDB mode, the model cannot simply declare an answer. The FINAL call requires supporting quotes, and a verifier then checks each quote by exact string matching (after light whitespace normalisation) against the retained source text.

• If a quote is found, the answer ships with the quote and its character offsets — a citation a human can independently check.
• If a quote is not found — the model paraphrased, misremembered, or invented it — the answer is rejected and the model is sent back into the loop with the failure explained.
• A three-strike rule stops the model from spiralling: after repeated verification failures the run ends honestly rather than looping forever.

10. Budgets and safety rails

Every query runs under hard caps, so a confused run costs a bounded amount and then stops:

Additional rails: damping against pointless re-verification loops, a variable-recovery fallback (if the loop dies but the model had already computed the answer into a variable, it is salvaged), sandbox restart on runaway code cells, and timeouts on every root-model call. When many questions share one batched loop (the answer-csv path), the caps scale sub-linearly — each extra question adds only half a question's budget — so a confused batch can't burn the full per-question budget times the batch size.

11. Why it gets cheaper over time

DocDB is built for working sessions — many questions against the same documents — and several kinds of work are paid once and reused:

There is also a corpus cache: when you point answer-csv at a folder of documents, the resulting corpus.db is keyed by a hash of the files' identities, so re-running never re-ingests unchanged documents. Long runs checkpoint their partial answers, so an interrupted job resumes instead of re-paying.

12. When to use DocDB (and when not to)

RNSR's own benchmarks are candid about this — DocDB is not always the right tool.

One line: read a document → use the context window; interrogate a matter → use DocDB.

Where to go deeper