Metadata-Version: 2.5
Name: mrdb
Version: 0.1.2
Summary: A typed, incremental, cross-language binary database: declare tables and actions in Python; read, query, and subscribe to them from a generated TypeScript client over static files.
Requires-Python: >=3.12
Provides-Extra: auth
Requires-Dist: cryptography>=42; extra == 'auth'
Provides-Extra: dev
Requires-Dist: watchdog>=6.0.0; extra == 'dev'
Provides-Extra: s3
Requires-Dist: boto3>=1.34; extra == 's3'
Description-Content-Type: text/markdown

# mrdb INTERNALS

The one file an agent should read before working on mrdb. It folds
`NAVIGATION.md`, `docs/layers.md`, and the essential contract rules that
used to be spread across `mrdb/docs/*`. User-facing docs live in
`../pages/mrdb-docs/mrdb-guide/`; engineering history in
`../pages/mrdb-docs/mrdb-story/`.

## The three rings (import law)

- Ring 1 CORE (`core/`) imports only itself; its TS mirror lives under
  `assets/core/` (browser SDK) and `assets/server/` (node kernel).
- Ring 2 SDK (`mrdb/__init__`, `mrdb.engine`) is the ONLY bridge between
  core and everything above.
- Ring 3 BATTERIES (`extensions/*`) import only Ring 2, never core
  internals, never each other's privates.
- The app tier (`mrdb/cli.py`, `mrdb/cli_ops.py`) sits above the batteries:
  it may use Ring 2, battery facades, and core internals (operator
  commands are app-tier privilege). The user-owned project content
  (repo-root `app.py`, `config.py`, `ui/`) is outside the governed tree.
- `scripts/check_imports.py` enforces all edges against a baseline.

## Layer law (normative)

The core chapters mirror the tiers; import direction is strictly downward;
same-layer edges allowed. One module, one layer, no exceptions - a module
missing from this table FAILS the checker (rules M1/M2).

```
L0  model.schema, model.codec, model.tsident, model.uuid7,
    storage.base.backend (protocol-only seam), storage.base.fs,
    storage.base.lock, storage.base.names,
    wiring.config, wiring.loader, runtime.shardmap, runtime.httpsrv
L1  wiring.registry,
    storage.drivers.posix, storage.drivers.s3, storage.drivers.select,
    storage.drivers.s3_publish, storage.drivers.s3_botocore,
    storage.range_engine, storage.records.format
L2  storage.records.reader, storage.records.reader_core,
    storage.records.writer, storage.records.writer_core
L3  storage.records.overlay
L4  database.table
L5  database.actions
L6  database.memory, database.db, database.engine, database.reader_pool,
    database.index_maint, query.plan, query.where, query.exec,
    runtime.owner, runtime.client
L7  runtime.live, runtime.coordinator, runtime.replica
L8  mrdb/__init__, mrdb.engine, mrdb/cli, mrdb/cli_ops        (SDK + app tier)
L9  extensions/*, mmr/*                                       (batteries + host)
```

Checker rules: R1 layer creep; R2/R3 core importing root/extensions; R4
extensions import only mrdb/mrdb.engine; R5 no underscore-private reach-
through (use public seams); R6 no function-level imports (lazy only to
break a cycle / defer heavy dep, with a comment naming it); R7 no
cross-battery imports except declared EXTENSION.requires.

Maintenance rule: adding a core module requires a row here in the same
changeset; moving a module updates the table and nothing else.

### Domain bands (how to read the core tiers)

Each core domain follows the same grammar - contracts, engine, providers,
facade - so the L0-L7 chain reads as four per-domain stacks rather than one
flat ladder:

| Domain | contracts | engine | providers (dispatch seam) | facade |
|---|---|---|---|---|
| `model/` | schema, tsident, uuid7 | codec (plan cache) | - | `model/__init__` |
| `storage/` | base/backend (Protocols), records/format (wire), base/names | range_engine, records/{reader_core,writer_core} | drivers/* dispatched by `drivers/select.py` | records/{reader,writer,overlay} |
| `database/` | table (declarative), actions | db, memory | - | engine, reader_pool |
| `query/` | plan (the Plan algebra + byte-bound compilation) | exec (ops over a Plan) | where (the pluggable syntax front end) | `query/__init__` |
| `runtime/` | shardmap, httpsrv (pure transport) | live, coordinator, replica | owner/client roles | live serve API |
| `wiring/` | registry (hook types + manifest) | topo-sort activation | the batteries | `activate()` |

`wiring/` is the plugin contract, not composition policy: core consumes the
hook buckets downward (`runtime/live.py` reads guards/routes at request
time), so `registry.py` cannot move above core.

## Task -> files to read (and only these)

| Task | Read |
|---|---|
| Storage format change | `core/storage/records/format.py` + `../pages/mrdb-docs/mrdb-guide/11-reference/wire-contract.md` + goldens (`tests/backend/runtime/golden/*.json`) |
| Segments / epoch fencing | `writer_core.py` + `reader_core.py` (`normalize_segments`, adoption, conditional publish) |
| Read path / queries | `records/reader.py` (path facade) over `reader_core.py`; driver selection `drivers/select.py` |
| Write path / commit | `records/writer.py` + `database/db.py` |
| Batching / read-your-writes | `records/overlay.py` + `database/db.py` (DatabaseWriter) |
| Live kernel / actions | `runtime/live.py` + `runtime/httpsrv.py` (asyncio HTTP) + `database/actions.py` |
| Node kernel control plane | `assets/server/kernel/kernel.ts` + `serve.ts` (parity pinned by `tests/backend/conformance/test_kernel_conformance.py`) |
| Replication / failover | `runtime/replica.py` + `runtime/coordinator.py` + guide 08-distribution.md |
| Sharding / routing | `runtime/shardmap.py` + `assets/server/shards.ts` + parity vectors |
| Bucket limits | guide 09-s3-limitations.md |
| New extension | guide 06-extensions.md + `extensions/auth/` as reference + `mrdb extension new` |
| TS client / browser | `assets/index.ts` + guide 07-deployment.md |
| Node writer / cross-runtime | `assets/core/storage/node.ts` + `lock.ts` + wire-contract.md |
| Ownership / leases | `base/lock.py` + `runtime/owner.py` + wire-contract.md + lease matrix test |
| Driver conformance / types | `base/backend.py` (runtime-checkable Protocols) + isinstance test + check_types baseline |

## Cross-runtime parity pairs

Same concept, findable under the same name on both sides:

| Python (core/) | TypeScript (assets/) |
|---|---|
| `runtime/live.py` | `server/kernel/kernel.ts` + `serve.ts` |
| `runtime/shardmap.py` | `server/shards.ts` |
| `storage/base/lock.py` | `core/storage/lock.ts` |
| `model/codec.py` | `core/format/codec.ts` |
| `model/schema.py` | `core/format/spec.ts` |
| `storage/base/names.py` | `core/format/names.ts` |
| `storage/records/format.py` | `core/format/writer.ts` (+ spec.ts) |
| `query/` (plan, where, exec) | `core/query/` (plan.ts, where.ts, exec.ts) |

A change to one side must keep the other side's goldens green.

## Invariants (must hold through any change)

1. **Byte-frozen format**: frames, base header/index, CRC placement,
   generation retention. Goldens are normative. The one approved amendment
   (P9/D29) added META-ONLY fields (`meta.epoch`, `meta.segs`).
2. **Frozen npm surface**: every export keeps name and behavior; new
   subpaths enter as additive `{browser: null}` entries.
3. **Frozen Python SDK**: `mrdb` + `mrdb.engine` pinned by
   `tests/backend/unit/test_engine_sdk.py`.
4. **Lock order**: per-table lock(s) first, db-global last. Never inverted.
5. **Cross-runtime lockstep**: one normative contract (goldens +
   wire-contract.md), two implementations.
6. **Single drainer-writer** per kernel; idempotent LWW re-application.
7. **Readers are snapshots**; compaction-race retries stay internal to the
   reader, both runtimes.
8. **One owner per db dir**; fencing = epoch segments + version-
   conditional publish; takeover only through the normal claim path with
   fail-closed rules (foreign or possibly-alive owners never fenced).
9. **Byte-range serving**: no caller may full-read a large object; reads
   go through `engine_for(storage).wrap(handle, name)`; `read_all` capped
   at MAX_READ_ALL_BYTES (1 MiB), `allow_large=True` = reviewable escape
   hatch.

## The executable contract web (do not break silently)

All under `tests/backend/runtime/golden/` unless noted:

- `contract.json` - machine-readable constants (versions, bounds, fold
  policy, lease timing, reserved names); asserted by BOTH suites.
- `cases.json` - codec row vectors both runtimes decode identically.
- `table.json`, `sharded.json` - byte-exact journal+base fixtures.
- `rejections.json` - malformed-input corpus (16 cases) both runtimes
  reject identically; tolerated pins prevent silent tightening; journal
  cases pin py-fails-closed / ts-degrades.
- `reader-vectors.json`, `auth-vectors.json` - semantic vectors on shared
  fixtures.
- `range-cache.json` - scripted range-cache conformance executed by both
  suites; divergences encoded per runtime IN THE DATA.

Rule: semantic changes land in the contract first; both runners stay
green. If you add a limitation or divergence, document it (guide 09 for S3)
or encode it in a contract - never leave it implicit.

### Contract conventions (for range-cache-style suites)

- Runners: Python `tests/backend/parity/test_parity_conformance.py`,
  TypeScript `tests/frontend/tests/parity-conformance.test.ts`. Python is
  the reference runtime - write cases against its behavior first, then port
  the runner.
- Resource bytes are deterministic: `byte i = i & 0xff`.
- Outcome classes map each runtime's exceptions to shared names:
  `value` (programmer error), `eof` (beyond the committed resource),
  `error` (anything else).
- Counter names use the TS spelling (`sourceFetches`); runners normalize
  (`RangeEngine.snapshot().source_fetches`).
- Adding an area: copy `range-cache.json`, keep cases small enough to
  review at a glance - the matrix grows by accretion, not by generality.

## Subtle seams (edit carefully, run contracts after)

- `core/model/codec.py` + plan cache (weakref-keyed 4-tuple entries,
  sweeps past 512): encode/decode fast paths must keep byte-identical
  output.
- `core/storage/records/writer_core.py`: fused frame path has an optimistic
  pass with restart-on-null fallback preserving validation order.
- `core/storage/records/reader_core.py`: ParsedBlock blob+arrays form +
  binary search; BLOCK_CACHE_MAX=1024 budget; tests scale off it
  dynamically.
- `range_engine.py`: serve-through assembly means reads NEVER depend on
  cache residency; seeds (mmaps) are zero-copy and capacity-exempt.
- POSIX append/publish syscall order is trace-pinned FROZEN
  (test_posix_trace.py) - do not reorder fsync/rename sequences.

## Standing rules (engineering law, from the architecture plan)

1. No new role on `.owner` unless an existing role is retired in the same
   changeset.
2. No new cross-runtime mechanism without naming its contract artifact or
   conformance suite AT DESIGN TIME ("keep in sync by hand" is rejected at
   review).
3. Every mechanism declares its consumers in its module docstring.
4. Speculation is staged, not landed early.

The full decision register (D1-D31), anti-patterns list, twin strategy
matrix, and amendment history live in
`../pages/mrdb-docs/mrdb-story/97-design-register.md`.

## Bench gates and methodology

- **P0 anchor** (tree `5ab49381`, AMD Ryzen 9 PRO 8945HS): flock acquire
  6.94 us; single-row commits memory 130 us / file 179 us; batched 2.23
  us/row; bench.py write ~470k/s vs sqlite ~970k/s, reads ~460k/s vs
  ~103k/s.
- **Gates**: no metric worse than 10% vs baseline; contended handoff may
  differ but must not exceed 2x; scale-invariance (bench_scale.py flat vs
  prefill) is a standing gate.
- **Sensitive signal for lock/write work**: memory-backend single-row
  commits (fsync is a RAM no-op there).
- **Trace before timing**: an added hot-path syscall is a seam bug
  regardless of timing (P5 rule; the syscall-order trace test pins it).
- **Same-session A/B** when comparing trees: extract the old tree via
  `git archive`, bench via PYTHONPATH alternating old/new runs so machine
  drift cancels; bench.py's sqlite rows are the drift control.
- Benches are manual-not-CI by policy; S3 numbers are directional only
  (note bucket+region).

## Known deferred seams and declined work

- `Database.open(path_or_url, *, storage=, coordinator=)` injection was
  DEFERRED with a named design trap: an injected coordinator's strict name
  walk refuses symlinked components, while commit-path locking locks
  THROUGH the memory-table tmpfs target - a memory-table-aware coordinator
  lock-resolution rule must be specified first.
- Local readers retain full baseCrc verification; a future remote driver
  must pin object version + prove a trusted whole-object checksum before
  range-only reads may skip a download.
- `lock.py` monolith split DECLINED (hygiene not risk) at review round 5;
  revisit if lock.py grows new responsibilities beyond lease + flock +
  claim guard.

## Gates (run before declaring done)

```sh
cd mrdb && timeout 600 python -m pytest tests/backend -q -p no:cacheprovider
cd mrdb && timeout 600 bun test tests/frontend/tests
python scripts/check_imports.py            # from repo root
python scripts/fix_style.py                # dry-run; --write to apply
timeout 300 python scripts/check_types.py  # diff vs recorded baseline
```

Bench gates: `tests/backend/performance/bench_scale.py` must stay flat vs
prefill (scale-invariance); `bench.py` sqlite rows are the drift control.
Record notable runs (machine/load/tree state) when they change the story.

Known flake: `test_posix_trace.py::test_append_recovery_keeps_the_frozen_
legacy_syscall_order` can fail under load; rerun alone before diagnosing.
