Metadata-Version: 2.4
Name: omnidb-server
Version: 0.1.0
Summary: MCP-native agent memory & retrieval server over SQLite (WAL + FTS5) with an hnswlib vector sidecar.
License: Apache-2.0
Requires-Python: >=3.11
Description-Content-Type: text/markdown
Requires-Dist: mcp<2,>=1.9
Requires-Dist: numpy>=1.26
Requires-Dist: httpx>=0.27
Provides-Extra: semantic
Requires-Dist: fastembed>=0.3; extra == "semantic"
Provides-Extra: hnsw
Requires-Dist: hnswlib>=0.8; extra == "hnsw"
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: pytest-cov>=5; extra == "dev"
Requires-Dist: ruff>=0.6; extra == "dev"
Requires-Dist: hypothesis>=6.100; extra == "dev"

# OmniDB — MCP-native agent memory & retrieval server

A local-first, single-binary-feel **agent memory and retrieval server over SQLite**, exposed to AI
agents through the Model Context Protocol. This is the revised v1 that came out of the August 2026
research + adversarial review (see `../OmniDB Vault/`): an MCP memory/retrieval server, not a database.

- **Substrate**: SQLite in WAL mode (`synchronous=NORMAL`, `busy_timeout=5000`) — one directory on disk,
  copyable as a unit (`omnidb.db` + WAL sidecars + vector index file). The single-file mandate was dropped.
- **Retrieval**: BM25 (FTS5, porter tokenization) fused with dense vector search via Reciprocal Rank Fusion,
  then a gentle recency/frequency rerank. Hybrid retrieval runs in-process; SQLite table-valued functions
  (`VECTOR_SIM` / `MEMORY_RECALL`) need a native extension and are the documented upgrade path for the
  future Go/Rust port.
- **Vectors**: hnswlib (M=16, efC=200) inner-product on L2-normalized vectors == cosine, persisted as a
  sidecar file; exact brute-force fallback always available. Deletions are mark-deleted.
- **Memory model**: plain relational tables (`episodic | semantic | procedural`) — no bespoke LSM, no fixed
  decay religion (weights are config). Consolidation distills episodic → semantic via a configurable LLM,
  with an offline stub when none is set.

## Quickstart

```bash
pip install -e ".[hnsw,semantic,dev]"   # hnsw/semantic extras optional; falls back gracefully
pytest
omnidb-server serve                     # MCP stdio server on $OMNIDB_HOME (default ~/.omnidb)
python -m omnidb_server serve           # equivalent
python -m omnidb_server stats
```

### Claude Desktop / any MCP client

See [`examples/claude-desktop-config.example.json`](examples/claude-desktop-config.example.json)
— includes the env vars for real (fastembed) semantics and a working `HF_HOME`
override. Minimal version:

```json
{
  "mcpServers": {
    "omnidb": {
      "command": "python",
      "args": ["-m", "omnidb_server", "serve"]
    }
  }
}
```

## Tools

| Tool | Purpose |
|---|---|
| `remember(content, kind, metadata, links)` | persist a memory (episodic/semantic/procedural), optional graph links |
| `recall(query, k, kind, k_fetch)` | hybrid BM25+dense recall with recency/frequency rerank; pool floors at `k` |
| `update_memory(memory_id, content, metadata)` | edit content (re-embeds + reindexes via FTS trigger) and/or merge metadata |
| `search(sql)` | read-only SELECT/WITH escape hatch against SQLite |
| `related(memory_id, max_hops, label)` | undirected graph walk (recursive CTE, cycle-safe, hops clamped 1-8) |
| `forget(memory_id)` / `forget_many(ids)` | soft delete one / many |
| `consolidate(hours)` | episodic → semantic distillation |
| `stats()` | counts, edges, embedder/index info, dead-ratio, all file sizes |

Error messages carry `OMN-*` codes from the original spec's taxonomy (`[OMN-020]` read-only violation, `[OMN-021]` not found, `[OMN-006]` dimension mismatch, `[OMN-011]` timeout, ...).

## Configuration

Precedence: **CLI flags > environment > `omnidb.toml` > defaults**. The TOML file is looked up in
the CWD and the data directory (field names mirror the config dataclass):

```toml
half_life_hours = 168
rrf_k = 60
k_fetch = 50
persist_every_ops = 64      # vector-sidecar fsync debouncing
dead_compact_ratio = 0.2    # hnsw tombstone ratio triggering rebuild-on-open
```

## Operations

```bash
omnidb-server compact --home D:/path/to/data   # force full vector-index rebuild
python -m omnidb_server stats                  # includes pending_compaction, reconciled_at, WAL size
omnidb-server check                            # FTS integrity, index drift, orphan edges, WAL size
omnidb-server purge [--vacuum]                 # hard-delete soft-deleted rows; --vacuum reclaims space
omnidb-server export D:/backup/memories.jsonl  # dump all memories + edges to JSONL
omnidb-server import D:/backup/memories.jsonl  # restore from a JSONL dump
```

## Configuration (environment)

| Variable | Default | Meaning |
|---|---|---|
| `OMNIDB_HOME` | `~/.omnidb` | data directory |
| `OMNIDB_EMBEDDER` | `auto` | `auto`/`hashing` are offline-safe (never download); `fastembed` opts into real semantics (`[semantic]` extra, ~100MB model on first run). Embedder identity is pinned per database. |
| `OMNIDB_INDEX` | `auto` | `auto` \| `hnsw` \| `brute` |
| `OMNIDB_HALF_LIFE_HOURS` | `168` | recency half-life for rerank |
| `OMNIDB_RECENCY_WEIGHT` / `OMNIDB_FREQ_WEIGHT` | `0.15` / `0.05` | rerank weights (RRF base is weight-free) |
| `OMNIDB_K_FETCH` | `50` | candidates fetched per leg before fusion |
| `OMNIDB_RRF_K` | `60` | RRF smoothing constant |
| `OMNIDB_MAX_SEARCH_ROWS` | `500` | row cap for the `search` tool |
| `OMNIDB_LLM_BASE_URL` / `_MODEL` / `_API_KEY` | unset | OpenAI-compatible endpoint used by `consolidate`; stub when unset. With one configured, a failed call consumes nothing (episodes stay retryable) |

## Benchmarking

```bash
python bench/baseline.py   # writes bench/results.md
```

Measures hnswlib (M=16, efC=200) recall@10 vs exact brute force across an ef sweep at
384/768 dims on clustered synthetic data (1k/10k/100k), plus the SQ8 experiment.
Findings so far: at 768-dim/10k, SQ8+rerank (0.947) beats raw HNSW at ef=128 (0.837);
at 100k the clustered-synthetic distribution is strongly pessimistic for graph ANN
(tight clusters create hub structure), where quantized scoring + f32 rerank stays
robust (0.90) but at full-scan cost (~120ms). Treat as a floor, not a forecast —
re-run against your real corpus before drawing conclusions.

## Encryption at rest

OmniDB is local-first and writes only to `OMNIDB_HOME`; it phones home to no one. For
data-at-rest protection we recommend OS-level disk encryption (BitLocker / FileVault /
LUKS) — that covers the stolen-disk threat model without adding a dependency or a key
you must manage.

A SQLCipher swap is documented but deliberately not bundled: encrypting the SQLite file
would require `pysqlcipher3`, whose Windows wheels for Python 3.14 are effectively
unobtainable, so shipping it as an extra would break installs rather than protect them.
If your threat model needs per-file encryption beyond full-disk, compile SQLCipher and
point this package's connection at it — the store layer is plain `sqlite3` DB-API and
needs no code changes.

## Limitations (v1, deliberate)

- Single-writer process per home directory (SQLite WAL semantics). The vector sidecar is
  last-writer-wins: run one server per home; startup reconciliation after a crash between the DB and
  the index file is deferred.
- The Python `sqlite3` module cannot register virtual tables, so SQL++-style TVFs arrive with the
  native port; until then hybrid queries go through the tools.
- Hashing embedder matches surface forms, not paraphrases — install `fastembed` for real semantics.
  Embedder identity is pinned per database; switching requires a fresh home.
- No QPS targets: this serves one agent session locally. Gates are interactive latency, cold start,
  and RAM at scale (re-baselined empirically at true embedding dimensions, not SIFT-128 folklore).
