Metadata-Version: 2.4
Name: akshamdb
Version: 0.1.0
Summary: Fast, persistent, embeddable vector database
License: Apache-2.0
Keywords: vector database,semantic search,embeddings,RAG,HNSW,ANN
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: C++
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: numpy>=1.24
Provides-Extra: server
Requires-Dist: flask>=2.3; extra == "server"
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: pytest-benchmark; extra == "dev"

# AkshamDB — Fast, Persistent, Embeddable Vector Database

**v0.1.0** — A self-hosted vector database built from scratch in Python and C++. Designed for semantic search and RAG applications where you want to own the entire stack.

---

## What Is This?

AkshamDB combines three production systems under one engine:

- **LSM Store** — Write-Ahead Log (with fsync) → Memtable → SSTables with Bloom filters and background compaction. Crash-safe ACID transactions.
- **HNSW Index** — C++ approximate nearest-neighbor graph implementing the full Malkov & Yashunin 2018 algorithm. GIL-free search, persistent thread pool, AVX2 batch scoring.
- **Query Engine** — 5-step pipeline: metadata filter pushdown (O(1)) → ANN retrieval → candidate merge → LSM fetch → hybrid scoring (70% cosine + 30% Okapi BM25).

Optional scale layers: **Product Quantization** (4–32× memory compression) and **IVF coarse index** (Voronoi-cell partitioning for 10M+ vector corpora).

Unlike hosted solutions (Pinecone, Weaviate), you own the whole stack — storage format, index, scoring, and server.

**Dependencies:** Core requires only `numpy`. The optional REST API server (`pip install akshamdb[server]`) additionally needs Flask. No sklearn/scipy in the core path — the BM25 keyword index is implemented from scratch.

---

## Architecture

```
┌──────────────────────────────────────────┐
│          REST API  (Flask)               │
│   akshamdb-server / python -m akshamdb   │
└─────────────────┬────────────────────────┘
                  │
┌─────────────────▼────────────────────────┐
│             AkshamDB  (public API)       │
│ akshamdb.open() → add / search / delete  │
└─────────────────┬────────────────────────┘
                  │
┌─────────────────▼────────────────────────┐
│            VectorDBEngine                │
│  ┌───────────┐  ┌────────┐  ┌─────────┐ │
│  │  LSMStore │  │  HNSW  │  │ Query   │ │
│  │  (Python) │  │ (C++)  │  │ Engine  │ │
│  └───────────┘  └────────┘  └─────────┘ │
│  ┌───────────┐  ┌────────┐              │
│  │    PQ     │  │  IVF   │  (optional)  │
│  │(compress) │  │(coarse)│              │
│  └───────────┘  └────────┘              │
└──────────────────────────────────────────┘
      │               │            │
 [WAL + SSTables]  [index.bin]  [manifest.json]
 [Bloom filters]   [pq_*.pkl]   [ivf_index.pkl]
```

---

## What Has Been Built

### Phase 1 — Foundation

| Component | Details |
|---|---|
| **LSM Store** | WAL append (with fsync for crash durability) → in-memory OrderedDict (memtable) → SSTable files on flush. Atomic WAL truncation after each flush. Background compaction thread merges SSTables with atomic rename. Thread-safe SSTable ID allocation. Bloom filter per SSTable skips disk on definite misses. Full ACID: `begin / commit / rollback`. |
| **HNSW (C++)** | Multi-layer navigable small-world graph implementing Malkov & Yashunin 2018. Flat contiguous vector storage. Format v3 serialization (4-byte `HNSW` magic header + version + full edge lists). Supports reading legacy v1/v2 files. Exposed to Python via pybind11 with GIL released during `search()`. |
| **Query Engine** | Five-step pipeline: (1) filter pushdown on in-memory metadata index, (2) ANN retrieval via HNSW, (3) intersect filter ∩ ANN candidates, (4) fetch docs from LSM, (5) hybrid cosine + Okapi BM25 scoring (alpha=0.7 vector, 0.3 BM25). |
| **Embedder** | `all-mpnet-base-v2` via Sentence-Transformers. 768-dim float32 vectors. |
| **Reranker** | `cross-encoder/ms-marco-MiniLM-L-6-v2` CrossEncoder for post-search reranking. |
| **Chunker** | `ingestion/chunker.py` — sentence, paragraph, and word-based strategies with configurable chunk size (default 512 chars) and overlap (default 128 chars). |
| **REST API** | Optional Flask server (`pip install akshamdb[server]`). Start with `akshamdb-server` or `python -m akshamdb.server`. Insert, search, batch search, get, delete, transactions, health, stats endpoints. |

### Phase 2 — Parallelism & Throughput

| Feature | Details |
|---|---|
| **GIL-free Search** | C++ `HNSW::search()` releases the GIL — concurrent Python threads run in native code without waiting on each other. |
| **Persistent Thread Pool** | C++ thread pool started at engine init. Reused across calls, no per-query spawn overhead. Controlled by `Config.PARALLEL_SEARCH_WORKERS`. Explicit destructor prevents thread leak on shutdown. |
| **Batch Search (search_batch_reuse)** | Phase 1: all query traversals run in the C++ thread pool in parallel. Phase 2: build a union of candidates and score them with an AVX2 blocked kernel — each candidate vector loaded exactly once for all queries. Python-layer scoring is vectorised via numpy batch matmul. Typical 3–10× throughput over a Python loop of single searches, depending on corpus size and batch size. |
| **Python-level Parallel Fallback** | `search_batch_parallel` uses `ThreadPoolExecutor` as a fallback when `search_batch_reuse` is unavailable. GIL-free C++ search lets threads genuinely overlap. |

### Phase 3 — Scale & Reliability

| Feature | Details |
|---|---|
| **Three-tier Startup** | Tier 1 (Fast): saved HNSW node count matches LSM doc count → load index + remap IDs only. Tier 2 (Incremental): saved HNSW has fewer nodes → load index + add only delta docs from WAL. Tier 3 (Full rebuild): no saved index. Startup cost is O(delta), not O(N). |
| **Product Quantization (PQ)** | Subspace k-means codebooks (no sklearn dependency). `train → encode → adc_distances` pipeline. Asymmetric Distance Computation replaces exact cosine when active. Codebooks and codes persist to disk (`pq_codebooks.pkl`, `pq_codes.npy`). Enable when memory > ~50 GB. |
| **IVF Coarse Index** | Partitions corpus into K Voronoi cells via k-means. Search probes only `nprobe` nearest cells. Supports IVF+PQ mode (ADC scoring inside lists). Enable when HNSW latency is too high at 10M+ vectors. |
| **Profile-Driven Optimization** | `tools/profile_hotspots.py` — cProfile analysis of the full insert + search pipeline. Reports time breakdown by subsystem (HNSW graph build, WAL/JSON, LSM/SSTable, threading, Python overhead), hottest functions, WAL microbenchmarks. Includes a `_build_recommendations()` engine that rates bottlenecks **HIGH / MEDIUM / INFO** and prints concrete action items. |
| **Runtime Index Tuning** | `index.set_ef_search(n)` tunes HNSW search quality at runtime without rebuilding the graph. Takes effect on the next search call. |
| **pip-installable Package** | `setup.py` wires CMake into pip's build process. `pip install -e .` configures and compiles the C++ extension automatically. `akshamdb-server` CLI entry point registered via `pyproject.toml`. |
| **Bloom Filters** | Per-SSTable Bloom filter saved alongside each `.bin` file. Skips disk I/O for definite key misses during reads. |
| **Ghost Node Handling** | On rollback, `id_map` entries are purged so rolled-back docs are invisible to all future searches. HNSW graph nodes can't be removed — they stay as ghost nodes. On `close()`, the index file is not overwritten if ghosts exist, so the last clean save is preserved for fast-path startup. |

### Phase 5 — Correctness & Performance Fixes (v0.3.0)

| Fix | Details |
|---|---|
| **Real Okapi BM25** | The keyword scorer was previously sklearn TF-IDF cosine similarity. It is now a pure-Python Okapi BM25 implementation (k1=1.5, b=0.75) backed by an inverted index. Only documents containing at least one query term are scored, making retrieval sub-linear in corpus size. No sklearn dependency in the core path. |
| **search_batch O(N²) fix** | Previously, each query in a batch called `cosine_similarity(query_vec, full_matrix)` individually — O(M×N×V) with poor BLAS utilisation. Real BM25 with an inverted index eliminates the dense matrix multiply entirely. Python-layer vector scoring is now a single numpy batch matmul instead of per-doc cosine loops. |
| **Vectorised candidate scoring** | `_hybrid_score_and_rank` now stacks all candidate vectors into a matrix and computes all cosine similarities in one `doc_mat @ query` call, replacing N individual `np.dot` calls. |
| **print() → logging** | All `print()` calls in `db_engine.py`, `lsm_store.py`, and `ivf_index.py` replaced with `logging.getLogger(__name__)`. Control verbosity via Python's standard logging config. |
| **sklearn removed from core deps** | `scikit-learn` is no longer a required dependency. Core akshamdb only needs `numpy`. |

### Phase 4 — Correctness & Algorithm Fixes (v0.2.0)

| Fix | Details |
|---|---|
| **Real HNSW Beam Search** | `search_layer` now uses two priority queues (Malkov & Yashunin 2018, Algorithm 2): `C` (min-heap of candidates to expand) and `W` (max-heap of current best `ef` neighbors). Early termination fires when the closest unexplored candidate is farther than the worst node in `W`, giving O(ef · log(ef) · M) complexity vs the previous O(N) BFS. |
| **SELECT-NEIGHBORS-HEURISTIC** | Neighbor selection during `add_point` now uses Algorithm 4 from the paper. The heuristic rejects a candidate `c` if an already-selected neighbor `r` satisfies `dist(c, r) < dist(c, query)` — meaning `r` already "covers" the direction toward `c`. This prevents graph connectivity from collapsing into one region of space and keeps recall high at all ef values. Falls back to filling from remaining candidates if the heuristic is too aggressive. |
| **Neighbor Degree Cap** | `mmax0 = 2M` at layer 0, `mmax = M` at upper layers, matching the paper exactly. Over-degree neighbors are pruned using the same heuristic, not a simple truncation. |
| **Thread-safe RNG** | `get_random_level()` uses `thread_local std::mt19937` seeded from `std::random_device{}()` — each thread gets its own generator, eliminating lock contention on the level-assignment path. |
| **Format v3 Serialization** | Index files now begin with a 4-byte magic `HNSW` header followed by an integer version field. This makes format detection unambiguous — the old v2 sentinel (`FORMAT_VERSION=2` as the first int) collided with `dim=2` in v1 files. v1 and v2 files are still readable on load. |
| **WAL Fsync Durability** | `_write_wal()` calls `f.flush()` + `os.fsync(f.fileno())` after every write. WAL truncation after flush uses an atomic rename (`wal.log.trunc` → `wal.log`) to prevent partial truncation on crash. |
| **LSM Thread-safe SST IDs** | `_alloc_sst_id()` now acquires `_sst_lock` internally so concurrent compaction threads can't race to allocate the same SSTable ID. |
| **HybridSearch In-place Upsert** | `add_documents()` checks `_doc_id_to_idx` before appending. If the same `doc_id` is added twice, the text is updated in-place at the existing slot instead of creating a ghost entry. The O(1) swap-remove in `remove_document()` and O(1) update in `update_document()` are correct in all cases. |
| **Delete Syncs BM25** | `delete()` now calls `_bm25.remove_document()` and syncs `query_engine.hybrid_search` so deleted documents don't appear in keyword results after removal. |
| **Parameter Clamping** | HNSW constructor clamps `M`, `ef_construction`, and `ef_search` to ≥ 1 so an invalid configuration can never cause an empty-heap crash in `search_layer`. |
| **Dead Code Removed** | Duplicate `Metrics` class removed from `utils/schema.py` (the real one lives in `utils/metrics.py`). Dead `distance(vector, vector)` private method removed from C++ HNSW (was never called). |

---

## Installation

**Prerequisites:** Python 3.8+, CMake 3.15+, GCC/Clang with C++14 support

### Option A — install from wheel (fastest)

```bash
python3 -m venv venv
source venv/bin/activate
pip install akshamdb-0.1.0-cp310-cp310-linux_x86_64.whl
```

### Option B — pip install from source (recommended for development)

`setup.py` wires the CMake build into pip, so a single command configures, compiles, and installs the C++ extension:

```bash
python3 -m venv venv
source venv/bin/activate

# Core only (no embedding model, no server)
pip install -e .

# With REST API server
pip install -e ".[server]"

# Development (pytest, benchmarks)
pip install -e ".[dev]"
```

### Option C — manual build

```bash
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

cd cpp && mkdir build && cd build
cmake .. -Dpybind11_DIR=$(python -m pybind11 --cmakedir) -DCMAKE_BUILD_TYPE=Release
make -j$(nproc)
cp hnsw_cpp*.so ../../akshamdb/
cd ../../
```

### Rebuild wheel from source

```bash
rm -f akshamdb/hnsw_cpp*.so
python3 -m build --wheel
# wheel lands at dist/akshamdb-0.1.0-cp310-cp310-linux_x86_64.whl
```

Use `python -m build` (PEP 517), not `python setup.py bdist_wheel` directly — the
latter bypasses `pyproject.toml`'s `[project]` metadata and silently produces a
broken `akshamdb-0.0.0` wheel instead.

### Verify

```bash
python3 -c "import akshamdb; print(akshamdb.__version__)"
```

---

## Quick Start (Python API)

```python
import akshamdb

# Open (or create) a database
db = akshamdb.open("./my_db", dim=768)

# Insert a single document
db.add("doc_1", vector, text="Hello world", metadata={"source": "wiki"})

# Insert many at once (faster than a loop)
db.add_many(ids, vectors, texts=texts, metadata=meta_list)

# Search — returns [{"id", "score", "text", "metadata"}, ...]
results = db.search(query_vec, top_k=5)

# Filtered search — metadata keys are ANDed
results = db.search(query_vec, top_k=5, filter={"source": "wiki"})

# Hybrid search — blends vector cosine (70%) with BM25 (30%)
results = db.search(query_vec, top_k=5, text="what is deep learning")

# Batch search — parallel ANN, much faster than N individual searches
batch = db.search_batch([vec1, vec2, vec3], top_k=5)

# Fetch and delete
doc = db.get("doc_1")
db.delete("doc_1")

# Check membership
if "doc_1" in db:
    print("exists")

# Stats
print(db.info())  # documents, index_nodes, data_path, performance

db.close()
```

### Context Manager

```python
with akshamdb.open("./my_db", dim=768) as db:
    db.add("doc_1", vector)
    results = db.search(query_vec, top_k=5)
# db.close() called automatically
```

### Transactions

```python
with db.transaction():
    db.add_many(ids, vectors, texts=texts)
# commits on clean exit, rolls back on any exception
```

> **Ingest throughput note:** Outside a transaction, `add_many()` saves the HNSW index to disk after every call (crash-safe default). This limits throughput to roughly 100–200 docs/s for 768-dim vectors on a typical SSD. Wrapping bulk ingestion in `with db.transaction()` batches all writes into one WAL flush and one index save, achieving 800–1200+ docs/s. For multi-million-document corpora, always use transactions.

### Logging

AkshamDB uses standard Python logging — no output is printed unless you configure a handler:

```python
import logging
logging.basicConfig(level=logging.INFO)   # show startup and rebuild messages
# or silence everything:
logging.getLogger("akshamdb").setLevel(logging.WARNING)
```

### With Embedding

```python
from embedding.embedder import Embedder
import akshamdb

embedder = Embedder()   # loads all-mpnet-base-v2

with akshamdb.open("./my_db", dim=768) as db:
    texts = ["Machine learning is a subset of AI.", "Neural networks learn from data."]
    vecs  = embedder.encode(texts)
    db.add_many(["doc_0", "doc_1"], list(vecs), texts=texts, metadata=[{"src": "wiki"}] * 2)

    q_vec = embedder.encode(["What is deep learning?"])[0]
    for r in db.search(q_vec, top_k=3, text="deep learning"):
        print(f"{r['score']:.3f}  {r['text']}")
```

### With Chunking + Ingestion

```python
from ingestion.loader import prepare_documents
from embedding.embedder import Embedder
import akshamdb

raw_docs = ["Long document text here ...", "Another document ..."]
chunks, ids, metadata = prepare_documents(raw_docs)

embedder = Embedder()
vecs = embedder.encode(chunks)

with akshamdb.open("./my_db", dim=768) as db:
    db.add_many(ids, list(vecs), texts=chunks, metadata=metadata)
```

---

## REST API

### Start the Server

```bash
# Default: port 8000, data at ./data
akshamdb-server

# Custom settings via env vars
AKSHAMDB_PATH=/my/db AKSHAMDB_DIM=768 AKSHAMDB_PORT=9000 AKSHAMDB_TOP_K=10 akshamdb-server

# Or: python -m akshamdb.server

# Production (gunicorn, 4 workers)
gunicorn -w 4 -b 0.0.0.0:8000 "akshamdb.server:create_app()"
```

### Endpoints

| Method | Endpoint | Description |
|---|---|---|
| `POST` | `/documents` | Insert one or many documents |
| `GET` | `/documents/<id>` | Fetch document (`?include_vector=true` for raw vector) |
| `DELETE` | `/documents/<id>` | Delete a document |
| `POST` | `/search` | Single search — hybrid vector + BM25 |
| `POST` | `/search/batch` | Batch search — parallel ANN |
| `GET` | `/health` | Live count: documents + index nodes |
| `GET` | `/stats` | Full performance metrics |

### Examples

```bash
# Insert (text auto-embedded if sentence-transformers is installed)
curl -X POST http://localhost:8000/documents \
  -H "Content-Type: application/json" \
  -d '{"documents": [
        {"text": "Machine learning is cool", "metadata": {"source": "wiki"}},
        {"id": "my-id", "vector": [0.1, 0.2, ...], "text": "raw vector insert"}
      ]}'

# Search by text query
curl -X POST http://localhost:8000/search \
  -H "Content-Type: application/json" \
  -d '{"query": "deep learning", "top_k": 5}'

# Search with metadata filter
curl -X POST http://localhost:8000/search \
  -H "Content-Type: application/json" \
  -d '{"query": "AI", "top_k": 5, "filters": {"source": "wiki"}}'

# Batch search
curl -X POST http://localhost:8000/search/batch \
  -H "Content-Type: application/json" \
  -d '{"queries": [{"query": "deep learning"}, {"query": "neural nets"}], "top_k": 3}'

# Health check
curl http://localhost:8000/health

# Stats
curl http://localhost:8000/stats
```

---

## Configuration

All flags live in `utils/schema.py` → `Config`:

```python
class Config:
    # Storage
    DATA_PATH         = "data"
    MEMTABLE_LIMIT    = 1000   # flush to SSTable after N documents in memory
    COMPACT_THRESHOLD = 4      # merge SSTables when this many accumulate
    BATCH_SIZE        = 1000

    # HNSW Index
    VECTOR_DIM            = 768   # all-mpnet-base-v2
    HNSW_M                = 16    # graph connections per node (16 for production recall)
    HNSW_EF_CONSTRUCTION  = 200   # build quality (higher = better recall, slower inserts)
    HNSW_EF_SEARCH        = 150   # candidates explored per query (150 for high recall)

    # Query
    DEFAULT_TOP_K     = 5
    ENABLE_RERANKING  = True
    FILTER_PUSHDOWN   = True

    # Parallelism
    ENABLE_METRICS          = True
    ENABLE_PARALLEL_SEARCH  = True    # GIL-free per-query parallel scoring
    PARALLEL_SEARCH_WORKERS = None    # None = os.cpu_count()
    ENABLE_BATCH_REUSE      = True    # AVX2 union-of-candidates batch path

    # Product Quantization — enable when memory > ~50 GB
    ENABLE_PQ = False
    PQ_M      = 96     # subspaces (must divide VECTOR_DIM evenly)
    PQ_K      = 256    # centroids per subspace (uint8 → max 256)

    # IVF Coarse Index — enable when corpus > ~10M vectors
    ENABLE_IVF     = False
    IVF_N_CLUSTERS = 256   # Voronoi cells — rule of thumb: sqrt(corpus_size)
    IVF_NPROBE     = 8     # cells probed at search time
```

**Tuning cheat sheet:**

| Situation | Setting |
|---|---|
| Need better recall | Raise `HNSW_EF_SEARCH` (150+) or `HNSW_M` (16–32) |
| Slow inserts | Lower `HNSW_EF_CONSTRUCTION` (100) — small recall cost |
| Memory > 50 GB | `ENABLE_PQ = True` — 4–32× compression, ~5% recall loss |
| Corpus > 10M vectors | `ENABLE_IVF = True`, `IVF_N_CLUSTERS = int(sqrt(N))` |
| Recall too low with IVF | Raise `IVF_NPROBE` or disable IVF and use HNSW only |
| Runtime ef tuning | `db._engine.index.set_ef_search(200)` — no rebuild needed |

---

## Benchmarking

All tools are in `tools/`. Data directories are generated at runtime and are not committed.

| Script | What it does |
|---|---|
| `tools/e2e_benchmark.py` | Core benchmark function: insert N docs, run M queries, report insert throughput and search latency (avg / p50 / p99). Optional Recall@K vs brute-force. |
| `tools/run_bench_suite.py` | Runs four preset scenarios: large 20K (vector-only), hybrid 5K (text+vector), HNSW m8/ef20, HNSW m16/ef50. |
| `tools/compare.py` | Head-to-head: AkshamDB vs FAISS vs ChromaDB — insert throughput, search p50/p99, memory delta, Recall@K, recovery time. |
| `tools/profile_hotspots.py` | cProfile hotspot analysis: time breakdown by subsystem (HNSW, WAL, LSM, threading), top hot functions, WAL microbenchmarks, and a recommendation engine that rates bottlenecks HIGH / MEDIUM / INFO with specific action items. |
| `tools/tune_parallel_search.py` | Sweep over thread-pool sizes and ef values. Writes a JSON report of QPS vs latency trade-offs. |
| `tools/cleanup_baseline_benchmarks.py` | Deletes old `baseline_benchmark_YYYYMMDD_HHMMSS/` dirs, keeping the newest N. |
| `run_baseline_benchmark.py` | Quick baseline: `python run_baseline_benchmark.py <n_docs> <n_iters>` |

### Run the full suite

```bash
cd tools
python run_bench_suite.py
# results saved to benchmark_results/
```

### Profile hotspots

```bash
python tools/profile_hotspots.py
```

### Compare against FAISS / ChromaDB

```bash
pip install faiss-cpu chromadb          # optional — skipped gracefully if missing

python tools/compare.py                                  # 10K vectors, dim=128
python tools/compare.py --n 50000 --dim 256 --queries 500
python tools/compare.py --n 10000 --skip-chroma --out results/bench.json
```

### Tune parallel search

```bash
python tools/tune_parallel_search.py             # quick sweep (n=5000)
python tools/tune_parallel_search.py --n 20000   # larger sweep
```

---

## Testing

```bash
python3 test_core.py                              # quick smoke test
python3 -m pytest tests/test_engine.py -v        # unit + integration suite
python3 -m pytest tests/test_reliability.py -v   # crash recovery and WAL tests
python3 -m pytest tests/test_benchmark_integration.py -v   # benchmark integration
python3 run_baseline_benchmark.py 1000 10        # 1000 docs, 10 iterations
```

---

## Data Layout

After the database is created at `<data_path>/`:

```
<data_path>/
  wal.log           — Write-Ahead Log (append-only, JSON lines, fsynced after each write)
  manifest.json     — ordered list of active SSTable files
  sst_1.bin         — SSTable (pickled dict of documents)
  sst_1.bloom       — Bloom filter for sst_1.bin
  ...
  index.bin         — HNSW graph (C++ binary format, v3 serialization with magic header)
  pq_codebooks.pkl  — PQ subspace codebooks (if ENABLE_PQ)
  pq_codes.npy      — encoded corpus codes, shape (N, M) uint8 (if ENABLE_PQ)
  pq_hnsw_ids.npy   — hnsw_id per row in pq_codes.npy (if ENABLE_PQ)
  ivf_index.pkl     — IVF inverted lists and centroids (if ENABLE_IVF)
```

**Index format compatibility:**
- v3 (current): 4-byte magic `HNSW` + int version + data. Unambiguous detection.
- v2 (readable): `FORMAT_VERSION=2` sentinel as first int.
- v1 (readable): raw dim as first int. Edge lists not saved — requires full rebuild on load.

---

## Troubleshooting

**C++ extension not found**
```bash
cd cpp && rm -rf build && mkdir build && cd build
cmake .. -Dpybind11_DIR=$(python -m pybind11 --cmakedir) -DCMAKE_BUILD_TYPE=Release
make -j$(nproc) && cp hnsw_cpp*.so ../../akshamdb/
```

**Data not persisting** — always call `db.close()` or use the context manager:
```python
with akshamdb.open("./data", dim=768) as db:
    db.add(...)
# close() is called automatically
```

**Slow search** — raise ef_search at runtime without rebuilding:
```python
db._engine.index.set_ef_search(200)
# or permanently in Config:
Config.HNSW_EF_SEARCH = 200
Config.HNSW_M         = 16
```

**High memory usage** — enable PQ compression:
```python
Config.ENABLE_PQ = True
# Codebooks are trained automatically on the next full rebuild
```

**Search recall too low** — raise IVF nprobe or disable IVF:
```python
Config.IVF_NPROBE = 32   # probe more cells
# or
Config.ENABLE_IVF = False  # fall back to full HNSW scan
```

**Old index file fails to load** — v3 reader handles v1/v2/v3 automatically. If you see a load error, delete `index.bin` and let it rebuild from LSM on next startup.

---

## Changelog

### v0.3.0 (2026-06-24)
- **[Fix] Real Okapi BM25** — keyword scorer replaced: TF-IDF cosine similarity (sklearn) → Okapi BM25 with an inverted index (k1=1.5, b=0.75, stdlib only). Scoring is now sub-linear in corpus size.
- **[Fix] search_batch O(N²)** — batch BM25 no longer runs dense matrix multiplies per query. Real BM25 inverted index eliminates the bottleneck. Python scoring path vectorised with numpy batch matmul.
- **[Fix] Vectorised candidate scoring** — `_hybrid_score_and_rank` uses a single `doc_mat @ query` call instead of per-doc `np.dot` loops.
- **[Fix] print() → logging** — all lifecycle/rebuild messages in `db_engine.py`, `lsm_store.py`, `ivf_index.py` use `logging.getLogger(__name__)`. No output unless you configure a handler.
- **[Fix] sklearn removed from core** — `scikit-learn` is no longer a required dependency. Core install needs only `numpy`.
- **[Docs] Ingest throughput clarified** — transaction requirement for high throughput documented explicitly.
- **[Docs] REST API (Flask server) and optional features (PQ, IVF) documented.**

### v0.2.0 (2026-06-19)
- **[Fix] Real HNSW beam search** — dual priority-queue implementation (Algorithm 2, Malkov & Yashunin 2018). O(ef·log(ef)·M) vs previous O(N) BFS.
- **[Fix] SELECT-NEIGHBORS-HEURISTIC** — neighbor selection during insert now uses Algorithm 4. Diverse neighbor sets keep the graph navigable across all regions of space.
- **[Fix] Neighbor degree cap** — `mmax0 = 2M` at layer 0, `mmax = M` at upper layers. Over-degree lists pruned with the heuristic, not a truncation.
- **[Fix] Thread-safe RNG** — `get_random_level()` uses `thread_local mt19937` seeded per thread from `random_device`. No mutex needed on the level-assignment path.
- **[Fix] Index format v3** — `HNSW` magic prefix + version field eliminates format-detection ambiguity. v1/v2 files still load correctly.
- **[Fix] WAL fsync** — `_write_wal()` calls `fsync` after every entry. Truncation uses atomic rename.
- **[Fix] LSM SST ID thread safety** — `_alloc_sst_id()` holds `_sst_lock` internally.
- **[Fix] HybridSearch duplicate doc_id** — `add_documents()` updates in-place on duplicate; no zombie BM25 entries.
- **[Fix] Delete syncs BM25** — deleted docs removed from keyword index and query engine state.
- **[Fix] HNSW destructor** — explicit `~HNSW()` releases the thread pool, no resource leak on shutdown.
- **[Fix] Parameter clamping** — `M`, `ef_construction`, `ef_search` clamped to ≥ 1.
- **[Remove] Dead code** — duplicate `Metrics` class in `schema.py`; dead `distance(vector, vector)` in C++ HNSW.
- **[Config] HNSW_M default raised** from 8 → 16 for production-quality graph connectivity.
- **[Config] HNSW_EF_SEARCH default raised** from 50 → 150 for high recall on real corpora.

### v0.1.0
- Initial release: LSM store, HNSW index, query engine, hybrid BM25+vector scoring, PQ, IVF, batch search, REST API.

---

## Built With

- **C++ + pybind11** — HNSW graph, AVX2 distance kernel, persistent thread pool, GIL-free search
- **Python** — LSM store, query engine, hybrid search, ingestion pipeline, REST API
- **Sentence-Transformers** — `all-mpnet-base-v2` embedding model (768-dim)
- **CrossEncoder** — `ms-marco-MiniLM-L-6-v2` reranker
- **Flask** — REST API server
- **NumPy / scikit-learn** — vector ops, TF-IDF BM25 scoring, PQ k-means
