Metadata-Version: 2.4
Name: proximadb_embedded
Version: 0.5.0
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Rust
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Operating System :: MacOS
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: Microsoft :: Windows
Requires-Dist: numpy>=1.21.0
Requires-Dist: pytest>=7.0.0 ; extra == 'dev'
Requires-Dist: pytest-asyncio>=0.20.0 ; extra == 'dev'
Requires-Dist: pytest-benchmark>=4.0.0 ; extra == 'dev'
Requires-Dist: mypy>=1.0.0 ; extra == 'dev'
Requires-Dist: ruff>=0.0.280 ; extra == 'dev'
Provides-Extra: dev
License-File: LICENSE
License-File: NOTICE
Summary: ProximaDB - High-performance vector database with embedded mode
Keywords: vector,database,embedded,similarity,search,ml,ai,embeddings
Home-Page: https://github.com/anvai-labs/proximaDB
Author-email: Vijaykumar Singh <vijay@anvaiops.com>
License: Apache-2.0
Requires-Python: >=3.11
Description-Content-Type: text/plain; charset=UTF-8
Project-URL: Documentation, https://github.com/anvai-labs/proximaDB#readme
Project-URL: Homepage, https://github.com/anvai-labs/proximaDB
Project-URL: Issues, https://github.com/anvai-labs/proximaDB/issues
Project-URL: Repository, https://github.com/anvai-labs/proximaDB

= ProximaDB
:toc: left
:toclevels: 2
:icons: font

[.lead]
**Object-storage-native context serving for SQL, JSON, and vectors**

ProximaDB is a **SQL-first context engine on object storage**, written in Rust. It keeps **one
governed record** on object storage and serves it through standard interfaces. PostgreSQL wire is
the front door, and vectors, JSON and relationships are first-class inside SQL. REST, gRPC and
Arrow Flight serve SDKs and services. Relational, vector, document and graph views are
projections of that one record, and the engine routes each query to the executor that fits it.
Postgres-compatible OLTP is next (link:docs/12-design/adr/ADR-096-product-direction-sql-first-context-engine.adoc[direction: ADR-096]).

What you can adopt **today** is narrower than that vision, and it is stated exactly: one typed
record and one canonical REST API for metadata-aware vector retrieval. Every other capability
graduates only through the link:docs/00-product/MVP_TRUST_CORRIDOR.toml[machine-checked MVP trust corridor] and the support contract in link:docs/SUPPORTED_SURFACE.adoc[`docs/SUPPORTED_SURFACE.adoc`].
Where this README and that contract disagree, the contract wins.

---

== The Vision: One Record, Every Workload, Object-Storage Economics

AI applications and agents need semantic retrieval, transactional state, analytics,
relationships and history over the **same** data. Today that means five databases and the
pipelines between them. ProximaDB's approach:

* **One canonical record.** An identifier, typed properties and embeddings. Documents, graphs
  and time series are projections of it, not copies.
* **Standard outside, co-designed inside.** Clients speak pgwire, REST, gRPC or Arrow Flight.
  Inside, storage format, cache, engine and network are optimised together against the cost that
  dominates in the cloud: **object-store round trips and egress**, not CPU
  (link:docs/12-design/CODESIGN_DIMENSIONAL_ARCHITECTURE_2026_06_19.adoc[co-design architecture]).
* **The engine routes; the client does not choose.** One SQL statement goes to the engine that
  fits it.
* **Tenancy is structural.** Every byte lives under `data/{tenant}/{namespace}/…` and every I/O
  boundary carries the tenant for isolation and metering.

[source,mermaid]
----
%%{init: {"theme": "neutral"}}%%
flowchart TB
  subgraph Clients["Clients"]
    PSQL["psql · JDBC<br/>ORMs · BI"]
    SDK["SDKs<br/>Python · Node · Go · Rust"]
    AGENT["Agents · MCP"]
  end

  subgraph Front["Interfaces"]
    PGW["PostgreSQL wire :5433"]
    REST["REST v2 :5678"]
    GRPC["gRPC v2 :5678"]
    FLIGHT["Arrow Flight :5678"]
  end

  GOV["Governance<br/>tenant isolation<br/>ABAC · row policies<br/>OIDC · SCRAM<br/>metering"]

  subgraph Engines["Engine per workload"]
    VEC["Vector / ANN<br/>SST · HELIX · NOVA"]
    OLAP["SQL analytics<br/>DataFusion"]
    OLTP["SQL OLTP<br/>transactions (ADR-095)"]
    DOC["Document"]
    GRAPH["Graph · ORION"]
    TS["Time series · events"]
  end

  SUB["Storage substrate<br/>WAL (local / durable volume) → memtable →<br/>PAX / Parquet segments + manifests"]

  OBJ[("Object storage<br/>S3 · Blob · GCS<br/>source of truth")]

  PSQL --> PGW
  SDK --> REST
  SDK --> GRPC
  AGENT --> REST
  SDK --> FLIGHT
  PGW & REST & GRPC & FLIGHT --> GOV
  GOV --> VEC & OLAP & OLTP & DOC & GRAPH & TS
  VEC & OLAP & OLTP & DOC & GRAPH & TS --> SUB
  SUB --> OBJ

  classDef supported fill:#d9f2e3,stroke:#2e7d4f,color:#123
  classDef beta fill:#fff1cc,stroke:#a77b00,color:#321
  classDef planned fill:#e8e8f4,stroke:#5b5b8a,color:#223,stroke-dasharray: 4 3
  class REST,VEC,SUB,OBJ supported
  class PGW,GRPC,OLAP,DOC,GRAPH,TS,GOV beta
  class FLIGHT,OLTP planned
----

_Legend: green = Supported, amber = Beta, dashed = Experimental or Planned. Tiers follow
link:docs/SUPPORTED_SURFACE.adoc[`SUPPORTED_SURFACE.adoc`]. "SQL OLTP" is dashed because
Postgres-compatible OLTP is planned (see <<oltp-roadmap>>), even though autocommit DML and basic
transactions over pgwire work today at Beta._

---

== Where It Stands Today

[cols="2,1,4", options="header"]
|===
| Capability | Tier | Notes

| Vector CRUD + search with metadata filters (REST v2, gRPC v2)
| *Supported*
| SST is the Supported engine (pass `"engine": "sst"`; omitting it auto-routes to Beta HELIX until v0.5, TD-ENGSEL-1). Filters apply on both scan and vector search.

| Hybrid retrieval (dense + BM25, RRF / weighted fusion)
| Beta
| Canonical REST `/api/v2/collections/{id}/search`.

| PostgreSQL wire (pgwire)
| Beta
| ANSI SQL for psql and other simple-query clients (extended-protocol drivers fail on many queries today: asyncpg on relational SELECTs, psycopg 3 on parameterized and repeated queries; JDBC untested; TD-PGEXT-1), SCRAM-SHA-256 auth; `ORDER BY embedding <-> '<literal>'::vector LIMIT k` over REST-created collections (Beta). `VECTOR(n)` columns created over pgwire work only in tables that also have a JSON or JSONB column (TD-SQLCOLL-1); projected distances and `vector_search()` arrive in v0.6 (TD-XMODAL-4). `BEGIN` / `COMMIT` / `ROLLBACK` run per ADR-018 P2.D. With the opt-in `[oltp] atomic_commit`, COMMIT is all-or-nothing (one WAL group frame, ADR-095). Without it, a mid-COMMIT failure can leave earlier statements durable. In both modes, statements inside a transaction do not see its own writes until COMMIT. Savepoints and DDL inside a transaction return `0A000`. OLTP parity is planned in link:docs/12-design/adr/ADR-095-postgres-compatible-oltp-on-object-storage.adoc[ADR-095].

| SQL analytics (DataFusion over Parquet / Iceberg)
| Beta
| Parquet-backed tables route to DataFusion. TPC-H / TPC-DS over pgwire are counted ratchets.

| Federated SQL functions (`VECTOR_SEARCH`, `GRAPH_QUERY`, `DOCUMENT_QUERY`, `LOGS`, `METRICS`)
| Beta
| Through `POST /api/v2/unified/federated`. Only `VECTOR_SEARCH` and `DOCUMENT_QUERY` run; `VECTOR_SEARCH` repeats a collection's first result (TD-FEDVEC-1).

| Document, graph (ORION), observability, time series / events
| Beta
| Canonical-record convergence and recovery parity are still in progress (TD-WAL-2).

| HELIX / NOVA engines
| Beta
| VIPER is **deprecated** (ADR-093). Do not create new VIPER collections.

| Arrow Flight as a customer transport, distributed cluster execution, SWIFT / RAPTOR
| Experimental
| Feature-gated; no product claims.

| MLflow-compatible tracking / registry wire
| Opt-in
| Off unless `PROXIMADB_MLFLOW_COMPAT_ENABLE` is set.

| Postgres-compatible OLTP on object storage
| Planned
| <<oltp-roadmap>>
|===

---

== How a Query Flows

[source,mermaid]
----
%%{init: {"theme": "neutral"}}%%
flowchart LR
  C["Client SQL / REST / gRPC"] --> T["Tenant context<br/>(auth → TenantContext)"]
  T --> L["Parse · bind · lower<br/>(relational pipeline)"]
  L --> S{"ComputeScheduler<br/>route_select"}
  S -- "Parquet-backed scan / agg / join" --> DF["DataFusion<br/>vectorized kernels"]
  S -- "point / short / native table" --> NV["Native Volcano<br/>PK fast path"]
  S -- "ORDER BY embedding &lt;-&gt; q / VECTOR_SEARCH" --> AN["Vector engines<br/>SST · HELIX · NOVA"]
  S -- "GRAPH_QUERY" --> OR["ORION graph"]
  DF & NV & AN & OR --> R["Read = memtable ∪ segments<br/>minus dead records (is_dead)"]
  R --> O[("Object storage<br/>ranged GETs · footer pruning · cache")]
----

The scheduler owns engine selection, and every route emits a per-query I/O trace (requests,
bytes, cache hits). That trace is how routing costs are measured instead of guessed
(link:docs/12-design/adr/ADR-064-query-engine-pipeline-decision-dispatch-lowering-trace.adoc[ADR-064]).

== The Storage Spine

[source,mermaid]
----
%%{init: {"theme": "neutral"}}%%
flowchart LR
  W["Write<br/>(INSERT / UPDATE / DELETE / upsert)"] --> WAL["WAL append<br/>fsync per sync_mode"]
  WAL --> M["Memtable<br/>(bounded; spills)"]
  M -- "flush: size · time · capacity budget" --> SEG["Immutable segments<br/>PAX (vector / multimodal)<br/>Parquet Layer A (relational)"]
  SEG --> MAN["Manifest commit (CAS)<br/>Iceberg-readable"]
  MAN -- "only after durable" --> TR["WAL truncation<br/>at unflushed minimum"]
  SEG -- "compaction" --> SEG
  subgraph Paths["Every key built by DrPathBuilder"]
    P["data/{tenant}/{namespace}/{collection}/…"]
  end
  SEG -.-> P
----

* **Durability:** the WAL lives on local disk or a durable volume (ADR-069). Segments and
  manifests live on object storage, and a manifest commit must be durable before WAL truncation
  (ADR-094).
* **Read correctness:** every read surface filters tombstoned and TTL-expired records with one
  predicate, every write invalidates caches after commit, and `Strong` reads bypass the cache.
* **Formats:** PAX for vector and random access, Parquet for relational scans and interop.
  Changes are mixed-read-safe and version-marked (ADR-010, ADR-094).

---

[[roadmap]]
== Roadmap

One plan, by release (link:docs/current/ROADMAP.adoc[STRATEGIC_ROADMAP]); each
release closes on executable exit criteria:

* **v0.5 — Correct and durable.** One total order for record versions on every read surface; group
  commit and REST/gRPC writes on the atomic path; bounded memtable and WAL; tenant safety on every
  surface.
* **v0.6 — Vectors and context inside SQL.** One vector-SQL plan, hybrid retrieval, documents and
  graph as projections, ORM compatibility; VIPER, SWIFT and RAPTOR removed.
* **v0.7 — Evidence on real object stores.** Crash/restart and economic suites on a cloud store;
  published, ledger-backed comparisons.
* **v1.0 — SQL front door in the supported promise.** pgwire SQL and transactions become Supported.

[[oltp-roadmap]]
=== Postgres-Compatible OLTP on Object Storage

*Planned — link:docs/12-design/adr/ADR-095-postgres-compatible-oltp-on-object-storage.adoc[ADR-095].*
The goal is to serve OLTP SQL with PostgreSQL semantics while data stays on object storage. Only
the uncheckpointed WAL tail lives on a network-attached volume (EBS, Azure Disk, GCP PD/Hyperdisk,
or EFS) that survives compute loss and **reattaches to a replacement machine**.

[source,mermaid]
----
%%{init: {"theme": "neutral"}}%%
flowchart LR
  APP["Apps · ORMs · pgbench"] --> N1
  subgraph N1["Compute (replaceable)"]
    TX["Transactions<br/>LSN snapshots · row locks"]
  end
  TX -- "1 commit record,<br/>group-commit fsync" --> VOL[("Durable WAL volume<br/>EBS · Azure Disk · PD · EFS")]
  TX -- "checkpoint" --> OBJ[("Object storage<br/>segments · manifests · WAL archive")]
  N1 -- "boot: win writer epoch (CAS)" --> OBJ
  VOL -. "compute lost → reattach<br/>to a new machine, replay tail" .-> N2["Compute (new)"]
----

How this compares: Aurora, Neon/Lakebase, Azure HorizonDB and AlloyDB make "the log is the
database" with a storage fleet. Classic RDS, Cloud SQL and Azure Flexible Server fsync to an
attached volume and recover by reattaching it. ProximaDB takes the reattach model for **only the
WAL tail** and keeps everything else on object storage. The volume stays small and fast to move,
and the analytics and vector engines read the same data with no CDC copy.

Ranked plan (each step is default-OFF until its conformance ratchet passes):

. **Atomic commit record.** A transaction is one WAL record (TD-PGOLTP-1). *Available opt-in:*
  `[oltp] atomic_commit = true`. Cross-session group commit is next.
. **Durable volume + writer-epoch fencing + reattach recovery.** No split-brain (TD-PGOLTP-2).
  *Available opt-in:* `[oltp.durable_volume]` (root, `control_url`). Epoch stamping in the WAL is
  next.
. **PK / UNIQUE / FK on by default, inside the commit** (TD-PGOLTP-3).
. **MVCC snapshots, read-your-writes, row locks, `FOR UPDATE`, REPEATABLE READ** (TD-PGOLTP-4).
. **Sequences / `SERIAL` / `IDENTITY`, `RETURNING`, `ON CONFLICT`** (TD-PGOLTP-5).
. **Conformance ratchet:** pgbench invariant, isolation anomalies, ORM matrix, reattach crash
  matrix (TD-PGOLTP-6).
. Secondary indexes, WAL archive for PITR and cross-AZ restore, savepoints and transactional DDL
  (TD-PGOLTP-7..9).

---

== Performance

Only figures recorded as *measured* in the evidence ledger
(`docs/_internal/roadmap/BENCHMARK_EVIDENCE.toml`) are quoted here. They are reference points,
not an SLA.

[cols="1,2", options="header"]
|===
| Metric | Measured reference
| Recall@10 | ~97% (historical internal HNSW reference; not an SLA). HNSW M=32, SST engine, 100K vectors, single host: ledger entry `recall_at_10`
|===

* link:docs/10-quality/benchmarks/BENCHMARKS.adoc[Benchmark notes]: methodology, hardware, scaling
* link:docs/10-quality/benchmarks/ENGINE_PERFORMANCE_BENCHMARK.adoc[Engine performance]
* link:scripts/run_benchmarks.sh[Run the benchmarks yourself]

---

== Quick Start

The supported first-run path is Docker plus the canonical v2 smoke test. The smoke fails on any
incorrect status, missing record, filter leak, unsupported write-contract signal or deprecated
route.

[source,bash]
----
docker run --rm -d --name proximadb-mvp -p 5678:5678 vjsingh1984/proximadb:latest
python3 scripts/mvp_smoke.py --base-url http://127.0.0.1:5678
----

See link:docs/01-quick-start/CONTEXT_SERVING_MVP.adoc[Context Serving MVP Quickstart] for the
verified request sequence and production caveats. The release page is authoritative for available
binary assets.

=== From Source (Rust)

[source,bash]
----
git clone https://github.com/anvai-labs/proximaDB
cd proximaDB
cargo run --release --bin proximadb-server   # REST/gRPC/Flight on 5678, pgwire on 5433
----

---

== API Examples (REST v2)

=== Vector Search (Supported)

[source,bash]
----
curl -X POST http://localhost:5678/api/v2/collections/docs/search \
  -H "Content-Type: application/json" \
  -d '{
    "vector": [0.1, 0.2, 0.3],
    "top_k": 10,
    "filters": [
      {"field": "category", "op": "eq", "value": "tech"}
    ]
  }'
----

=== Document Insert (Beta)

[source,bash]
----
curl -X POST http://localhost:5678/api/v2/document-collections/docs/documents \
  -H "Content-Type: application/json" \
  -d '{
    "document": {"title": "Runbook", "service": "api", "severity": "warn"}
  }'
----

=== Graph Traverse (Beta)

[source,bash]
----
curl -X POST http://localhost:5678/api/v2/graphs/topology/traverse \
  -H "Content-Type: application/json" \
  -d '{
    "start_node_id": "service-api",
    "max_depth": 2
  }'
----

=== Log Ingestion (Beta)

[source,bash]
----
curl -X POST http://localhost:5678/api/v2/observability/namespaces/prod/logs \
  -H "Content-Type: application/json" \
  -d '{
    "message": "request failed",
    "severity": "error",
    "service": "api"
  }'
----

=== Federated SQL (Beta)

[source,bash]
----
curl -X POST http://localhost:5678/api/v2/unified/federated \
  -H "Content-Type: application/json" \
  -d '{
    "query": "SELECT * FROM VECTOR_SEARCH('\''products'\'', '\''[0.1, 0.2, 0.3]'\'', 10)"
  }'
----

---

== SQL Extensions (Beta)

These table functions run through the federated query endpoint
(`POST /api/v2/unified/federated`, shown above), not over pgwire. They belong to the legacy
federated path that TD-FEDPATH-1 isolates; the direction is standard SQL over pgwire (ADR-096,
TD-XMODAL-4). Known defect: `VECTOR_SEARCH` returns a collection's first query's rows on every
later call (TD-FEDVEC-1).

[source,sql]
----
-- Vector
SELECT * FROM VECTOR_SEARCH('products', '[0.1, 0.2, 0.3]', 10);

-- Document: JSON path filter (the document collection must exist)
SELECT * FROM DOCUMENT_QUERY('orders', 'status = "pending"');
----

`GRAPH_QUERY`, `LOGS`, `METRICS` and cross-model joins through this endpoint do not work as
documented today.

---

== Web UI Dashboard (Beta)

[source,bash]
----
cd ui && npm install && npm start   # Vite dev server on :3000, proxies /api to :5678
----

SQL editor (Monaco), graph explorer (Cytoscape.js), system metrics, dark/light theme.

---

== Documentation

* link:docs/README.md[Documentation home]
* link:docs/SUPPORTED_SURFACE.adoc[Support contract]: what is Supported / Beta / Experimental / Planned
* link:docs/current/VISION.adoc[Vision]: the Context Database
* link:docs/12-design/README.adoc[Architecture index] and
  link:docs/12-design/SYSTEM_MAP_2026_05_30.adoc[system map]
* link:docs/12-design/adr/README.adoc[Architecture decision records], including
  link:docs/12-design/adr/ADR-094-unified-storage-substrate-and-compute-seam.adoc[ADR-094 (storage substrate)] and
  link:docs/12-design/adr/ADR-095-postgres-compatible-oltp-on-object-storage.adoc[ADR-095 (Postgres-compatible OLTP)]
* link:docs/01-quick-start/[Quick start] · link:docs/02-guides/[Guides] ·
  link:docs/05-concepts/[Concepts] · link:docs/03-api-reference/[API reference] ·
  link:docs/04-operations/[Operations] · link:docs/06-internals/[Internals]
* link:CHANGELOG.md[Changelog]

---

== License

Apache License 2.0. See link:LICENSE[LICENSE].

Open-core boundary: the full single-node engine is free forever with no feature gates. See
link:OPEN_CORE.md[OPEN_CORE.md] for what the commercial layer sells (operations, never features).

== Branching and Releases

* `develop` is the integration branch. Feature branches open PRs into `develop` (CI-gated).
* Promotion is staged: `develop` → `qa` (staging validation) → `main` (release). Releases are
  tagged on `main`.
* `qa` and `main` always reflect promoted states. Nothing merges directly to either without
  passing through the previous stage.

