Metadata-Version: 2.5
Name: dot-search
Version: 2.0.0
Summary: Augment existing database tables with vector and BM25 search
Project-URL: Homepage, https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-search
Project-URL: Repository, https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-search
Project-URL: Issues, https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-search/-/issues
Author-email: Kannon For Deep Tech <louis.letarnec@deepika.ai>
License-Expression: AGPL-3.0-or-later
License-File: LICENSE.md
Keywords: bm25,deepika,embeddings,hybrid,open-toolbox,search,vector
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: <3.14,>=3.12
Requires-Dist: dot-inference<3,>=2.0
Requires-Dist: pgvector>=0.3
Requires-Dist: psycopg2-binary>=2.9
Requires-Dist: pydantic>=2.0
Requires-Dist: sqlalchemy>=2.0
Requires-Dist: typing-extensions>=4.16
Description-Content-Type: text/markdown

# dot-search

[![PyPI](https://img.shields.io/pypi/v/dot-search)](https://pypi.org/project/dot-search/)
![Python Version](https://img.shields.io/badge/python-3.12%2B-blue)
[![Licence: AGPL v3](https://img.shields.io/badge/licence-AGPL--3.0--or--later-blue)](LICENSE.md)
[![Pipeline](https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-search/badges/main/pipeline.svg)](https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-search/-/pipelines)

**Add vector, keyword and exact search to the database tables you already have.**

```python
from dot_search import SearchEngine, TableConfig, EmbeddingConfig

engine = SearchEngine(db_url="postgresql+psycopg2://user:pass@localhost/mydb")

engine.index(
    TableConfig(table="articles", embeddings=[EmbeddingConfig(key="body", source_column="body", dimension=1536)]),
    batch_size=500,
)

for hit in engine.search("neural networks", "articles"):
    print(hit.id, hit.score)
```

`dimension` is the vector size your embedding model returns — 1536 for OpenAI's
`text-embedding-3-small`, 1024 for Mistral's `mistral-embed`. It has to match, or
indexing fails on the first batch.

## Why dot-search

Adding semantic search to an existing application usually means standing up a
separate vector database, then keeping it in sync with the tables that hold the
real data — a second source of truth, a sync job, and a new failure mode.

dot-search takes the other route: it adds embedding columns and search indexes
**to your existing tables**, in your existing database. Your rows stay where they
are, joins keep working, and a search result is a row you already own. Vector,
BM25 and exact matching can run together, with results merged through Reciprocal
Rank Fusion.

## Features

- Vector search on any text column, through pgvector
- BM25 keyword search, through ParadeDB `pg_search`
- Exact substring matching
- Hybrid search fusing several strategies via Reciprocal Rank Fusion
- Several independent indexes on the same table
- Row filtering with a parameterised `Filter` API — filter values are bound, so
  they can safely carry end-user input
- Declarative configuration, persisted so an index can be reopened later
- Batch or caller-driven indexing
- Pluggable embedders, defaulting to [dot-inference](https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-inference)
- SQLite backend for local development and testing

## Requirements

### PostgreSQL (production)

Requires [pgvector](https://github.com/pgvector/pgvector) and [ParadeDB pg_search](https://github.com/paradedb/paradedb):

```sql
CREATE EXTENSION vector;
CREATE EXTENSION pg_search;
```

### SQLite (development and testing)

Uses [sqlite-vec](https://github.com/asg017/sqlite-vec) and FTS5. Convenient for
tests, but not intended for production.

## Installation

```bash
pip install dot-search
```

## Configuration

Embeddings are produced through `dot-inference`, which reads its settings from
`DOTI_`-prefixed environment variables:

| Variable | Example |
|---|---|
| `DOTI_EMBEDDING__PROVIDER` | `openai` |
| `DOTI_EMBEDDING__OPENAI__MODEL_NAME` | `text-embedding-3-small` |
| `DOTI_EMBEDDING__OPENAI__API_KEY` | your API key |

Works with any OpenAI-compatible API (OpenAI, Mistral, OpenRouter, vLLM, TGI,
Ollama, …). See the [dot-inference settings](https://gitlab.com/deepika6190303/deepika-open-toolbox/dot-inference)
for the full list, or pass your own embedding function instead.

## Quick start

```python
from dot_search import (
    BM25Config,
    EmbeddingConfig,
    ExactConfig,
    Filter,
    SearchConfig,
    SearchEngine,
    TableConfig,
)

engine = SearchEngine(db_url="postgresql+psycopg2://user:pass@localhost/mydb")

# --- 1. Index a table with vector search ---
engine.index(
    TableConfig(
        table="articles",
        embeddings=[EmbeddingConfig(key="body", source_column="body", dimension=1536)],
    ),
    batch_size=500,
)

# --- 2. Search ---
for hit in engine.search("neural networks", "articles"):
    print(hit.id, hit.score)

# --- 3. Combine several strategies on the same table ---
engine.index(
    TableConfig(
        table="articles",
        embeddings=[
            EmbeddingConfig(key="body", source_column="body", dimension=1536),
            EmbeddingConfig(key="title", source_column="title", dimension=1536),
        ],
        bm25=[
            BM25Config(key="title_bm25", source_column="title"),
            BM25Config(key="body_bm25", source_column="body"),
        ],
        exact=[ExactConfig(key="name_exact", source_column="name")],
    ),
    batch_size=500,
)

# --- 4. Hybrid search with row filters and weight overrides ---
results = engine.search(
    "fermentation and gut health",
    "articles",
    SearchConfig(
        limit=10,
        where=[
            Filter(column="published_year", op="gte", value=2022),
            Filter(column="topic", op="eq", value="health"),
        ],
        weights={"body": 1.0, "title": 0.3, "title_bm25": 0.5, "body_bm25": 2.0},
    ),
)

# --- 5. Single-strategy search ---
results = engine.search("fermentation", "articles", SearchConfig(strategy="bm25"))
results = engine.search("Dupont", "articles", SearchConfig(strategy="exact"))

# --- 6. Several independent indexes on one table ---
engine.index(
    TableConfig(
        table="articles",
        index_id="article_titles",
        embeddings=[EmbeddingConfig(key="title_only", source_column="title", dimension=1536)],
    ),
    batch_size=500,
)
results = engine.search("gut health", "article_titles")

# --- 7. Index specific rows (caller-driven batching) ---
engine.index(
    TableConfig(
        table="articles",
        embeddings=[EmbeddingConfig(key="body", source_column="body", dimension=1536)],
    ),
    row_ids=[1, 5, 10],
)
```

## Search strategies

| Strategy | What it uses |
|----------|-------------|
| `"hybrid"` | Vector + BM25 + exact (any configured), fused via RRF (default) |
| `"vector"` | Vector similarity only |
| `"bm25"` | BM25 keyword search only |
| `"exact"` | Substring (`LIKE`) search only |

### `min_score`

`SearchConfig.min_score` drops results scoring below a threshold. It is only
meaningful for single-strategy searches (`vector`, `bm25`, `exact`), where raw
scores are preserved and interpretable — a cosine similarity, for instance. It is
rejected in `hybrid` mode, because RRF scores are rank-based and carry no
absolute meaning; use `limit` there instead.

## Filtering

Row filters are expressed with `Filter` objects rather than SQL fragments:

```python
SearchConfig(where=[
    Filter(column="status", op="eq", value="published"),
    Filter(column="lang", op="in", value=["fr", "en"]),
    Filter(column="deleted_at", op="is_null"),
])
```

Available operators: `eq`, `ne`, `lt`, `lte`, `gt`, `gte`, `in`, `not_in`,
`like`, `ilike`, `is_null`, `is_not_null`.

Column names are validated as plain identifiers, and values are passed to the
database as bound parameters — never interpolated into SQL. Filter values are
therefore safe to build from end-user input.

## Batching modes

`index()` requires exactly one of `batch_size` or `row_ids`:

- **`batch_size=N`** — dot-search reads all rows and batches embeddings internally in chunks of N.
- **`row_ids=[...]`** — the caller decides which rows to embed. dot-search only reads, embeds, and writes those specific rows in a single pass. This lets you drive batching from the outside — useful when you want to control concurrency, retry individual batches, or stream IDs from a queue.

Setup steps (creating columns, saving config) run in both modes — they're idempotent.

## Stability

`dot-search` follows semantic versioning: everything exported from the top-level
package is covered, anything underscore-prefixed is internal and may change in
any release. Public names are never removed without a deprecation period.

```toml
dependencies = ["dot-search>=2.0,<3"]
```

See [docs/VERSIONING.md](docs/VERSIONING.md) for the full policy.

## Roadmap

- [ ] Async engine API
- [ ] Reranking stage after fusion
- [ ] Incremental re-indexing on row updates

## Documentation

| Document | Contents |
|---|---|
| [docs/DEVELOPMENT.md](docs/DEVELOPMENT.md) | Environment setup, tests, code style |
| [docs/VERSIONING.md](docs/VERSIONING.md) | Versioning, deprecation policy, how to depend on this package |
| [docs/PUBLISHING.md](docs/PUBLISHING.md) | Cutting a release |
| [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) | Architecture and data-flow diagrams |
| [CHANGELOG.md](CHANGELOG.md) | Release history |

## Contributing

Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the DCO
sign-off requirement, the licensing terms that apply to contributions, and how to
submit a change.

## Licence

Copyright (C) 2026 Kannon For Deep Tech (deepika)

This software is distributed under the GNU Affero General Public License,
version 3 or later — see [LICENSE.md](LICENSE.md).

A commercial licence is available for use in proprietary environments.
Contact: louis.letarnec@deepika.ai
