Metadata-Version: 2.4
Name: grainz
Version: 0.1.0.dev0
Summary: The embedding lifecycle, end to end — recommend, generate, store, search, cache, track, model, inspect, and visualize embeddings behind one consistent surface.
Project-URL: Homepage, https://github.com/piyush182004/grainz
Project-URL: Repository, https://github.com/piyush182004/grainz
Project-URL: Documentation, https://github.com/piyush182004/grainz/tree/main/docs
Project-URL: Issues, https://github.com/piyush182004/grainz/issues
Author: Piyush Kumar Mondal
License-Expression: MIT
License-File: LICENSE
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Typing :: Typed
Requires-Python: <3.14,>=3.10
Requires-Dist: kaleido>=0.2
Requires-Dist: numpy>=1.24
Requires-Dist: plotly>=5.20
Requires-Dist: qdrant-client>=1.9
Requires-Dist: redis>=5.0
Requires-Dist: scikit-learn>=1.4
Requires-Dist: scipy>=1.10
Requires-Dist: sqlite-vec>=0.1
Requires-Dist: umap-learn>=0.5
Provides-Extra: all
Requires-Dist: pillow>=10.0; extra == 'all'
Requires-Dist: sentence-transformers>=3.0; extra == 'all'
Requires-Dist: soundfile>=0.12; extra == 'all'
Requires-Dist: torch>=2.2; extra == 'all'
Requires-Dist: transformers>=4.40; extra == 'all'
Provides-Extra: dev
Requires-Dist: hypothesis<=6.140.3,>=6.100; extra == 'dev'
Requires-Dist: mypy<2.0,>=1.10; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Requires-Dist: ruff<0.16,>=0.15; extra == 'dev'
Provides-Extra: models
Requires-Dist: pillow>=10.0; extra == 'models'
Requires-Dist: sentence-transformers>=3.0; extra == 'models'
Requires-Dist: soundfile>=0.12; extra == 'models'
Requires-Dist: torch>=2.2; extra == 'models'
Requires-Dist: transformers>=4.40; extra == 'models'
Description-Content-Type: text/markdown

<p align="center">
  <img src="./assets/grainz-logo.svg" alt="grainz" width="560">
</p>

<p align="center">
  <em>still building</em>
</p>

<p align="center">
  <img src="https://img.shields.io/badge/python-3.10--3.13-blue" alt="Python 3.10-3.13">
  <img src="https://img.shields.io/badge/mypy-strict-blue" alt="mypy strict">
  <img src="https://img.shields.io/badge/license-MIT-green" alt="MIT License">
</p>

**The embedding lifecycle, end to end.** Recommend a model, generate, store, search, cache, track usage, model, inspect, and visualize embeddings through one consistent surface — built on a single primitive, the `Embeddings` object.

`grainz` — every embedding starts as a grain of information; this toolkit is where you grow it.

---

## Why grainz

Working with embeddings today means gluing together four or five libraries — one to generate, one to store, one to model, one to visualize — each with its own types and none aware of how often you're actually hitting a model. `grainz` unifies that lifecycle behind a single object. Everything the library does consumes and returns `Embeddings`, so the whole toolkit feels like one tool instead of five.

- **Three calls, not thirty lines.** `Index(path)` → `.add(items)` → `.search(query)`. Storage, hybrid retrieval, caching, and usage tracking are wired for you — and every piece stays reachable when you outgrow the defaults.
- **One primitive.** `Embeddings` = vectors + metadata + optional labels. Every function speaks it.
- **Text, image, and audio.** CLIP for images, CLAP for audio, both sharing a text-queryable embedding space. Same `Embeddings`, same stores, same search.
- **Caching that pays for itself.** Reworded repeat questions return from cache instead of re-hitting the model. Set `cache_ttl="1h"` and watch `calls_saved` climb.
- **Pluggable storage.** Swap `json` → `sqlite-vec` → `qdrant` with one line; behavior is identical, enforced by a shared conformance suite.
- **Usage tracking.** Calls made, calls saved, hit rate, tokens, latency — provider-agnostic, no pricing tables to go stale.
- **Modeling, inspection, visualization** — all on the same embeddings, wrapping proven libraries rather than reinventing them.

## Install

```bash
pip install grainz
```

Installs in seconds. That covers storage, hybrid search, semantic caching, usage
tracking, modeling, quality checks, and visualization.

Neural embedding models (sentence-transformers for text, CLIP for images, CLAP for audio)
live behind an extra, because torch is several gigabytes and most pipelines never need it:

```bash
pip install "grainz[models]"
```

You only need that if you want `grainz` to *generate* embeddings for you. Bringing your own
vectors, or using the built-in dependency-free `hashing` encoder, works with the base install.

Two backends talk to services you run yourself. Their client libraries are already installed:

| Feature | Needs |
|---|---|
| `RedisCache` | a Redis server (`docker run -p 6379:6379 redis`) |
| `QdrantStore` | a Qdrant server (`docker run -p 6333:6333 qdrant/qdrant`) |

Everything else — including `SqliteVecStore`, the default — is local and needs nothing.

## Quickstart

```python
import grainz as gz

index = gz.Index("support.db", cache_ttl="1h")
index.add(["To reset your password, click Forgot Password on the login page.",
           "Invoices live under Billing > History.",
           "Cancel your subscription from Account Settings."])

hits = index.search("how do I reset my password?")
print(hits.metadata[0]["text"])
```

Three lines, and you already have a persistent vector store, hybrid (vector + keyword)
retrieval, a semantic cache, and usage tracking.

### Proof it works: caching a real LLM

Six support questions through `gemma:2b` on Ollama. Three are rephrasings of earlier ones —
real generations, real timings ([`examples/ollama_cache_demo.py`](./examples/ollama_cache_demo.py)):

```
#   question                                      source       seconds
------------------------------------------------------------------------
1   How do I reset my password?                   gemma:2b      3.106s
2   Where can I find my invoices?                 gemma:2b      2.898s
3   How can I reset my password?                  CACHE         0.046s
4   How do I cancel my subscription?              gemma:2b      2.899s
5   I forgot my password, how do I reset it?      CACHE         0.074s
6   Where do I download my invoices?              CACHE         0.035s

avg LLM latency: 2.968s      avg cache latency: 0.052s
cache hits were 57x faster than calling the model
calls_made=3  calls_saved=3  hit_rate=50%
```

Question 5 shares almost no wording with question 1. A cache keyed on the string misses it
entirely; matching on **meaning** catches it.

### Why `cache_ttl` matters

Users ask the same question in different words. `Index` embeds each query once, checks
whether a **near-identical** question was already answered within `cache_ttl`, and if so
returns that answer without re-embedding or re-searching — so no model call, no API spend.

```python
index = gz.Index("support.db", cache_ttl="1h", cache_threshold=0.95)

index.search("how do I reset my password")      # miss -> real search
index.search("how do I reset my password")      # hit  -> free
index.search("how can I reset my password?")    # hit  -> free (reworded, still matches)

print(index.report())
# calls_made=1, calls_saved=2, hit_rate=0.67
```

- `cache_ttl` — how long an answer stays fresh: `"30s"`, `"15m"`, `"1h"`, `"7d"`, or seconds.
  `None` never expires. Lower it when your underlying data changes often.
- `cache_threshold` — how similar two questions must be to count as the same one
  (`0.95` strict, `0.85` loose).
- `add()` clears the cache automatically, so newly indexed content is visible to the very
  next search rather than hidden until the TTL lapses.
- `cache_backend=gz.RedisCache(...)` — share the cache across processes so a warm answer
  from one replica serves all of them.

### Images and audio

Same three calls; just load a model that can see or hear.

```python
index = gz.Index("photos.db", model="clip-vit-base-patch32")
index.add(["cat.jpg", "dog.jpg", "beach.png"], modality="image")

# CLIP puts text and images in one space, so a text query ranks images.
hits = index.search("a photo of a cat")
```

Audio works identically with `model="clap-htsat-unfused"` and `modality="audio"`.

### The pieces are still yours

`Index` is composition, not a wall. Every part stays public and reachable, so you can
drop to the lower level whenever the defaults stop fitting:

```python
index.encoder   # the loaded model
index.store     # SqliteVecStore / JsonStore / QdrantStore
index.cache     # SemanticCache
index.tracker   # UsageTracker

# ...or bypass Index entirely and wire it yourself:
store = gz.SqliteVecStore("index.db")
store.add(gz.registry.load("bge-small-en-v1.5").encode(texts))
results = gz.hybrid_search(store, "query", encoder=model, k=5, alpha=0.7)
```

## The rest of the lifecycle

Every operation below takes the same `Embeddings` object, so nothing needs converting
between steps.

```python
emb = index.store.all()      # or model.encode(texts) directly

# Model — via scikit-learn under the hood
clf   = gz.Classifier(n_neighbors=3).fit(train_emb)      # or gz.Regressor
preds = clf.predict(test_emb)
anomalies = gz.detect_anomaly(emb, contamination=0.05)   # bool array, IsolationForest

# Inspect — vector-native quality, not generic dataframe QA
report = gz.quality_report(emb, reference=last_weeks_emb)
# QualityReport(duplicate_count=3, dead_dimensions=[], effective_rank=41.2, drift=0.18, ...)

# Visualize
reduced = gz.project(emb, method="umap")   # or method="pca"
fig = gz.plot(reduced, color_by="topic")   # cluster scatter
fig = gz.plot(emb, kind="heatmap")         # similarity heatmap, any dimension
fig = gz.plot(emb, kind="bar")             # class distribution
fig = gz.plot(emb, kind="variance")        # per-dimension variance
fig.write_image("plot.png")                # static PNG, no browser needed
```

## Features

Every feature has its own page with a runnable example and its real output —
**[full documentation here](./docs/)**.

### Core

| Feature | What it does | Docs |
|---|---|---|
| **`Index`** | Encoder + store + cache + tracker in three calls | [index-facade.md](./docs/index-facade.md) |
| **`Embeddings`** | The one primitive every module speaks: vectors + metadata + labels | [embeddings.md](./docs/embeddings.md) |

### Models and retrieval

| Feature | What it does | Docs |
|---|---|---|
| **`recommend_model()`** | Ranks catalog models against your constraints, with a stated reason per rank | [registry.md](./docs/registry.md) |
| **`registry.load()`** | Loads any catalog model, incl. a dependency-free offline baseline | [registry.md](./docs/registry.md) |
| **`SqliteVecStore`** | Single-file embedded vector store — the default | [stores.md](./docs/stores.md) |
| **`JsonStore`** | Human-readable store for tests and demos | [stores.md](./docs/stores.md) |
| **`QdrantStore`** | Server-backed store for scale | [stores.md](./docs/stores.md) |
| **`hybrid_search()`** | Vector + BM25 keyword retrieval, blended by `alpha` | [search.md](./docs/search.md) |
| **CLIP / CLAP** | Image and audio encoding; text queries rank media results | [multimodal.md](./docs/multimodal.md) |

### Efficiency

| Feature | What it does | Docs |
|---|---|---|
| **`SemanticCache`** | Reworded repeat questions return from cache — no model call | [cache.md](./docs/cache.md) |
| **`MemoryCache` / `RedisCache`** | Per-process, or shared across replicas | [cache.md](./docs/cache.md) |
| **`UsageTracker`** | Calls made, calls saved, hit rate, tokens, latency. Never pricing | [usage.md](./docs/usage.md) |
| **Provider adapters** | OpenAI-compatible, Gemini, and Ollama response shapes | [usage.md](./docs/usage.md) |

### Analysis

| Feature | What it does | Docs |
|---|---|---|
| **`Classifier` / `Regressor`** | Train on embeddings; swap in any scikit-learn estimator | [modeling.md](./docs/modeling.md) |
| **`detect_anomaly()`** | Flag outlier vectors via IsolationForest | [modeling.md](./docs/modeling.md) |
| **`quality_report()`** | Duplicates, dead dimensions, dimensional collapse, drift | [quality.md](./docs/quality.md) |
| **`project()`** | PCA or UMAP reduction, returned as `Embeddings` | [visualization.md](./docs/visualization.md) |
| **`plot()`** | Scatter, similarity heatmap, class distribution, per-dim variance → PNG or interactive | [visualization.md](./docs/visualization.md) |

## Known limitations

Stated plainly, because knowing where a tool stops working is part of using it well:

- **`hybrid_search` loads the entire corpus on every call.** Fine under ~100k vectors;
  unusable at millions (~95 GB RAM for 9 GB of vectors — measured, see
  [`search.md`](./docs/search.md)). Use `store.search()` above that.
- **The catalog's `quality_tier` values are illustrative**, hand-written for ranking, not
  scraped from a live benchmark.
- **Duplicate detection and similarity heatmaps are O(n²)** in memory.
- **No quantization** — vectors are stored `float32` and round-trip bit-exactly. Smaller
  storage (`int8`, `bit`) is a real feature that hasn't been built.

Still open: a CI pipeline and a `mypy --strict` gate.

## Storage backends

| Backend | Use it for | Vector search | Server |
|---|---|---|---|
| `JsonStore` | Testing, portability, demos | No (linear scan) | No |
| `SqliteVecStore` | **Local & embedded — the default** | Yes | No |
| `QdrantStore` | Production & scale | Yes | Yes |

All three implement the same `VectorStore` protocol and pass the same conformance suite. Your code doesn't change when you switch.

## Usage tracking

`grainz` measures **usage, not billing**. Price tables go stale, differ per provider, split input/output rates, and don't apply to local models at all — so `grainz` tracks the numbers that mean the same thing everywhere.

```python
tracker.report()
# UsageReport(
#   total_requests=1000,
#   calls_made=340,     calls_saved=660,     hit_rate=0.66,
#   tokens_sent=48_200, tokens_saved=93_600,
#   time_actual=21.8s,  time_saved=42.3s,    time_estimated=True,
#   by_model={
#     "openrouter/claude-sonnet-4.5": {calls_made: 120, calls_saved: 280, hit_rate: 0.70},
#     "ollama/bge-m3":                {calls_made: 220, calls_saved: 380, hit_rate: 0.63},
#   }
# )
```

Works with any provider — OpenAI-compatible endpoints (OpenAI, OpenRouter, Together, Groq, Mistral), Gemini native, and local models via Ollama or torch. A thin adapter normalizes token counts and latency; providers that report no tokens still yield calls and hit rate.

## Dataset quality checks

`quality_report()` catches the failure modes that are specific to embeddings, not the ones a generic dataframe QA tool already covers:

| Check | Catches |
|---|---|
| Near/exact duplicates | Cosine similarity ≥ threshold — data leakage between train/test, redundant indexing |
| Dead / low-variance dimensions | A dimension that's constant (or nearly) across every vector — dead capacity |
| Effective rank | Exponentiated entropy of the singular-value spectrum — **dimensional collapse** (all vectors pointing the same way), a classic failure mode of poorly-trained encoders |
| Drift | Centroid shift vs. a reference set, normalized by the reference's own spread — is this week's data still the same distribution as last week's? |

```python
report = gz.quality_report(new_emb, reference=baseline_emb, duplicate_threshold=0.98)
if report.effective_rank < new_emb.dim * 0.1:
    print("warning: embeddings look collapsed")
if report.drift > 2.0:
    print("warning: this batch has drifted from baseline")
```

## Repository layout

```
src/grainz/      the library itself -- the only thing that ships in the wheel
tests/           automated tests. Run by CI, not meant to be read (pytest)
examples/        runnable scripts that teach the library
benchmarks/      scripts that measure it; the source of the numbers in docs/
docs/            one page per feature, each with a verified example
```

The three script directories look similar but answer different questions:

| Directory | Question it answers | Who runs it |
|---|---|---|
| `tests/` | "Is it still correct?" | CI, on every change |
| `examples/` | "How do I use this?" | You, once, while learning |
| `benchmarks/` | "How fast / how big / where does it break?" | You, when you need evidence |

Keeping them apart is why the docs can quote hard numbers: every limitation claimed in
[`docs/`](./docs/) traces back to a script in `benchmarks/` you can rerun yourself.

Generated output (charts, `.db` files, downloaded media) and model weights are gitignored —
rerun the script that made them.

## Design principles

- **Scope discipline.** If it doesn't operate on `Embeddings`, it isn't in `grainz`.
- **Wrap, don't rebuild.** Modeling wraps scikit-learn; visualization wraps UMAP + Plotly. The value is the unified surface, not new internals.
- **Typed and tested.** Full type hints, `py.typed`, `mypy --strict` clean, property-based tests with `hypothesis`, and 184 tests covering every module.
- **Honest measurement.** Cache correctness and call accounting are first-class, tested invariants. Estimated figures are flagged as estimates, and every performance claim traces to a script in [`benchmarks/`](./benchmarks/) you can rerun.

## Contributing

Contributions welcome — see [`CONTRIBUTING.md`](./CONTRIBUTING.md) for setup, the three checks
a change has to pass (`pytest`, `ruff`, `mypy`), and a list of real open gaps worth taking on.

Maintaining a fork or release of your own? [`PUBLISHING.md`](./PUBLISHING.md) documents the
PyPI process end to end.

## Project status

Pre-release, and honest about it. The core loop — index, search, cache, track — works and is
tested. Before depending on this in production, read the **Known limitations** above; the
`hybrid_search` scaling ceiling is the one that matters most.

## License

`grainz` is released under the [MIT License](./LICENSE) — free to use, modify, and
distribute, including commercially.

### Dependencies

Every dependency is permissively licensed and commercially usable:

| License | Packages |
|---|---|
| BSD-3-Clause | numpy, scipy, scikit-learn, umap-learn, soundfile, torch |
| Apache-2.0 | qdrant-client, sentence-transformers, transformers |
| MIT | redis, plotly, kaleido, sqlite-vec, pillow (MIT-CMU) |

### ⚠️ Model weights are licensed separately

This is the part people miss. `grainz` is MIT, and the libraries above are permissive — but
the **model weights** downloaded by `registry.load()` are third-party artifacts under their
own terms, which `grainz` neither owns nor relicenses.

The `license` field in the model catalog is an **unverified convenience hint** for filtering,
hand-maintained in [`catalog.py`](./src/grainz/registry/catalog.py). It can drift as models
are relicensed. **Before using any model commercially, read its actual model card** (each
entry's `hf_id` links to the source).

The same applies to whatever data *you* index. `grainz` stores and searches your content; it
makes no claim about your right to use it.

## Disclaimer

Provided "as is", without warranty of any kind, as stated in the [LICENSE](./LICENSE). Neither
the author nor contributors are liable for any claim, damages, or other liability arising from
use of this software.

You are responsible for:

- the licensing and legal use of any **models** you load
- the licensing, privacy, and legal use of any **data** you index
- validating outputs before relying on them in production

Nothing here is legal advice. If you're deploying this commercially and licensing matters to
your situation, talk to someone qualified.
