Metadata-Version: 2.4
Name: ico-cache
Version: 1.0.3
Summary: Semantic caching middleware for LLM applications: 3-tier (exact/semantic/context-aware) cache that cuts API token cost and latency by not re-running repeated prompts through the model
Author: Dev
Project-URL: Homepage, https://github.com/DEV-S-SHAH/ICO
Project-URL: Repository, https://github.com/DEV-S-SHAH/ICO
Project-URL: Documentation, https://github.com/DEV-S-SHAH/ICO/blob/main/docs/ARCHITECTURE.md
Classifier: Operating System :: OS Independent
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: Microsoft :: Windows
Classifier: Operating System :: MacOS
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: fastapi<1.0.0,>=0.110.0
Requires-Dist: uvicorn<1.0.0,>=0.28.0
Requires-Dist: qdrant-client<2.0.0,>=1.8.0
Requires-Dist: redis<6.0.0,>=5.0.0
Requires-Dist: fastembed<1.0.0,>=0.2.0
Requires-Dist: pydantic<3.0.0,>=2.6.0
Requires-Dist: pydantic-settings<3.0.0,>=2.2.0
Requires-Dist: pypdf>=3.0.0
Requires-Dist: beautifulsoup4<5.0.0,>=4.12.0
Requires-Dist: lancedb<1.0.0,>=0.5.0
Requires-Dist: structlog<26.0.0,>=24.1.0
Requires-Dist: litellm>=1.40.0
Requires-Dist: prometheus-client<1.0.0,>=0.20.0
Requires-Dist: opentelemetry-api>=1.20.0
Requires-Dist: opentelemetry-sdk>=1.20.0
Provides-Extra: loaders
Requires-Dist: pymupdf>=1.24.0; extra == "loaders"
Requires-Dist: tree-sitter>=0.23.0; extra == "loaders"
Requires-Dist: tree-sitter-python>=0.23.0; extra == "loaders"
Requires-Dist: tree-sitter-javascript>=0.23.0; extra == "loaders"
Requires-Dist: tree-sitter-go>=0.23.0; extra == "loaders"
Requires-Dist: pdf2image>=1.16.0; extra == "loaders"
Requires-Dist: pytesseract>=0.3.10; extra == "loaders"
Requires-Dist: Pillow>=10.0.0; extra == "loaders"
Requires-Dist: odfpy>=1.4.1; extra == "loaders"
Requires-Dist: python-docx>=1.1.0; extra == "loaders"
Requires-Dist: openpyxl>=3.1.0; extra == "loaders"
Requires-Dist: python-pptx>=0.6.23; extra == "loaders"
Provides-Extra: observability
Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.20.0; extra == "observability"
Requires-Dist: langfuse>=2.0.0; extra == "observability"
Provides-Extra: audit
Requires-Dist: bandit>=1.7.0; extra == "audit"
Requires-Dist: pip-audit>=2.6.0; extra == "audit"
Provides-Extra: dev
Requires-Dist: ico-cache[audit,loaders,observability]; extra == "dev"
Requires-Dist: pytest>=8.0.0; extra == "dev"
Requires-Dist: pytest-asyncio>=1.0.0; extra == "dev"
Requires-Dist: ruff>=0.4.0; extra == "dev"
Requires-Dist: mypy>=1.8.0; extra == "dev"
Dynamic: license-file

# ico-cache

Semantic caching middleware for **LLM applications**. `ico-cache` sits between any
coding agent, RAG pipeline, or LLM-based system and the model itself, caching model
responses so repeated or reformulated prompts are never re-run through the LLM.

It uses three complementary caching techniques:

| Layer | Technique | Purpose |
| --- | --- | --- |
| **L1** | Exact | Instant hits for identical repeat queries (Redis / SQLite). |
| **L2** | Semantic | Matches paraphrased and reworded queries (Qdrant / LanceDB). |
| **L3** | Context-aware | Resolves multi-turn, context-dependent queries with dual vectors. |

Key benefits, in order: **lower API token cost** (fewer calls to the LLM), **lower
latency** (a cache hit returns in milliseconds instead of seconds), and minimal
cache read/write overhead itself. A hard metadata gate (+ a 0% false-hit baseline)
prevents near-miss cross-entity / cross-topic false positives.

A reference implementation that showcases ico-cache end-to-end (against SEC filings,
FastAPI + Streamlit) lives in `apps/financial-rag-demo` of the [repository] — it is
a demo, not the product.

## Install

Requires Python 3.11+.

```bash
pip install ico-cache
pip install "ico-cache[loaders,observability]"   # document loaders + tracing
```

## Quickstart (Zero-Infra Embedded Mode)

No external services — LanceDB (vectors), SQLite (exact), FastEmbed (local ONNX embeddings).

```python
import asyncio
from ico_cache import CacheEngine
from ico_cache.backends.vector.lancedb_store import LanceDBStore
from ico_cache.backends.exact.sqlite_store import SQLiteStore
from ico_cache.backends.embedding.fastembed_embedder import FastEmbedder

async def main():
    engine = CacheEngine(
        embedder=FastEmbedder(),
        vector_store=LanceDBStore(uri="./lancedb"),
        exact_store=SQLiteStore(db_path="cache.db"),
        metadata_filter_keys=["project", "topic"],
        adaptive_threshold=True,
    )

    query = "What does the fetch_user(id) function return?"
    result = await engine.resolve(query)
    if result["source"] == "MISS":
        answer = {"text": "It returns the user record matching id, or None when not found."}
        engine.set_l1(query, answer)
        await engine.async_write_l2(query, answer)
    else:
        print(f"Cache HIT via {result['source']}: {result['response']}")

asyncio.run(main())
```

See the [main repository README](https://github.com/DEV-S-SHAH/ICO) for universal
document ingestion, distributed server mode, Kubernetes deployment, testing, and
benchmarking. The JavaScript SDK is published as `ico-cache-js`.

[repository]: https://github.com/DEV-S-SHAH/ICO
