Metadata-Version: 2.4
Name: sqlens
Version: 0.6.0
Summary: Schema intelligence layer for LLMs — enriched database context, not SQL generation.
Project-URL: Homepage, https://github.com/pabloandr/sqlens
Project-URL: Documentation, https://github.com/pabloandr/sqlens#readme
Project-URL: Issues, https://github.com/pabloandr/sqlens/issues
Author: Pablo Andrés
License-Expression: MIT
License-File: LICENSE
Keywords: bigquery,context,database,llm,rag,schema,sql
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Database
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.10
Provides-Extra: all
Requires-Dist: chromadb>=0.4; extra == 'all'
Requires-Dist: google-cloud-bigquery>=3.0; extra == 'all'
Requires-Dist: numpy>=1.24; extra == 'all'
Requires-Dist: psycopg2-binary>=2.9; extra == 'all'
Requires-Dist: sentence-transformers>=2.0; extra == 'all'
Provides-Extra: bigquery
Requires-Dist: google-cloud-bigquery>=3.0; extra == 'bigquery'
Provides-Extra: dev
Requires-Dist: mypy>=1.0; extra == 'dev'
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff>=0.4; extra == 'dev'
Provides-Extra: numpy
Requires-Dist: numpy>=1.24; extra == 'numpy'
Provides-Extra: postgresql
Requires-Dist: psycopg2-binary>=2.9; extra == 'postgresql'
Provides-Extra: vector
Requires-Dist: chromadb>=0.4; extra == 'vector'
Requires-Dist: sentence-transformers>=2.0; extra == 'vector'
Description-Content-Type: text/markdown

# sqlens

**Schema intelligence layer for LLMs.** Enriched database context, not SQL generation.

sqlens introspects your database schema, enriches it with descriptions, statistics, inferred relationships, sample data, and business domain tags, then retrieves only the relevant tables for any natural language query — formatted and optimized for an LLM's context window.

## Why?

Most NL-to-SQL tools fail not because of the LLM, but because of the garbage context they feed it. A raw DDL dump tells a model nothing about what `usr_acct_bal_dt` means, which tables relate to each other implicitly, or what values `country_code` actually contains.

sqlens solves the **context problem** and stays out of the **generation problem**. It's the R+A in RAG — retrieval and augmentation — without the G. You bring your own LLM.

## Install

```bash
pip install sqlens                   # core (keyword retrieval only)
pip install sqlens[bigquery]         # + BigQuery connector
pip install sqlens[postgresql]       # + PostgreSQL connector
pip install sqlens[numpy]            # + cosine similarity retrieval
pip install sqlens[vector]           # + chromadb vector search
pip install sqlens[all]              # everything
```

## Quick start

```python
from sqlens import SQLens

# Connect and introspect — BigQuery
ctx = SQLens.from_bigquery(project="my-project", dataset="analytics")

# Or PostgreSQL
ctx = SQLens.from_postgresql("postgresql://user:pass@localhost:5432/mydb")

# Enrich the schema (one-time, ~2-3 min for 80 tables)
ctx.enrich(
    descriptions=True,    # rule-based column/table descriptions
    stats=True,           # cardinality, null%, min/max, top values
    relations=True,       # infer implicit foreign keys
    samples=3,            # 3 representative rows per table
    domains=True,         # auto-tag tables by business domain
)

# Save to disk (don't re-enrich every time)
ctx.save("./catalog.json")

# Later: load and retrieve context for a query
ctx = SQLens.load("./catalog.json")

context = ctx.get_context(
    "monthly active users by country",
    max_tables=5,
    level="standard",
    domain="auto",        # auto-detect relevant domain, pre-filter
)

# Use with any LLM
print(context.to_prompt())   # LLM-optimized text
print(context.to_dict())     # structured dict/JSON
```

## How it works

```
Database → Introspect → Enrich → Catalog (JSON)
                                      ↓
                              Domain Filter (optional)
                                      ↓
                              Retrieval (keyword/cosine/vector)
                                      ↓
                              Context Output → .to_dict() / .to_prompt()
```

**Enrichment** adds metadata the LLM needs: human-readable descriptions (via heuristics or optional LLM), column statistics, inferred relationships, sample data, and business domain tags.

**Retrieval** finds the relevant tables for a query. Three tiers auto-detected by what's installed: keyword/TF-IDF (zero deps), numpy cosine similarity, or chromadb vector search.

**Domain-scoped retrieval** pre-filters tables by business domain before search. On an 83-table schema, this reduces the search space from 83 to ~12 tables, dramatically improving precision.

## Domain-scoped retrieval

```python
# Explicit domain
ctx.get_context("revenue Q1", domain="sales")

# Auto-detect from query (supports English + Spanish)
ctx.get_context("ventas en Ecuador", domain="auto")

# Manual domain tagging
ctx.set_domain("orders", ["sales", "finance"])
ctx.set_domain("campaign_clicks", ["marketing"])
```

## CLI

sqlens ships with a command-line tool for scripting and CI pipelines.

```bash
# Introspect a database and save the catalog
sqlens inspect --bigquery my-project.analytics -o catalog.json
sqlens inspect --postgresql "postgresql://user:pass@host/db" -o catalog.json

# Enrich the catalog (one or more enrichers)
sqlens enrich catalog.json --descriptions --stats --relations --samples 3 --domains

# Retrieve context for a natural language query
sqlens context catalog.json "monthly active users by country" --max-tables 5
sqlens context catalog.json "revenue by region" --domain auto --level full
sqlens context catalog.json "orders last week" --json   # output as JSON
```

Add `--verbose` (or `-v`) to any command for debug output.

## Cosine retrieval

With `numpy` installed, sqlens automatically upgrades from keyword matching to cosine similarity search. With `sentence-transformers` also installed, it uses a real semantic embedding model:

```bash
pip install sqlens[numpy]                        # cosine with hash embeddings
pip install sqlens numpy sentence-transformers   # cosine with semantic model
```

The embedding model is loaded once per `SQLens` instance and cached — repeated `get_context()` calls pay zero reload cost. The retriever auto-detects what's available at runtime with no code changes required.

## LLM descriptions

By default, descriptions use rule-based heuristics (`usr_acct_bal_dt` → "user account balance date"). For higher quality, pass any LLM as a callable:

```python
# Anthropic
import anthropic
client = anthropic.Anthropic()

ctx.enrich(
    descriptions=lambda prompt: client.messages.create(
        model="claude-sonnet-4-20250514",
        max_tokens=200,
        messages=[{"role": "user", "content": prompt}],
    ).content[0].text,
)

# Google Gemini
import google.generativeai as genai
genai.configure(api_key="YOUR_KEY")
model = genai.GenerativeModel("gemini-2.0-flash")

ctx.enrich(
    descriptions=lambda prompt: model.generate_content(prompt).text,
)
```

## Primary key inference

When a database doesn't expose primary key constraints (e.g., BigQuery), sqlens infers them automatically using three heuristics, in order:

1. A column named exactly `id` → primary key
2. A column named `{singular_table_name}_id` (e.g., `user_id` in a `users` table) → primary key
3. The first NOT NULL column with an `_id` suffix → primary key

Inferred keys are marked `pk_source: "inferred"` in the catalog metadata so you can distinguish them from database-declared constraints. This feeds the relationship inferrer, which needs at least one PK per table to work.

## Incremental updates

sqlens fingerprints each table's structure. When you re-introspect, only tables that changed get re-enriched:

```python
ctx = SQLens.load("./catalog.json")
ctx.set_connector(BigQueryConnector(...))
ctx.refresh()  # only re-enriches changed tables
ctx.save("./catalog.json")
```

## License

MIT
