Metadata-Version: 2.5
Name: distillql
Version: 0.1.12
Summary: One query engine over the warehouses you already run, with the context an AI agent needs to answer correctly: SQL as the real user, local joins in DuckDB, a glossary drafted from query history, and an explain record with lineage, policies and cost.
Project-URL: Homepage, https://distillql.com
Project-URL: Documentation, https://distillql.com/docs/
Project-URL: Changelog, https://distillql.com/docs/changelog
Author: DistillQL
License-Expression: Apache-2.0
License-File: LICENSE
License-File: NOTICE
Keywords: ai-agents,data-warehouse,duckdb,federated-query,mcp,postgres,semantic-layer,sql
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Typing :: Typed
Requires-Python: >=3.11
Requires-Dist: adbc-driver-manager>=1.0
Requires-Dist: adbc-driver-postgresql>=1.0
Requires-Dist: duckdb>=1.1
Requires-Dist: keyring>=24
Requires-Dist: lkml>=1.3
Requires-Dist: pyarrow>=17
Requires-Dist: pyyaml>=6
Requires-Dist: sqlglot>=25
Provides-Extra: bigquery
Requires-Dist: google-cloud-bigquery-storage>=2.25; extra == 'bigquery'
Requires-Dist: google-cloud-bigquery>=3.25; extra == 'bigquery'
Provides-Extra: databricks
Requires-Dist: databricks-sql-connector>=3.0; extra == 'databricks'
Provides-Extra: eval
Requires-Dist: httpx>=0.27; extra == 'eval'
Requires-Dist: mcp>=1.2; extra == 'eval'
Provides-Extra: mcp
Requires-Dist: mcp>=2.3; extra == 'mcp'
Provides-Extra: snowflake
Requires-Dist: snowflake-connector-python>=3.12; extra == 'snowflake'
Description-Content-Type: text/markdown

# DistillQL

One query engine over the warehouses a company already runs, with the context an AI agent needs to answer correctly. It runs SQL and Python as the real user on each warehouse, joins across them in a local DuckDB, drafts a glossary of the company's terms from its query history, and returns an explain record with lineage, policies and cost for every answer.

Most companies use their data team as a report factory: design metrics, build dashboards, answer tickets. Data is worth far more when the people with the questions can answer them safely themselves, and the data team edits the meaning instead of producing the reports. This platform is the engine that makes AI self-service correct, governed and cheap: DuckDB-like ease for engineers, an agent for everyone else, one planner under both.

Version 0.1.12 is an alpha. Docs: [distillql.com/docs](https://distillql.com/docs/).

## Install

```
pip install distillql            # the command line and Python API, for DuckDB and Postgres
pip install 'distillql[mcp]'     # adds the MCP server, so an AI app can call DistillQL's tools
```

Python 3.11 to 3.13. Extras for the warehouses: `bigquery`, and the experimental `snowflake` and `databricks`.

## Try it in five minutes

`distillql demo` writes a sample shop: a DuckDB file, a short query history and a config. Every command below runs on it with no server.

```
distillql demo                                   # writes ./distillql-demo
distillql crawl distillql-demo/distillql.yaml --out distillql-demo/context.yaml
distillql glossary distillql-demo/context.yaml   # the terms the history uses, drafted
distillql review distillql-demo/context.yaml     # what a person should confirm, most useful first
distillql sql distillql-demo/distillql.yaml "select status, count(*) from shop.main.orders group by status"
```

The crawl reads every table's profile and the query history. From those it drafts definitions: `revenue` is `sum(orders.amount) where status = 'paid'`, because 52 past runs computed it that way, and test accounts are left out of customer counts because the column comment and every query say so. A person accepts or corrects a draft with `distillql review --accept REF`; accepted definitions outrank drafted ones.

## Connect your database

```
distillql init
```

`init` asks for the connection, tests the login, and asks where to keep the password if one is needed: the system keychain, an environment variable, a command such as a secret manager or a cloud sign-in token, a prompt each time, `~/.pgpass`, or the config file itself. Then it writes `distillql.yaml`, crawls, and adds DistillQL to Claude Code if it is installed. Every question has a flag, for scripts and agents; `distillql init --help` lists them. DistillQL writes a password into a file only when you choose that.

TLS settings and client certificates go in the uri as for `psql` (`?sslmode=verify-full&sslrootcert=...`), and `ssh: ana@bastion` on an engine reaches a database behind a jump host with your own `ssh`.

From Python, hand over a connection you already have open (psycopg 3, psycopg2 or SQLAlchemy). DistillQL opens its own with the same login and keeps the password in memory:

```python
con = distillql.connect(engines=[distillql.Postgres.from_connection("app", my_psycopg_connection)])
```

A config can name more than one engine. SQL then names tables as `engine.schema.table`, or `schema.table` when only one engine has that table.

```yaml
engines:
  sales:    { kind: duckdb, path: sales.duckdb, history: queries.jsonl }
  postgres: { kind: postgres, service: analytics, history: "analytics/models/**/*.sql" }
identity:
  user: ana
  credentials:
    postgres: { role: analyst }          # SET LOCAL ROLE on every fragment, in a read-only transaction
```

**Query history.** The glossary comes from past queries. Postgres keeps them in `pg_stat_statements`: the extension must be loaded, and the login needs `pg_read_all_stats` to see other people's queries. That log stores `status = 'paid'` as `status = $1`, so a term built on it stays undefined until `history:` points at SQL that keeps its values, such as saved queries or dbt's compiled models. The log still supplies the counts. DuckDB keeps no log, so its `history` is the only source: a JSON-lines file of `{"sql": "...", "calls": 12, "user": "ana"}`, or `.sql` files.

**dbt.** If your team defines metrics in dbt, add `dbt: ~/analytics` to the engine that holds the models and crawl again. DistillQL reads the project's `target/` folder: each metric becomes a verified term, descriptions fill in missing comments, and tests mark keys, joins and allowed values. It needs no dbt install or account. `distillql gaps` then lists what the history computes that dbt does not define, and where the two disagree.

**Ossie and LookML.** `ossie: path` reads an Apache Ossie semantic model (dbt's `target/osi_document.json`, or a Snowflake semantic view's export), and `lookml: folder` reads a LookML project's files with no Looker instance. Their metrics and measures become verified terms beside dbt's, and a term two layers define the same way says they agree. `distillql glossary --ossie FILE` writes the glossary back out as Ossie.

| Engine | Status |
|---|---|
| DuckDB | Tested |
| Postgres | Tested, including row-level security and per-user roles |
| BigQuery | Tested on a live project |
| Snowflake, Databricks | Experimental: written to each warehouse's documented API but not yet run against a live account. They warn on first use. |

## Use it from Python

```python
import distillql

con = distillql.connect("distillql.yaml")
res = con.sql("""
  select o.customer_id, sum(o.amount) as spend, s.plan
  from sales.main.orders o
  join postgres.app.subscriptions s on s.customer_id = o.customer_id
  where s.plan <> 'free'
  group by 1, 3
""")
res.head()                                   # rows stay in DuckDB; you get a handle
res.aggregate({"total": "sum(spend)"}, by=["plan"])
print(res.explain().summary())               # each fragment, the identity each engine saw, pushdown, lineage, cost
```

## Ask with any LLM

`distillql ask` answers a question with a model on your machine (Ollama or LM Studio) or, if your config allows it, a cloud model. DistillQL runs the conversation itself, so no AI app is needed, and keeps personal columns and most rows back from the model.

```bash
distillql ask distillql.yaml "net revenue from paid orders last month" --model gpt-oss:20b
```

See [use any LLM](https://distillql.com/docs/any-llm) for the limits and the config.

## Connect an agent

`distillql mcp` serves six tools over MCP, on standard input and output:

- `glossary`: the company's terms on one page; with a question, the terms it names come first and in full
- `search_context`: the tables, metrics and reviewed queries that fit a question
- `query`: runs SQL as the user and returns a handle, a 3-row sample and notes, never every row; `profile` gives the explain record
- `profile`: describes, filters, aggregates or charts a handle without moving its rows
- `write`: saves a handle as a table in the local DuckDB session, or as a `.parquet` or `.csv` file in the config's `write_dir`; it never writes to a connected database
- `verify`: proposes an answer's SQL as a reviewed query; it stays a proposal until a person accepts it

DistillQL calls no model itself, so it needs no API key. Your AI app runs the tools and pays for its model under your own plan.

Claude Code:

```
claude mcp add distillql -- distillql mcp distillql.yaml --context .distillql/context.yaml
```

Claude Desktop, in `claude_desktop_config.json`:

```json
{"mcpServers": {"distillql": {"command": "distillql",
  "args": ["mcp", "/path/to/distillql.yaml", "--context", "/path/to/context.yaml"]}}}
```

Apps that connect by URL, such as ChatGPT (developer mode) and Claude on the web, need a public HTTPS address. Run the server over HTTP, and put a tunnel such as `cloudflared tunnel --url http://127.0.0.1:8000` in front of it:

```
distillql mcp distillql.yaml --context .distillql/context.yaml --http --public-url https://your-tunnel.example
```

Add the address with `/mcp` on the end as a connector in your app. When the app opens the approval page, the server prints a pairing code for that request beside the site the approval sends you back to. Check the site, then enter the code. Anyone with the address and a code can query as you, so keep codes to yourself. Approvals last until the server restarts.

## What the planner refuses

- **Anything but one SELECT.** Writes are refused before anything runs, including a DELETE inside a CTE.
- **A query over its limits.** `cap_rows` and `cap_bytes` refuse a part whose estimate is over the limit, before it runs.
- **A query with no credential for an engine.** An identity needs a login or a role for each engine it reads.
- **A filter that could change the rows.** Filters and key lists are pushed down only where an outer join cannot be affected.

## Reporting a problem

Use the form at [distillql.com/interest](https://distillql.com/interest) and say what you ran and what happened. That includes security problems: say so in the form, and leave out the details until we reply.

## License

Apache-2.0. See `LICENSE` and `NOTICE` in the source archive.
