Metadata-Version: 2.4
Name: bills-sql-lineage
Version: 0.0.6
Summary: Personal SQL lineage generator.
Author: billwallis
Maintainer: billwallis
License-Expression: GPL-3.0-or-later
Project-URL: Homepage, https://github.com/billwallis/bills-sql-lineage
Project-URL: Repository, https://github.com/billwallis/bills-sql-lineage
Project-URL: Issues, https://github.com/billwallis/bills-sql-lineage/issues
Project-URL: Documentation, https://github.com/billwallis/bills-sql-lineage/blob/main/README.md
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Requires-Python: >=3.13
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: arguably>=1.3.0
Requires-Dist: sqlglot[c]<28.0.0,>=27.0.0
Dynamic: license-file

<span align="center">

[![Python](https://img.shields.io/badge/Python-3.13+-blue.svg)](https://www.python.org/downloads/)
[![tests](https://github.com/billwallis/bills-sql-lineage/actions/workflows/tests.yaml/badge.svg)](https://github.com/billwallis/bills-sql-lineage/actions/workflows/tests.yaml)
[![coverage](https://raw.githubusercontent.com/billwallis/bills-sql-lineage/refs/heads/main/coverage.svg)](https://smarie.github.io/python-genbadge/)

[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/billwallis/bills-sql-lineage/main.svg)](https://results.pre-commit.ci/latest/github/billwallis/bills-sql-lineage/main)
[![GitHub last commit](https://img.shields.io/github/last-commit/billwallis/bills-sql-lineage)](https://shields.io/badges/git-hub-last-commit)
[![PyPI Downloads](https://img.shields.io/pypi/dm/bills-sql-lineage)](https://shields.io/badges/py-pi-downloads)

</span>

---

# Bill's SQL Lineage 🔀

Personal SQL lineage generator.

Built on the following awesome libraries:

- [SQLGlot](https://github.com/tobymao/sqlglot) for SQL parsing
- [arguably](https://github.com/treykeown/arguably) for the CLI

## So... what is this? 🤔

> [!NOTE]
>
> This project is still in development and currently only parses the CTEs of an SQL query since that's all I need for now.

I write _a lot_ of SQL, and I often need to understand "lineage" in lots of different ways.

Things like [dbt](https://www.getdbt.com/) and [SQLMesh](https://sqlmesh.com/) are great for object lineage (tables, columns, etc.), but I often need to understand "lineage" like the CTE lineage in a query, among other things.

This project is just a personal tool to help me generate lineage diagrams (in [Mermaid](https://mermaid.js.org/) syntax) for SQL queries to fit that requirement.

Upcoming improvements will (hopefully) include:

- Semantic edges (e.g. distinguish between `JOIN`, `WHERE`, `UNION`, etc.)
- Support for parameterised queries (e.g. with Jinja blocks, like [dbt](https://www.getdbt.com/))
- Parsing only the `SELECT` part of a SQL file that includes DDL/DML commands
- Lineage for multiple files in a single diagram
- Column lineage to Mermaid

## Installation ⬇️

Grab a copy from PyPI like usual:

```
pip install bills-sql-lineage
```

## Usage 📖

> [!WARNING]
>
> This is likely to change significantly as the project evolves.

Pass the path to a SQL file to the `lineage` command to generate the lineage as a [Mermaid](https://mermaid.js.org/) diagram:

```
lineage path/to/file.sql
```

This will write a Mermaid diagram to `path/to/file.mermaid`. You can control the target path with the `--target` argument:

```
lineage path/to/file.sql --target path/to/output.mermaid
```

By default, the SQL dialect will be inferred by SQLGlot, but you can specify a dialect with the `--dialect` argument:

```
lineage path/to/file.sql --dialect snowflake
```

## Example 📝

Given the following SQL query:

```sql
with

aaa as (select 1 as aa),
bbb as (select 2 as bb),
ccc as (select 3 as cc, aa from aaa),
ddd as (select 4 as dd, aa from aaa where aa not in (select bb from bbb))

select *
from ccc
    inner join ddd using (aa)
```

...the following Mermaid diagram will be generated:

```mermaid
graph TD
    aaa
    bbb
    ccc
    ddd
    final
    aaa --> ccc
    aaa --> ddd
    bbb --> ddd
    ccc --> final
    ddd --> final
```

Note that the `final` node is an alias for the final `SELECT` statement since the final `SELECT` statement is not a CTE.

## Contributing

Install the dependencies:

```shell
pip install --editable . --group dev --group test
pre-commit install --install-hooks
```
