Metadata-Version: 2.4
Name: coltrace
Version: 0.1.0.dev10
Summary: Static Pandas column lineage, impact analysis, and local dependency graphs
Author: Cristiano Diniz
License-Expression: MIT
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: networkx<4,>=3.2
Requires-Dist: typer<1,>=0.12
Provides-Extra: dev
Requires-Dist: pytest>=8; extra == "dev"
Requires-Dist: ruff>=0.9; extra == "dev"
Requires-Dist: mypy>=1.14; extra == "dev"
Requires-Dist: build>=1.2; extra == "dev"
Requires-Dist: types-networkx>=3.2; extra == "dev"
Requires-Dist: sqlglot<31,>=30.24; extra == "dev"
Provides-Extra: sql
Requires-Dist: sqlglot<31,>=30.24; extra == "sql"
Provides-Extra: capture
Requires-Dist: pandas<4,>=2; extra == "capture"
Provides-Extra: viz
Requires-Dist: pyvis<0.4,>=0.3.2; extra == "viz"
Dynamic: license-file

# ColTrace

Static Pandas column lineage and potential impact analysis for Python projects.
ColTrace reads source files, identifies supported transformations, and connects
column versions across direct function calls. Static scanning never imports or
executes the project or reads its input data. Optional read capture observes
column labels while you explicitly execute your own pipeline.

**Development preview: `0.1.0.dev10`.** The supported subset is deliberately small.
Unsupported operations produce diagnostics and may interrupt lineage. A graph
with no edge is not proof that a dependency does not exist.

## Install locally

Requires Python 3.11+. From this checkout:

```sh
python -m venv .venv
# Activate .venv using your shell, then:
python -m pip install -e ".[dev,viz,sql,capture]"
coltrace --help
```

On Windows without activation, use `.\.venv\Scripts\python.exe -m coltrace`.
The `viz` extra supplies PyVis for HTML export; basic analysis and JSON need only
NetworkX and Typer. `sql` adds SQLGlot for bounded SQL parsing; `capture` adds
Pandas for optional schema observation. Pandas is not a core dependency.
Publication to PyPI is a
separate release step; these commands install the local project.

## Start at the data source

The source checkpoint recognizes supported SQL-to-Pandas and NoSQL-to-Pandas
extractions. Literal SQL projections, aliases and expressions describe source
fields before their first Python use. Every database column the SQL reads is a
selectable lineage node (`origin="table"`), so a column used only inside
`SUM(...)` or a `WHERE` clause can be traced to the DataFrame columns it feeds.
Query filters are control metadata rather than value dependencies of every
output. `spark.sql(build_query(table)).toPandas()` keeps its SQL text even with
an injected, unresolved session; SQL assembled from f-strings, `.format`, `%`
and joins keeps its known text, with unknown parts shown as `{expression}`.
`SELECT *`, `UNION`, tables read by name (`spark.table`, Snowpark
`session.table`, `dbt.ref`) and SQL files read inside the project are modeled
too; see [extraction architectures](docs/extraction-architectures.md).

`report.readers` inventories resolved `pd.read_*` calls with file, line, context,
return shape and resulting DataFrame identity. File schemas are unknown unless
declared or observed. To enumerate actual columns, including those never
explicitly referenced by the script:

```python
from coltrace import analyze_project, capture_reads

with capture_reads(project_root) as capture:
    run_pipeline()  # You explicitly execute your existing pipeline.

report = analyze_project(project_root, source_schemas=capture.schemas)
```

Capture retains column names and source locations, not rows or credentials.
Use `capture_dataframe_schema` for an already-loaded Pandas object, including a
SQL-to-Pandas conversion. Observations must attach to the corresponding analyzed
`.py` extraction site; arbitrary notebook-cell lineage remains unsupported.
Unknown SQL, wildcard schemas, dynamic calls and unsupported transformations
still produce gaps. Schema observation does not resolve hidden value lineage.

See [database source usage](docs/database-sources.md), the
[researched extraction matrix](docs/database-patterns.md), and
[read capture](docs/read-capture.md). Try the
[static SQL example](examples/database_pipeline/README.md) and
[executable capture example](examples/read_capture/README.md).
The [source checkpoint](docs/source-checkpoint.md) records validation and limits.

## Notebook projects and common Python structure

Databricks notebook source (`# Databricks notebook source`, `# COMMAND ----------`,
`# MAGIC`) and raw magics such as `%pip` are parsed with the magics neutralized
in place. `%run ./child` includes the child notebook in the caller's namespace,
statically. Nothing is executed. `.ipynb` files and live cells are not inputs.

Branches, `try` blocks, loops and early returns are joined: a column that may
hold different values after alternative paths gets an inferred merge version
reading each possibility, so lineage continues without being claimed as
confirmed. Loops over configured label collections (`for c in COLS`) are
unrolled exactly. Classes, nested functions and closures, decorators,
`*args/**kwargs`, `+=`, unpacking, comprehensions and ternaries are interpreted
rather than erasing state.

A documented catalog of Pandas/NumPy semantics covers common Series methods and
accessors, `np.where`/`pd.to_numeric`-style functions, row-selecting DataFrame
methods (`sort_values`, `head`, `reset_index`, `query`, …), whole-frame value
methods (`fillna`, `astype`, …), partial `.loc[mask, col]` updates and
`merge`/`concat` (inferred). When data enters through an injected helper such as
`run_query(sql)`, ColTrace follows the closure when possible; otherwise, a result
used as a DataFrame becomes an inferred frame whose SQL SELECT list seeds its
columns, and the scan suggests a source contract to confirm it.

See the [compatibility checkpoint](docs/compatibility-checkpoint.md), the
[Databricks-style example](examples/databricks_project/README.md) and the
[multi-module example](examples/robustness_pipeline/README.md).

Notebook output separates discovered files from modeled frames/columns, shows
configuration and versions, prints errors first, separates informational notes
(inferences) from warnings, and saves full diagnostics outside the analyzed tree
before attempting column selection. Listed static labels are not runtime
`df.columns`. Capture remains opt-in.

## Try the complete example

```sh
coltrace scan examples/pricing_pipeline
coltrace trace preco_referencia
coltrace impact preco_referencia
coltrace graph preco_referencia --open
coltrace export preco_referencia --format json --output .coltrace/pricing.json
```

The example connects extraction, pricing, and feature functions through explicit
calls in `pipeline.py`. No `prices.csv` file is needed to analyze it.

```text
valor_produto -> preco_bruto -> preco_referencia -> margem
                                      |              |
                                      +--------------+-> margem_percentual
custo ---------------------------------------------> margem
```

`impact preco_referencia` identifies `margem` and `margem_percentual` as direct
potential dependents. There is also a longer path through `margem`. A changed
input does not necessarily change every downstream value.

`scan` writes `.coltrace/scan.json` by default. `--out DIRECTORY` selects another
snapshot directory. Queries load that snapshot explicitly; they do not silently
rescan the project. Rescan after editing source, or use `--project DIRECTORY` to
analyze current files in memory. `--from FILE` selects an existing snapshot.

```sh
coltrace trace margem --project examples/pricing_pipeline --upstream --depth 2
coltrace impact preco_referencia --from .coltrace/scan.json --evidence confirmed
coltrace graph --from .coltrace/scan.json --output .coltrace/all.html
```

HTML files contain their visualization resources and work locally without a
server or CDN. A selected column opens its journey: direct reads, applications,
writes, derived columns and destinations. Frame passage is labeled separately
from a direct column read. Callback control and configuration appear in the
expandable creation history, rather than as direct uses of the selected column.
The graph uses generously spaced, numbered blocks in a vertical flow. The
scrollable sidebar contains the trace history, source code, and locations;
selecting a step highlights its matching graph block, and vice versa. Open Details
or Diagnostics without losing the current selection. The Outcomes tab shows
derived-column destinations and the DataFrames carrying the selected column.
Zoom in/out, use **Fit all**
for an overview, or **Reset view** to return to a readable scale.
`--view columns` shows the column dependency projection; `--operations` includes
operation nodes in that projection. A graph without a selected column shows the
whole project's column dependencies.

```sh
coltrace scan examples/text_pipeline
coltrace trace comment_text --function process_train
coltrace graph comment_text --function process_train --open
```

`trace` defaults to direct column actions, derived-column outcomes, and containing
DataFrames. `--full-history` includes creation ancestry and callback internals,
with control/configuration distinguished from value transformations.
`--diagnostics` also displays dependency lists and relevant diagnostics. Rescan after an
upgrade: schema 0.2 snapshots have no recorded history. See the
[column history guide](docs/column-history.md) and [current checkpoint](docs/history-checkpoint.md).
The [graph workspace checkpoint](docs/graph-workspace-checkpoint.md) describes the
current layout and [additional column checks](docs/other-column-checkpoint.md).
See the [column journey checkpoint](docs/column-journey-checkpoint.md) for the
usage-first view and its acceptance cases.

## Identity and ambiguity

Different DataFrames keep different identities, even when both contain a column
named `score`. Assignments create column versions. `frame["a"] = frame["a"] + 1`
connects the old and new versions; `frame["a"] = 0` cuts the old value's lineage.
History still records that overwrite without adding a false value dependency.

Selection by name uses the latest observed version within each DataFrame. If
several unrelated identities remain, the CLI lists candidates and exits with status 3.
Same-name copies connected by proven pass-through relations form one selectable
family, whose original observed column is selected. A rename, value derivation,
or shared label alone does not establish that equivalence.
Select one explicitly using `--column-id ID` or `--dataframe ID`. Use `--version`
for an older version, and `--module` or `--function` for exact source filters.
IDs are opaque, deterministic within equivalent scans, and may change when source
locations or call contexts change. They are not permanent database keys.

`--depth` counts column-to-column hops. `--relation derive|rename|pass` filters
relations, while `--evidence confirmed|all` filters their evidence. Filtering out
intermediate relations can disconnect a path. Direct and transitive lists use
the shortest eligible distance, so a node reached both ways appears as direct.

## Supported patterns

| Pattern | Support |
| --- | --- |
| Explicit Pandas imports and aliases | Recognized origins; names such as `df` alone are insufficient |
| Resolved `pd.read_*` calls | Located inventory; supported DataFrame readers plus explicit/observed source schemas; iterators/collections remain distinct |
| SQL-to-Pandas extraction | Bounded SQL projections, aliases, expressions and helpers; documented readers/conversions or explicit contracts; database columns as nodes; dialect fallback |
| MongoDB / DynamoDB extraction | Literal projections and bounded Mongo expressions; candidate fields stay partial without observed schema |
| Literal dictionary `DataFrame` construction | String column keys and literal values |
| `frame["column"]`, `frame.loc[:, "column"]` | Literal or statically known labels (`for c in COLS`, f-strings) |
| `frame["c"] = frame["a"] + frame["b"]` | Arithmetic, masks, ternaries, scalar constants, temporary values, versions |
| `alias = frame` | Same abstract object and shared mutation |
| `copy(deep=True)`, `head(n)`, `shape` | Bounded independent copies, row previews and immutable metadata observations |
| Conditional/exception/context/loop blocks, early returns | Separate inferred paths joined with inferred merge versions |
| Databricks notebook source, `%pip`, `%run` | Magics neutralized; `%run` included statically in the caller namespace |
| Series/DataFrame/NumPy catalog | Documented derivations, row selections, value methods, exports; see the subset |
| `merge`, `join`, `concat` | Inferred frame whose columns may come from any input |
| Classes, closures, decorators, unpacking | Interpreted; decorators assumed transparent (inferred) |
| Injected `run_query(sql)` helpers | Followed through closures, or inferred from DataFrame use with SQL columns |
| `rename(columns={"old": "new"})` | Literal, non-colliding map; new DataFrame, or shared mutation with `inplace=True` |
| `assign(b=lambda x: x["a"] * 2)` | Sequential simple lambdas; eager arguments use their original scope |
| `Series.apply(clean_text)` | Resolved scalar callbacks; bounded summaries and inferred evidence |
| `frame[["a", "b"]] = DataFrame(results.tolist(), index=frame.index)` | Fixed callback output width and matching source index |
| `dropna`, literal projection, boolean row filters | Preserve value lineage; predicate consumption tracked separately |
| `to_parquet`, `to_csv`, `to_excel`, `to_dict`, … | Located export/serialization consumption |
| Direct synchronous function calls | Explicit arguments/returns, bounded acyclic contexts and explicit imports |
| Annotated, uncalled functions | Separate inferred templates, not evidence of pipeline connections |

See [the supported subset and boundaries](docs/supported-patterns.md) for the exact
contract. Calls, branches, dynamic columns, and Pandas operations outside that
contract produce located diagnostics rather than invented connections.

The [text pipeline example](examples/text_pipeline/README.md) demonstrates
independent train/evaluation flows, multi-output text cleaning, row filters, and
exports. See the [NLP checkpoint](docs/nlp-checkpoint.md) for what was validated
against a real project and which downstream third-party operations remain opaque.

## Evidence and completeness

- `confirmed`: supported syntax and receiver identity establish the relation.
- `inferred`: the relation depends on an explicit assumption, such as an uncalled
  function's DataFrame annotation or an unknown external schema during rename.
- `unresolved`: insufficient evidence across an unsupported boundary.

`complete`, `partial`, and `unresolved` describe how much of an operation is
understood; evidence and completeness are separate. External column reads are
marked as external/unknown origins. They do not identify the real upstream data
source's column definition or guarantee its presence.

Use `--diagnostics` to inspect warnings and errors. Query output also includes
relevant gaps. An inconclusive trace prints `Not determined` for empty dependency
lists and shows the recognized DataFrame origin separately from column evidence.
See [troubleshooting incomplete traces](docs/troubleshooting.md).
The analyzer is not a Python type checker or an execution
validator. Its grammar is limited by the interpreter running ColTrace. Pandas
imports are assumed to refer to the real library; environment/module impersonation
is not resolved.

## Python API

```python
from coltrace import analyze_project
from coltrace.graph import LineageGraph

report = analyze_project("examples/pricing_pipeline")
graph = LineageGraph(report)
column = graph.select("preco_referencia")
impact = graph.impact(column.id)
print(report.to_json())
```

The earlier `scan_project` API remains available for raw occurrences and receiver
evidence. Its discovery report (`0.1`) is a different format from the lineage
snapshot (`0.5`). See [discovery JSON](docs/discovery-json.md) and
[lineage JSON](docs/lineage-json.md). Public facts are dataclasses; their serialized
form is independent of NetworkX and contains no AST or executable objects.

## Configuration

Only the scanned root's `pyproject.toml` is read:

```toml
[tool.coltrace]
exclude = ["generated", "legacy/*.py", "*_test.py"]
```

Glob patterns match case-sensitive, project-relative POSIX paths. Patterns with
no `/` also match individual path components. Built-in exclusions cover `.venv`,
`venv`, `.git`, `__pycache__`, `.coltrace`, `.ipynb_checkpoints`, `node_modules`,
tox/nox and pytest/mypy/Ruff caches, `build`, and `dist`. Linked files/directories
and Windows reparse points are skipped. A `.coltraceignore` file at the analyzed
root uses gitignore syntax (root-level subset, including `!` negation). The root
`.gitignore` is also applied unless `respect_gitignore = false` is set.
`sql_dialect = "bigquery"` names the dialect of SQL text whose reader does not;
text that does not parse is still retried in common dialects and reported.
`observers = ["notify_team", "client.*"]` declares project helpers that only read
the DataFrames they receive, so they do not interrupt lineage.
`ScanConfig(exclude=("generated",))` overrides TOML settings and extends built-ins.

## Development

```sh
python -m pytest -q
python -m ruff check src tests examples
python -m ruff format --check src tests examples
python -m mypy
python -m build
```

Tests include independent expected dependencies, forbidden edges, alias/version
semantics, cross-module calls, CLI, malformed snapshots, offline HTML, and a
sentinel proving that analyzed source is not executed. See the
[accuracy protocol](docs/accuracy.md), [current checkpoint](docs/nlp-checkpoint.md),
[decision log](docs/decisions.md), and [backlog](docs/backlog.md).

The GitHub Actions workflow configures Windows/Linux with Python 3.11–3.13. It
must run in the repository before those combinations can be claimed as verified.

Licensed under the [MIT License](LICENSE). Copyright 2026 Cristiano Diniz.
