Metadata-Version: 2.4
Name: costlens
Version: 0.1.0
Summary: Estimate what a data pipeline will cost before you run it
Author: Yaswanth Gajulapalli
License-Expression: MIT
Project-URL: Homepage, https://github.com/Yaswanth216599/costlens
Project-URL: Source, https://github.com/Yaswanth216599/costlens
Project-URL: Issues, https://github.com/Yaswanth216599/costlens/issues
Keywords: spark,pyspark,fabric,databricks,finops,cost,static-analysis,data-engineering,delta-lake
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Quality Assurance
Classifier: Topic :: System :: Monitoring
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Provides-Extra: dev
Requires-Dist: pytest>=8.0; extra == "dev"
Dynamic: license-file

# costlens

**Estimate what a data pipeline will cost before you run it.**

Every existing cost tool — Fabric Capacity Metrics, CloudZero, Vantage — tells
you what you already spent. This reads the code and tells you what it is about
to spend, and which line is responsible.

```
  costlens · daily_sales_load.ipynb

  Estimated cost      $1,832
  Capacity            10,178.3 CU-hours
  Runtime             ~5.3 days   fabric-medium-x2+configure, 40 vCores billed
  Data moved          71.5 TB read · 250 TB shuffled · 12.0 TB written
  Confidence          low

  Where the money goes
    cell 3, line 8      $1,249   68%  ████████····  shuffle join ×9
    cell 5, line 6        $414   23%  ███·········  project ×9
    cell 4, line 1      $54.24    3%  █···········  cartesian join

  15 findings, up to $1,827 recoverable in total

  HIGH   CL003  cell 3, line 3   No partition pruning on sales.fact_order_lines
           sales.fact_order_lines is partitioned by order_date into 1,460
           partitions, but none of the filters here touch those columns.
           fix   Filter on the partition column before anything else:
                 `.filter(F.col("order_date") >= start_date)`.
           saves $1,583 / 8,794.9 CU-hours
```

The same pipeline after acting on every finding: **$0.35**, in two minutes.

## Read the ranking first, the dollar figure second

Be clear about what the number is. Uncalibrated, costlens is built to get the
**ordering and the order of magnitude** right — which line dominates the bill,
and by roughly how much relative to the others. That is the part you act on, and
it is the part that holds up.

The absolute dollar figure comes from rate constants that have not yet been
checked against *your* capacity. Treat it as a size, not a quote. Every run
prints a confidence grade and the reasons behind it, and an uncalibrated
estimate is capped at medium on purpose.

The findings themselves are a different matter — a DataFrame computed nine times
instead of cached, work inside a loop running once per iteration, a filter that
misses the partition column. Those are read off the code, and they are right
whether or not the dollar amount is.

**Two real runs fix the number.** After a pipeline runs for real, record what it
actually took, then fit the rates to it:

```bash
costlens compare my_notebook.ipynb --actual-minutes 47 --record runs.json
costlens calibrate --runs runs.json --out calibration.json
costlens check my_notebook.ipynb --calibration calibration.json
```

From then on the estimates are grounded in what your capacity does rather than
in the model's defaults, and the confidence cap lifts.

## Quickstart — Fabric

Three steps. There is nothing to connect: no sign-in, no connection string, no
permission to request.

**1. Install it.**

```bash
pip install costlens
```

Or install the **costlens** extension from the VS Code Marketplace and let it
install the engine for you.

**2. Get your table sizes out of Fabric.**

Open any Fabric notebook, paste in
[`examples/fabric/catalog_cell.py`](examples/fabric/catalog_cell.py), run it
once, and download the `catalog.json` it writes to your Lakehouse.

Forty lines of `SHOW TABLES`, `DESCRIBE DETAIL` and `json.dump` — no
dependencies, and no costlens install needed on the Fabric side. **Metadata
only: it does not read a single row of your data, and nothing leaves your
tenant.**

**3. Put the two together.**

Drop `catalog.json` beside your notebooks and run:

```bash
costlens check my_notebook.ipynb
```

The catalog is found automatically in the directory you run from. Check it into
the repo and pull requests get priced too. It only needs regenerating when your
tables change shape.

## Using it, in full

```bash
pip install costlens
```

Then open a notebook in VS Code, or:

```bash
costlens check notebooks/
```

That is the whole thing. With no setup it reads the code and reports what it can
prove from the code alone — a cross join, a DataFrame computed nine times, a
Python UDF on every row — and says plainly that it cannot price them yet:

```
  Estimated cost      unknown -- no table sizes yet

  4 issue(s) found in the code itself
    HIGH   CL004  line 16   `enriched` is computed 9 times
    HIGH   CL008  line 46   This runs 5 times -- it is inside a loop
    ...
```

For dollar figures it needs to know how big your tables are, which is the one
thing that cannot be read off the source. Paste
[`examples/fabric/catalog_cell.py`](examples/fabric/catalog_cell.py) into any
Fabric notebook, run it once, and download the `catalog.json` it writes into
your repo. Forty lines, no dependencies, metadata only — it does not read a row
of your data. After that every run is priced, and the file only needs
regenerating when your tables change shape.

## Try it

```bash
pip install -e ".[dev]"

costlens check examples/daily_sales_load.py --catalog examples/catalog.json
costlens check examples/daily_sales_load_fixed.py --catalog examples/catalog.json
costlens explain examples/daily_sales_load.ipynb --catalog examples/catalog.json
pytest -q
```

Zero runtime dependencies. The engine has to run in three places that are all
hostile to a dependency tree — a VS Code extension host, a CI container, and a
Fabric Spark session whose image you do not control — so everything is stdlib.

## How it works

Four stages, each of which can be inspected on its own.

| Stage | Module | What it does |
|---|---|---|
| Read | `ingest.py` | `.py`, `.ipynb`, or a string → one source plus a map back to each notebook cell. `%%sql` bodies and `%%configure` pool sizing are pulled out rather than discarded. |
| Analyse | `analyze.py`, `sqlish.py` | Abstract interpretation of the AST into a lineage DAG. Loops become a repeat factor, functions are analysed at the call site, and Spark's laziness is modelled rather than assumed away. |
| Size | `costmodel.py` | Filters and projections push *down* to the scans (this is where partition pruning is decided); rows and bytes propagate *up*; execution counts propagate *down* from each action. |
| Judge | `rules.py`, `report.py` | Patterns with a fix and a price, ordered by what they would actually save. |

Three details are worth knowing, because they are what a simpler tool gets wrong:

**Laziness is modelled, not ignored.** A DataFrame consumed by three actions and
never cached is computed three times. A linter that walks statements in order
cannot see this; it only appears if you keep the graph and count executions from
the actions backwards.

**Loops are priced.** Work inside `for region in regions:` happens once per
region. This is the single largest source of real surprise bills, and it is
invisible to per-statement analysis.

**Savings are attributed, not asserted.** Each finding's number comes out of the
cost model. Fixes that only remove one stage's own time (replacing a Python UDF,
stopping a spill) are scoped to that node; fixes that shrink everything
downstream (partition pruning) are scoped to the lineage. The headline total
de-overlaps them, so no set of findings can promise to save more than the job
costs.

## Where the table sizes come from

Static analysis gives you the *shape* of the work. Table size is the dominant
term, and it has to come from outside the file.

**On Fabric, use [`examples/fabric/catalog_cell.py`](examples/fabric/catalog_cell.py).**
Paste it into a notebook, run it once, download the file. It enumerates the
whole lakehouse for you and handles the cross-workspace catalog naming that
makes `DESCRIBE DETAIL` fail on a four-part name. That is the entire Fabric side
of this tool.

The rest of this section is the manual route — for other engines, or for filling
in one table by hand:

```bash
costlens catalog-template notebooks/ --out catalog.json
```

That writes a skeleton listing every table the code touches. Fill it in from
the lakehouse — in a Fabric notebook:

```python
spark.sql("DESCRIBE DETAIL sales.fact_orders").select("sizeInBytes", "numFiles").show()
```

A checked-in `catalog.json` is what makes the estimate reproducible in a pull
request where no Spark session exists. Tables with no entry are estimated from a
placeholder and named in the confidence line — never silently guessed.

## Running the engine inside Fabric (advanced)

Not needed for normal use — the flow above is a catalog export and then
everything else in the editor. This is for costing a notebook without leaving
Fabric at all.

The engine has no dependencies, and Python imports straight out of a zip on
`sys.path`, so the whole thing travels as one 53 KB file:

```bash
python scripts/bundle.py
#   dist/costlens.zip           one file to upload; import it off sys.path
#   dist/costlens_notebook.py   one CELL to paste; carries the engine inside it
```

Upload the zip to your lakehouse Files and a notebook can cost itself:

```python
import sys
sys.path.insert(0, "/lakehouse/default/Files/costlens.zip")
import costlens
```

If that fails with a bare `ModuleNotFoundError`, the cause is almost always that
no lakehouse is mounted at `/lakehouse/default` rather than anything to do with
the package — run [`examples/fabric/00_check_setup.py`](examples/fabric/00_check_setup.py)
to find out which. The pasteable cell has no such failure mode: it unpacks
itself to `/tmp` on the driver and needs no lakehouse, no upload and no network.

[`examples/fabric/`](examples/fabric/) has three cells to paste in:

| Cell | What it answers | Needs the engine? |
|---|---|---|
| `catalog_cell.py` | How big is every table in this lakehouse? | no |
| `live_cost.py` | What is this session costing *right now*? | no |
| `cost_this_code.py` | What would costlens predict for this code? | yes |

The first two are plain Python against Spark and work in any notebook as-is.
Running `live_cost.py` and `cost_this_code.py` on the same work is the honest
test: the measurement and the estimate, side by side.

## Checking it against reality

```bash
costlens compare notebooks/daily_load.ipynb --actual-minutes 47 --record runs.json
```

```
  Predicted         $6.77       37.6 CU-hours   47 min
  Actual            $8.10       45.0 CU-hours   56 min
  Error             -16%   close -- the model predicts under
```

`--record` appends the run to a shared file; `costlens compare --runs runs.json`
scores every run in it at once. That scoreboard, and the sentence it lets you
finish — *"on our capacity it predicts within N%"* — is the only claim this
project makes that matters. [`docs/TEAM.md`](docs/TEAM.md) lays out a two-week
pilot around producing it.

## What a run that already happened cost

```bash
costlens price --minutes 47 --profile fabric-medium
```

No source, no prediction — just the arithmetic Fabric does. Its reason for
existing is comparison: run this against a real notebook's duration and
`costlens check` against the same notebook's source, and the gap between them is
how wrong the model is on your capacity.

## Calibration

The rate constants ship as engine defaults. Feed the tool runs that already
happened and it fits them to your capacity:

```bash
costlens calibrate --runs runs.json --catalog catalog.json --out calibration.json
costlens check notebooks/ --catalog catalog.json --calibration calibration.json
```

`runs.json` is a list of `{"file": "...", "actual_seconds": 1830}` — or
`actual_cu_hours`, if that is what your monitoring export gives you.

Predicted runtime decomposes into four additive terms (fixed overhead, scan,
shuffle, write), so the fit is least squares in four unknowns. Below four
samples it falls back to a single global scalar rather than pretending to
separate terms it cannot see. **A fit that does not beat the defaults is
refused**, not written — overfitting four noisy runs is the fastest way to make
the tool worse than the constants it replaced.

## Gating a pull request

```bash
costlens check notebooks/ --catalog catalog.json --budget 25
echo $?   # 1 if the estimate exceeds the budget
```

`--fail-on high` blocks on severity instead. `--format md` produces a
pull-request comment; `--format json` is the contract the editor extension and
any other consumer read. See [`examples/ci/`](examples/ci/) for GitHub Actions
and Azure Pipelines recipes.

## The editor extension

[`extension/`](extension/) is a VS Code extension that runs the same engine and
shows the cost where the code is written: a status bar figure, the dollar amount
at the end of the expensive lines, a squiggle and a fix on each finding, and a
full report on demand. It owns none of the analysis — it shells out to
`costlens check --format json`, so the editor, CI and the command line can never
disagree about what a pipeline costs.

```bash
cd extension && npm install && npm run compile
```

Then open the folder in VS Code and press F5.

## What it does not do

Stated plainly, because a cost tool that oversells itself gets switched off:

- **It is an estimate.** Confidence is reported on every run, with the reasons.
  Uncalibrated, on tables with known statistics, expect the right order of
  magnitude and the right ranking — which is enough to decide what to fix.
- **Code in imported modules is not followed.** A notebook that hands its work
  to `from mypkg import pipeline; pipeline.run()` is priced on what is left in
  the file. Every such call is named in the confidence notes with its cell and
  line, so a suspiciously small number tells you why it is small rather than
  leaving you to trust it.
- **Joins are assumed to be foreign-key joins.** A fan-out join produces more
  rows than the model predicts.
- **SQL is read shallowly.** Table names, join conditions, WHERE clauses, GROUP
  BY — enough to build the same DAG, not enough to reason about semantics.
- **Data-dependent branches are priced as if taken.** A cost gate wants an upper
  bound.
- **Fabric Spark is the engine it knows best.** The model is unit-agnostic and
  `databricks-jobs` is included to prove it, but accuracy on any engine comes
  from calibration runs on that engine.

## Layout

```
src/costlens/
  ingest.py      source in, one blob plus a line map out
  analyze.py     PySpark AST -> lineage DAG
  sqlish.py      shallow Spark SQL reader
  plan.py        the DAG
  catalog.py     table statistics, with an explicit unknown path
  costmodel.py   pushdown, sizing, charging, pricing
  rules.py       patterns, fixes, attributed savings
  report.py      text / JSON / Markdown
  calibrate.py   fit engine rates to observed runs
  profiles.py    named pool shapes and their prices
  cli.py         check / explain / calibrate / catalog-template / profiles
extension/       VS Code extension over the JSON output
examples/        a broken pipeline, the same one fixed, and a catalog
tests/           123 tests
```
