Metadata-Version: 2.4
Name: dbx-cost-guardian
Version: 0.1.0
Summary: Shift-left FinOps for Databricks: cluster linting, cost estimation, and CI policy gates
License: MIT
Requires-Python: >=3.9
Requires-Dist: click>=8.0
Requires-Dist: pyyaml>=6.0
Provides-Extra: dev
Requires-Dist: mypy; extra == 'dev'
Requires-Dist: pytest-cov; extra == 'dev'
Requires-Dist: pytest>=7.0; extra == 'dev'
Requires-Dist: ruff; extra == 'dev'
Description-Content-Type: text/markdown

# dbx-cost-guardian

> Shift-left FinOps for Databricks: catch the expensive mistake in the PR, not the invoice.

[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)

---

`dbx-cost-guardian` is a Python package that embeds Databricks cost awareness directly into your development workflow. It statically analyses cluster configs, job definitions, and PySpark source code to catch cost anti-patterns — **no live Databricks connection required**.

## Features

- **Cluster Config Linting** — 6 rules (DBX001–DBX006) covering compute type, autoscaling, spot instances, runtime version, SQL workloads, and driver sizing
- **PySpark Code Analysis** — AST-based scanning for `.toPandas()`, `.collect()`, Python UDFs, and Pandas import anti-patterns
- **Cost Estimation** — DBU-based cost projection using configurable price tables (no API key needed)
- **Policy Gates** — Configurable pass/fail thresholds for CI/CD integration
- **CLI + Python API** — Use from the command line or embed in notebooks and test suites
- **JSON Reports** — Machine-readable output for dashboards and automation

## Installation

```bash
pip install dbx-cost-guardian
```

For development:

```bash
pip install dbx-cost-guardian[dev]
```

## Quick Start

### CLI

```bash
# Scan cluster configs and source files
dbx-cost-guardian scan cluster.json notebook.py

# Scan entire directories
dbx-cost-guardian scan configs/ src/ notebooks/

# With custom config and JSON output
dbx-cost-guardian scan --config guardian.yml --json-out report.json configs/

# List available lint rules
dbx-cost-guardian scan --list-rules

# Strict mode (fail on warnings too)
dbx-cost-guardian scan --strict configs/
```

### Python API

```python
from dbx_cost_guardian import scan

result = scan(
    cluster_jsons=["configs/production.json"],
    source_files=["src/etl_pipeline.py"],
)

# Check policy outcome
if not result.policy_result.passed:
    print("Cost policy violated!")
    for msg in result.policy_result.violated_thresholds:
        print(f"  • {msg}")

# Inspect violations
for v in result.all_violations():
    print(f"[{v.severity.value}] {v.rule_id}: {v.message}")

# Check cost estimates
for e in result.cost_estimates:
    print(f"{e.source_file}: ${e.estimated_run_usd:.2f}/run, ${e.estimated_month_usd:.2f}/month")
```

## Configuration

Create a `guardian.yml` at your repo root (see [`examples/guardian.yml`](examples/guardian.yml) for all options):

```yaml
# Override DBU pricing for your contract
dbu_prices:
  jobs: 0.10
  all_purpose: 0.40

# Assume 2-hour runtime for cost estimation
assumed_runtime_hours: 2.0

# Skip rules that don't apply
skip_rules:
  - DBX001

# Policy thresholds
policy:
  max_estimated_cost_usd_per_run: 100.0
  max_estimated_cost_usd_per_month: 10000.0
  fail_on_error: true
  max_violations:
    error: 0
    warning: 20
```

## CI/CD Integration

### GitHub Actions

```yaml
- name: Run cost scan
  run: |
    pip install dbx-cost-guardian
    dbx-cost-guardian scan \
      --config guardian.yml \
      --json-out cost-report.json \
      configs/ src/ notebooks/
```

See [`examples/github-actions.yml`](examples/github-actions.yml) for a complete workflow.

### Pre-commit Hook

```yaml
repos:
  - repo: local
    hooks:
      - id: dbx-cost-guardian
        name: Databricks Cost Guardian
        entry: dbx-cost-guardian scan
        language: system
        types_or: [json, python]
        pass_filenames: true
```

## Rules Reference

| Rule ID | Severity | Description |
|---------|----------|-------------|
| DBX001 | ERROR | All-Purpose Compute used for jobs (4× cost) |
| DBX002 | WARNING | Missing or unbounded autoscaling |
| DBX003 | WARNING | No spot/preemptible instances |
| DBX004 | WARNING | Outdated Databricks runtime |
| DBX005 | WARNING | SQL workload on Spark cluster |
| DBX006 | WARNING | Oversized driver node |
| AST001 | ERROR | `.toPandas()` — collects full DataFrame to driver |
| AST002 | WARNING | `.collect()` — brings all rows to driver |
| AST003 | INFO | `.toDF()` — Pandas to Spark conversion |
| AST010 | WARNING | Pandas import in Spark context |
| AST020 | WARNING | Plain Python UDF (row-by-row serialisation) |

Full details in [`docs/rules.md`](docs/rules.md).

## Exit Codes

| Code | Meaning |
|------|---------|
| 0 | All policy gates passed |
| 1 | Policy violation detected |

## Non-Goals

- Real-time cluster monitoring (use Databricks built-in or Datadog)
- Actual spend ingestion (no API calls in MVP)
- Auto-fixing configs (lint-and-report only)

## Roadmap

| Version | Feature |
|---------|---------|
| v0.2 | Databricks Asset Bundle (DAB) YAML support |
| v0.2 | Delta Live Tables cost rules |
| v0.3 | Historical runtime from Databricks Jobs API |
| v0.3 | GitHub PR comment reporter |
| v0.4 | VSCode extension |
| v1.0 | Plugin API for custom rules |

## License

MIT
