Metadata-Version: 2.4
Name: dbt-prune
Version: 1.0.0
Summary: Find orphaned dbt models and snapshots across multiple manifests and open Copilot-powered cleanup tasks.
Author: Tarun Pass
Requires-Python: >=3.9
Requires-Dist: click<8.4,>=8.1
Requires-Dist: pydantic<3,>=2.8
Requires-Dist: pyyaml<7,>=6.0
Requires-Dist: requests<3,>=2.31
Requires-Dist: rich<15,>=13
Provides-Extra: datahub
Requires-Dist: acryl-datahub>=0.14.1; extra == 'datahub'
Provides-Extra: dev
Requires-Dist: mypy<2,>=1.11; extra == 'dev'
Requires-Dist: pytest<9,>=8.3; extra == 'dev'
Requires-Dist: ruff<1,>=0.6; extra == 'dev'
Requires-Dist: types-pyyaml<7,>=6.0.12.20250516; extra == 'dev'
Requires-Dist: types-requests<3,>=2.32.4.20250611; extra == 'dev'
Provides-Extra: gcs
Requires-Dist: google-cloud-storage>=2.0.0; extra == 'gcs'
Description-Content-Type: text/markdown

# dbt-prune

`dbt-prune` is a pip-installable dbt-core companion CLI for finding dbt models, snapshots, and sources that are no longer referenced by anything in your current project, configured sibling projects, or dbt exposures.

## Installation

```bash
pip install dbt-prune
```

Optional integrations stay lightweight by default:

```bash
pip install 'dbt-prune[gcs]'
pip install 'dbt-prune[datahub]'
```

## Checking the installed version

```bash
dbt-prune --version
```

## Configuration

By default `dbt-prune` looks for `dbt_prune.config.yml` in the dbt project root passed to `--project-dir`.

```yaml
manifests:
  - name: other_project
    type: local
    config:
      path: ../other_project/target/manifest.json

datahub:
  enabled: false
  gms_server_env_var: DATAHUB_GMS_SERVER
  token_env_var: DATAHUB_TOKEN
  platform: bigquery
  env: PROD

whitelist:
  - my_model
  - important_snapshot
```

### GCS manifest

Install the GCS extra and declare a `gcs` manifest entry:

```bash
pip install 'dbt-prune[gcs]'
```

```yaml
manifests:
  - name: analytics
    type: gcs
    config:
      project_id: data-mdm-prod
      bucket_name: tron-dbt-artifacts-prod
      object_name: orchestration/analytics/manifest.json
    excluded_packages:
      - customers
      - customer_platform
```

A `gs://` URI is accepted as shorthand, and `project_id` is optional (it then
falls back to your default credentials' project):

```yaml
manifests:
  - name: analytics
    type: gcs
    config:
      uri: gs://tron-dbt-artifacts-prod/orchestration/analytics/manifest.json
```

Unknown or misspelled config keys raise an error rather than being silently
ignored.

Optional GCS config fields:

| Field | Description |
|---|---|
| `credentials` | Path to a service-account JSON key file (uses ADC when omitted) |
| `impersonate_service_account` | Service account email to impersonate |

### Package filtering

Each manifest entry supports `excluded_packages` and `included_packages` (mutually exclusive). Nodes whose `package_name` is in `excluded_packages` are dropped from the manifest before orphan evaluation; `included_packages` keeps only matching nodes.

### Whitelisting models

Use the top-level `whitelist` to keep specific current-project models or snapshots from ever being reported as orphaned. Whitelisted assets never appear in `dbt-prune ls` output and are skipped by `dbt-prune run`:

```yaml
whitelist:
  - my_model
  - important_snapshot
```

Entries are matched against the node **name**. When you inspect a whitelisted asset directly with `-s/--select`, `dbt-prune` reports it as not orphaned with the reason `Whitelisted in dbt_prune.config.yml`.

### Optional manifests

Set `optional: true` on a manifest entry to allow it to fail without aborting the run:

```yaml
manifests:
  - name: staging
    type: gcs
    optional: true
    config:
      project_id: my-project
      bucket_name: my-bucket
      object_name: staging/manifest.json
```

The config file stores the **names** of the environment variables only. The actual DataHub GMS server URL and token must come from the environment at runtime:

```bash
export DATAHUB_GMS_SERVER=https://datahub.example.com
export DATAHUB_TOKEN=...
```

## How orphan detection works

`dbt-prune` only parses `manifest.json` files. It does **not** run `dbt`, compile models, or introspect a live warehouse.

It loads:
- the current project's `target/manifest.json`
- any configured external manifests from local paths or `gs://` URIs
- exposure dependencies from every loaded manifest

A current-project model, snapshot, or source is considered orphaned only when it has zero incoming references from any loaded manifest and is not referenced by any exposure. An unused source is one that no model in any loaded manifest selects via `source()`. If DataHub checking is enabled, downstream lineage consumers also keep a node off the orphan list.

## Usage

## Output

`dbt-prune` uses rich terminal output when stdout is a TTY, including colours,
status spinners, and concise success/warning/error lines. Piped or redirected
output is plain text with no ANSI escapes or emoji, and `--format json` remains
machine-readable. Use group-level flags before the command: `-v/--verbose` for
per-item detail, `-q/--quiet` for errors only, and `--no-color` (or `NO_COLOR=1`)
to force plain output.

### List all orphaned assets

```bash
dbt-prune ls --project-dir .
```

By default `ls` reports orphaned models, snapshots, and unused sources.

### Filter by resource type

Restrict the listing to a single resource type with `--resource-type` (`model`, `snapshot`, or `source`):

```bash
dbt-prune ls --resource-type source
```

### Inspect one specific model

```bash
dbt-prune ls --project-dir . -s my_model
```

When `-s/--select` is used, `dbt-prune` reports whether the named model or snapshot is orphaned and explains why.

### Filter by package

Scope orphan detection to a specific dbt package within your project using `-p`/`--package`:

```bash
dbt-prune ls -p marketing_core
```

Combine with `-s`/`--select` to inspect a specific model within a package:

```bash
dbt-prune ls -p marketing_core -s some_model_name
```

The same flag works for `run`:

```bash
dbt-prune run -p marketing_core --dry-run
```

### JSON output

```bash
dbt-prune ls --format json
```

### Start a Copilot agent session to delete every orphan

```bash
dbt-prune run --project-dir .
```

Behavior:
- scans all current-project models and snapshots by default
- builds a single prompt listing every orphaned asset
- starts one agent session via `gh agent-task create "PROMPT"`
- the prompt instructs the agent to delete the model files and every
  `properties.yml` / `schema.yml` reference to them, then open a pull request
- requires the GitHub CLI (`gh`) to be installed and authenticated

### Dry-run the cleanup plan

```bash
dbt-prune run --dry-run
```

### Restrict cleanup to one model

```bash
dbt-prune run -s my_model
```

### Limit the number of assets in the prompt

```bash
dbt-prune run --limit 5
```

## DataHub integration

When `datahub.enabled: true`, `dbt-prune` queries DataHub for downstream lineage before marking an otherwise unreferenced asset as orphaned.

Dataset URNs are built from each manifest node's materialized location:
`database.schema.alias` (falling back to `name` when alias is omitted). Ensure the manifest's `database` / `schema` / `alias` values match how your warehouse assets are ingested into DataHub.

Use `datahub.platform` and `datahub.env` to match your DataHub dataset URN convention:

```yaml
datahub:
  enabled: true
  gms_server_env_var: DATAHUB_GMS_SERVER
  token_env_var: DATAHUB_TOKEN
  platform: bigquery
  env: PROD
  extra_platforms: [dbt]   # also probed when the warehouse URN has no lineage
```

### Building URNs from a production manifest

If your local `manifest.json` is compiled against a dev target, its
`database` / `schema` / `alias` describe your dev tables, and DataHub will
correctly report that those have no downstreams. Point `location_manifest` at
another configured manifest (for example the prod artifact loaded from GCS) to
build URNs from production locations instead:

```yaml
manifests:
  - name: prod
    type: gcs
    config:
      project_id: data-mdm-prod
      bucket_name: tron-dbt-artifacts-prod
      object_name: orchestration/revshare/manifest.json

datahub:
  enabled: true
  platform: bigquery
  env: PROD
  location_manifest: prod
```

Nodes are matched by `unique_id`, falling back to name. Anything not found in
the location manifest falls back to the current-project location.

Lineage is resolved with a single-hop `DownstreamOf` relationships query. One hop
is all that is needed to know whether anything consumes the asset, and it avoids
DataHub's `maxRelations` limit that a full multi-hop lineage walk hits on highly
connected datasets. Deployments without that field fall back to
`searchAcrossLineage` with `maxHops: 1`. Because dbt-managed assets are often ingested under both the warehouse
platform and the `dbt` platform (as siblings), each URN in `platform` +
`extra_platforms` is probed and the asset is kept if *any* of them reports
downstreams. Set `extra_platforms: []` to probe only the warehouse platform.

GraphQL errors returned by DataHub are surfaced as warnings (or raised with
`--strict`) rather than being treated as "no downstreams".

If the required environment variables are missing or DataHub is unreachable, `dbt-prune` warns and continues unless you pass `--strict`.

## GitHub Copilot coding-agent PR automation

`dbt-prune run` builds a single cleanup prompt covering every orphaned asset in `dbt_prune.pr_agent` and starts a Copilot coding-agent session with `gh agent-task create`. The module also still exposes a REST-based issue creator (`GitHubCopilotPRAgent`) for teams that prefer issue-driven workflows.

## Development

```bash
python -m pip install -e '.[dev]'
ruff check .
pytest
mypy dbt_prune
```
