Metadata-Version: 2.4
Name: dbt-ci
Version: 1.9.4
Summary: CI/CD tool for dbt projects with intelligent change detection and selective execution
Author-email: Patrick Tannoury <patrick.tannoury@datablock.dev>
License: MIT License
        
        Copyright (c) 2026 DataBlock
        
        Permission is hereby granted, free of charge, to any person obtaining a copy
        of this software and associated documentation files (the "Software"), to deal
        in the Software without restriction, including without limitation the rights
        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
        copies of the Software, and to permit persons to whom the Software is
        furnished to do so, subject to the following conditions:
        
        The above copyright notice and this permission notice shall be included in all
        copies or substantial portions of the Software.
        
        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
        SOFTWARE.
        
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Topic :: Software Development :: Build Tools
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: click<9,>=8.1
Requires-Dist: pyyaml<7,>=6.0
Requires-Dist: dbt-core<2.0.0,>=1.10.13
Requires-Dist: slack-sdk<4,>=3.27
Provides-Extra: gcp
Requires-Dist: google-cloud-bigquery<4,>=3.20; extra == "gcp"
Requires-Dist: google-cloud-storage<4,>=2.14; extra == "gcp"
Provides-Extra: aws
Requires-Dist: boto3<2,>=1.34; extra == "aws"
Provides-Extra: docker
Requires-Dist: docker<8,>=7.0; extra == "docker"
Provides-Extra: all
Requires-Dist: dbt-ci[gcp]; extra == "all"
Requires-Dist: dbt-ci[aws]; extra == "all"
Requires-Dist: dbt-ci[docker]; extra == "all"
Provides-Extra: postgres
Requires-Dist: psycopg2-binary>=2.9.0; extra == "postgres"
Provides-Extra: mysql
Requires-Dist: pymysql>=1.0.0; extra == "mysql"
Provides-Extra: dev
Requires-Dist: dbt-ci[all]; extra == "dev"
Requires-Dist: pytest==9.1.1; extra == "dev"
Requires-Dist: black==26.10.1; extra == "dev"
Requires-Dist: mypy==2.4.0; extra == "dev"
Requires-Dist: flake8==7.4.1; extra == "dev"
Requires-Dist: duckdb==1.5.6; extra == "dev"
Requires-Dist: pytest-cov==7.1.0; extra == "dev"
Requires-Dist: pytest-mock==3.16.0; extra == "dev"
Dynamic: license-file

# dbt-ci

[![Tests](https://github.com/datablock-dev/dbt-ci/actions/workflows/tests.yml/badge.svg)](https://github.com/datablock-dev/dbt-ci/actions/workflows/tests.yml)
[![dbt-core](https://img.shields.io/badge/dbt--core-1.10%20%7C%201.11%20%7C%201.12-orange?logo=dbt&logoColor=white)](https://github.com/datablock-dev/dbt-ci/actions/workflows/tests.yml)

A CI tool for dbt (data build tool) projects that intelligently runs only modified models based on state comparison, supporting multiple execution environments including local, Docker, and dbt runners.

## How It Works

dbt-ci uses a **cache-based workflow**:

1. **`init`** - Downloads reference state from cloud storage (or uses local), compares with current code, and creates a cache of changes
2. **`run/delete/ephemeral`** - Use the cached state automatically (no need to re-specify state paths)

This design ensures:
- ✅ **Consistent state** across all commands in a CI run
- ✅ **Better performance** (no redundant state downloads)
- ✅ **Simpler CLI** (specify state once in init, reuse everywhere)

## Installation

### From PyPI (Recommended)

```bash
pip install dbt-ci
```

#### Extras

The base install deliberately carries only dbt-core and the CLI plumbing. The cloud SDKs
and the Docker client are large and most projects need at most one of them, so they are
installed on demand:

| Extra | Installs | Needed for |
|-------|----------|------------|
| `gcp` | `google-cloud-bigquery`, `google-cloud-storage` | `gs://` state/artifact URIs, the BigQuery connector used by `delete` and `migration` |
| `aws` | `boto3` | `s3://` state/artifact URIs |
| `docker` | `docker` | `--runner docker` |
| `all` | all of the above | |

```bash
pip install 'dbt-ci[gcp]'            # BigQuery + GCS
pip install 'dbt-ci[aws,docker]'     # S3 state, Docker runner
pip install 'dbt-ci[all]'            # everything
```

If a feature needs an extra you haven't installed, dbt-ci says which one and how to
install it rather than failing with an import traceback.

#### Supported dbt versions

| dbt-core | Status |
|----------|--------|
| 1.10.x | Supported, tested in CI |
| 1.11.x | Supported, tested in CI |
| 1.12.x | Supported, tested in CI |

dbt-ci needs Python 3.12 or newer. CI tests Python 3.12 and 3.13 on every dbt-core version
above, and Python 3.14 on dbt-core 1.12, the first release that supports it.

`pip install dbt-ci` resolves the newest dbt-core 1.x (`>=1.10.13,<2.0.0`). To stay on an
older minor version, pin it alongside dbt-ci, e.g. `pip install dbt-ci 'dbt-core~=1.11.0'`.

The published Docker images are tagged per dbt-core version (`dbt-1.10.23`, `dbt-1.11.15`,
`dbt-1.12.5`); `latest` tracks the newest supported minor version.

### From GitHub

```bash
# Install from main branch
pip install git+https://github.com/datablock-dev/dbt-ci.git@main

# Install a specific version
pip install git+https://github.com/datablock-dev/dbt-ci.git@v1.0.0
```

### Local Development

```bash
git clone https://github.com/datablock-dev/dbt-ci.git
cd dbt-ci
pip install -e ".[dev]"
```

Or with pipenv:

```bash
pipenv install --dev
```

`pyproject.toml` is the single source of truth for dependencies — the `Pipfile` installs
the project itself rather than restating them, so the two cannot drift apart. Add or
change a dependency in `pyproject.toml`, then run `pipenv lock`.

After installation, the tool is available as `dbt-ci`.

## Quick Start

**The Workflow:** Initialize once with `init`, then run commands that use the cached state.

### 1. Initialize State

First, initialize the dbt-ci state. This downloads/reads reference state and creates a cache:

```bash
dbt-ci init \
  --dbt-project-dir dbt \
  --profiles-dir dbt \
  --reference-target production \
  --state dbt/.dbtstate
```

**With Cloud Storage (GCS/S3):**
```bash
dbt-ci init \
  --dbt-project-dir dbt \
  --state-uri gs://my-bucket/dbt-state/manifest.json \
  --reference-target production \
  --state dbt/.dbtstate
```

### 2. Run Modified Models

After initialization, run commands use the cached state automatically:

```bash
# No need to specify --state again!
dbt-ci run \
  --dbt-project-dir dbt \
  --profiles-dir dbt
```

**With Docker:**
```bash
dbt-ci run \
  --runner docker \
  --docker-image ghcr.io/dbt-labs/dbt-bigquery:latest
```

## Commands

All commands share a set of **common options** (listed in the [Common Options](#common-options) section below). Command-specific flags are listed under each command.

---

### `init` - Initialize State

Creates initial state from your dbt project. **Always run this first.** Downloads reference manifest from cloud storage (if specified) and creates a local cache for subsequent commands.

```bash
dbt-ci init \
  --dbt-project-dir dbt \
  --profiles-dir dbt \
  --state-uri gs://my-bucket/manifest.json \
  --reference-target production \
  --state dbt/.dbtstate
```

**Flags:**

| Flag | Aliases | Env Var(s) | Default | Description |
|------|---------|-----------|---------|-------------|
| `--reference-target` | `--ref-target` | `DBT_REFERENCE_TARGET` | `None` | dbt target for the production/reference manifest |
| `--reference-path` | | `DBT_REFERENCE_PATH` | `reference` | **Deprecated — no effect.** The reference compile writes to dbt's own target path |
| `--reference-vars` | `--ref-vars` | `DBT_REFERENCE_VARS` | `None` | Variables to pass to dbt when compiling the reference manifest (YAML string or file path) |
| `--state-uri` | | `DBT_STATE_URI`, `STATE_URI` | `None` | Remote URI for the state manifest (e.g. `gs://bucket/manifest.json`, `s3://bucket/manifest.json`) |
| `--target-compile` | | `DBT_TARGET_COMPILE` | `false` | Run the second compile pass against the actual target |
| `--skip-reference-compile` | | `DBT_SKIP_REFERENCE_COMPILE` | `false` | Skip the compile pass against the reference/production state |
| `--comparison-strategy` | `--comparison` | `DBT_COMPARISON_STRATEGY` | `hybrid` | Strategy for detecting changed nodes: `dbt`, `git`, or `hybrid` |
| `--base-ref` | | `DBT_CI_BASE_REF` | Auto-detected | Base branch to diff against (e.g. `main`). Auto-detected from `GITHUB_BASE_REF` or git if not set |

> All [common options](#common-options) also apply.

**Git comparison and clone depth:** the `git` and `hybrid` strategies diff `origin/<base-ref>...HEAD` — the merge base — so only the commits on your branch count as changes. Shallow clones often don't contain the merge base; dbt-ci falls back to a direct diff and logs a debug message when that happens. For an accurate change set, fetch full history (`actions/checkout` with `fetch-depth: 0`).

How git's file-level changes are mapped to nodes:

- **Moved or renamed files** are reported by git as a delete of the old path plus an add of the new one. dbt identifies nodes by `unique_id`, which a move does not change, so a node that still exists in the target manifest is treated as **modified**, not deleted and re-added. Only nodes that are really gone are reported as deleted (and dropped by `delete`).
- **Macro changes** select every node that calls the changed macro, directly or through another macro.
- **YAML-only changes** (config, contracts, columns in a `schema.yml`) select the model the YAML configures. With `git`, every model configured by a changed YAML file is selected; with `hybrid`, only those that dbt also flags as modified.
- **`--dbt-project-dir`** is matched as a directory relative to the repository root, so `dbt`, `./dbt`, `dbt/` and an absolute path behave the same, and files in look-alike folders such as `analytics_dbt/` are ignored.

---

### `run` - Run Modified Models

Detects and runs models that have changed. Uses cached state from `init`, and fails if `init` has not run.

```bash
dbt-ci run --dbt-project-dir dbt --mode models
```

**Flags:**

| Flag | Aliases | Env Var(s) | Default | Description |
|------|---------|-----------|---------|-------------|
| `--mode` | `-m`, `--nodes`, `-n` | `DBT_NODES` | `all` | What to run: `all`, `models`, `seeds`, `snapshots`, `tests` |
| `--downstream-depth` | | `DBT_DOWNSTREAM_DEPTH` | Full graph | How many levels of downstream dependencies to include (dbt's `model+N`) |
| `--filters` | `-f` | | `None` | Extra resource-type filter (repeatable, choices: `models`, `seeds`, `snapshots`, `tests`). E.g. `--mode tests -f snapshots` to run only tests that have a snapshot dependency |

With `--mode all`, the steps run in dependency order: **seeds → snapshots → models → tests**, so models built on a changed snapshot read its new state and tests run once everything they check exists. The test step runs both data tests and dbt unit tests (dbt ≥ 1.8).

#### Limiting blast radius

By default a change selects its **entire** downstream graph. On a large project a change
to a core staging model therefore rebuilds almost everything, which is the opposite of
what change-based CI is for. `--downstream-depth` caps how far a change propagates:

```bash
dbt-ci run --downstream-depth 0   # only what changed
dbt-ci run --downstream-depth 1   # changed models and their direct children
dbt-ci run --downstream-depth 2   # two levels out
dbt-ci run                        # unlimited (default)
```

For a chain `customers → l1 → l2 → l3` where only `customers` changed:

| Flag | Models run |
|------|------------|
| `--downstream-depth 0` | `customers` |
| `--downstream-depth 1` | `customers`, `l1` |
| `--downstream-depth 2` | `customers`, `l1`, `l2` |
| *(omitted)* | `customers`, `l1`, `l2`, `l3` |

New and deleted nodes are always included regardless of depth — a new model has to run
whether or not anything depends on it yet. Every seed, snapshot and model the run builds
also gets its data and unit tests, even though a test sits one level below the node it
checks.

Deleted nodes no longer exist in the target manifest, so `run` does not look for their
downstream nodes there. A node that depended on a deleted one has changed itself (its `ref`
or test arguments differ) and is picked up as modified or new. This also covers generic
tests whose arguments changed: dbt gives them a new `unique_id`, so they show up as one
deleted and one new test.

> All [common options](#common-options) also apply.

**Examples:**
```bash
# Run only modified models
dbt-ci run --mode models

# Run modified models with defer to production
dbt-ci run --mode models --defer

# Run all modified resources (models, tests, seeds, etc.)
dbt-ci run --mode all

# With Docker
dbt-ci run --runner docker --mode models
```

---

### `ephemeral` - Ephemeral Environment

Clones changed models and their downstream dependencies into an isolated target schema using **`dbt clone`**, allowing integration testing without affecting production. Uses cached state from `init`.

> **Important:** `--target` and `--vars` must match the environment you want to clone into. The clone operation reads your `profiles.yml` to determine the target database/schema — if these are wrong, models will be cloned to the wrong location or the command will fail.

```bash
dbt-ci ephemeral \
  --target my-pr-env \
  --vars '{"use_production_data":"false"}'
```

**How it works:**
1. Reads the cached change set from `init`
2. Builds a selection of all affected models and their downstream dependencies
3. Runs `dbt clone --select <nodes>` targeting the specified environment
4. The cloned tables/views can then be used as the base for subsequent `dbt run` commands in the PR environment
5. Records the datasets it cloned into, so `finalize --clean-ephemeral` can drop them later

If `dbt clone` fails, `ephemeral` exits with a non-zero code on every runner.

**Flags:**

| Flag | Aliases | Env Var(s) | Default | Description |
|------|---------|-----------|---------|-------------|
| `--keep-env` | | `DBT_KEEP_ENV` | `false` | **Deprecated — no effect.** `ephemeral` never destroys the environment it creates; use [`finalize --clean-ephemeral`](#finalize---finalize-state) to tear it down |

> All [common options](#common-options) also apply.

---

### `delete` - Delete Removed Models

Detects and deletes models that have been removed from the project. Uses cached state from `init`.

```bash
dbt-ci delete --dry-run  # preview what will be deleted
dbt-ci delete            # execute deletions
```

The tables dropped are the ones in the **reference** manifest (usually production), so run the real delete only after the removal is merged, and use `--dry-run` everywhere else.

A node is only dropped if it is really gone from the current project. Nodes that still exist in the target manifest (for example a model whose file was moved or renamed, which git reports as delete + add) are skipped, as are relations that another target node still builds (for example through an `alias`). `delete` fails if `init` has not run, instead of passing without doing anything. On BigQuery the connection comes from the `profile` named in `dbt_project.yml` and the `--target` output in `profiles.yml` (its `project` and `location`).

**Flags:**

> Only [common options](#common-options) apply — no command-specific flags.

---

### `migration` - Migrate Partitioning Changes

Detects models whose partitioning configuration changed between the reference and target
state, and rebuilds the affected tables with the new partitioning spec. Uses cached state
from `init`.

BigQuery cannot change a table's partitioning in place, so each affected table is copied
into a temporary table with the new spec (keeping the model's `cluster_by`,
`require_partition_filter` and `partition_expiration_days`). The original is then renamed
to `<table>__dbt_ci_backup`, the copy is renamed into place, and only then is the backup
dropped — if the swap fails, the original data is still in the backup table. The table
migrated is the relation dbt resolves for the model, including its `alias`. Only **incremental** models are considered — other materializations are
rebuilt by dbt anyway.

```bash
dbt-ci migration --dry-run  # list the tables that would be rebuilt
dbt-ci migration            # apply the changes
```

> **Warning:** this rewrites tables in place. Always review the `--dry-run` output first.
> Requires a connector implementing a migration strategy — currently BigQuery only.

**Flags:**

> Only [common options](#common-options) apply — no command-specific flags.

---

### `report` - Summarise the Run

Renders the change set detected by `init` and the status of every command that has run
so far. In GitHub Actions the report is appended to the job summary automatically, so no
workflow wiring is needed beyond calling it. Each command's entry ends as `completed` or
`failed` however it exits, including dry runs and runs with nothing to do.

```bash
dbt-ci report                       # → $GITHUB_STEP_SUMMARY, or stdout locally
dbt-ci report --output report.md    # → a file
dbt-ci report --format json         # → machine-readable
dbt-ci report --pr-comment          # → also posted as a comment on the pull request
```

The report covers:

- **Change counts and node names**, grouped as modified / new / deleted. Long lists fold
  into a collapsible block.
- **Exposure impact** — the exposures downstream of the change set, including those
  downstream of *deleted* nodes, which is usually the case worth catching.
- **Command status and duration** for each of `init`, `run`, `delete`, `ephemeral` and
  `migration` that has run.

**Flags:**

| Flag | Aliases | Env Var(s) | Default | Description |
|------|---------|-----------|---------|-------------|
| `--output` | `-o` | `DBT_REPORT_OUTPUT` | `$GITHUB_STEP_SUMMARY` or stdout | Where to write the report |
| `--format` | `-F` | `DBT_REPORT_FORMAT` | `markdown` | `markdown` or `json` |
| `--pr-comment` / `--no-pr-comment` | | `DBT_NOTIFICATIONS_PR_COMMENT` | `false` | Also post the report as a comment on the GitHub pull request (`notifications.pr-comment` in the config file) |

> All [common options](#common-options) also apply.

#### Posting the report as a PR comment

With `--pr-comment` (or `notifications.pr-comment: true`), `report` also posts the
markdown report as a comment on the pull request. dbt-ci owns a single comment per pull
request and edits it on every later run instead of adding new ones. The comment is always
markdown, even with `--format json`.

The step needs a token that can write pull request comments:

```yaml
permissions:
  pull-requests: write

steps:
  - run: dbt-ci report --pr-comment
    env:
      GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
```

The pull request is detected from the GitHub Actions event, and GitHub Enterprise works
through `GITHUB_API_URL`. Posting the comment never fails the job: on a run that is not
for a pull request, without a token, or when GitHub rejects the request (pull requests
from forks get a read-only token), `report` logs why and carries on. Reports longer than
GitHub's 65,536-character comment limit are cut, with a note pointing to the job summary.

---

### `config` - Generate a Config File

Writes a commented `dbt-ci.config.yaml` skeleton with the common options pre-filled, so
you don't have to memorise every flag. See [Configuration File](#configuration-file).

```bash
dbt-ci config                          # create ./dbt-ci.config.yaml
dbt-ci config --output dbt/dbt-ci.config.yaml
dbt-ci config --force                  # overwrite an existing file
```

**Flags:**

| Flag | Aliases | Default | Description |
|------|---------|---------|-------------|
| `--output` | `-o` | `dbt-ci.config.yaml` | Destination path for the generated file |
| `--force` | `-f` | `false` | Overwrite the file if it already exists |

> This command does not take the [common options](#common-options).

---

### `finalize` - Finalize State

Run after `run`, `delete`, or `ephemeral` to upload artifacts and clean up the local cache for the next CI run.

```bash
dbt-ci finalize
dbt-ci finalize --artifacts-uri s3://my-bucket/dbt-artifacts/
```

**Flags:**

| Flag | Aliases | Env Var(s) | Default | Description |
|------|---------|-----------|---------|-------------|
| `--artifacts-uri` | | `DBT_ARTIFACTS_URI`, `ARTIFACTS_URI` | `None` | Object storage URI for uploading run artifacts such as the updated `manifest.json` (e.g. `s3://bucket/dbt-artifacts/`) |
| `--files` | | `DBT_FINALIZE_FILES` | `manifest` | Which artifacts to upload (repeatable): `manifest`, `cache`, `log` |
| `--clean-ephemeral` | `--destroy-ephemeral` | `DBT_CLEAN_EPHEMERAL`, `DBT_DESTROY_EPHEMERAL` | `false` | Drop the datasets that `ephemeral` cloned into (BigQuery). Datasets used by any node in the reference (production) manifest are never dropped, and a failed cleanup fails the step |

Uploaded artifacts land at `<artifacts-uri>/manifest.json`, `<artifacts-uri>/cache.json`
and `<artifacts-uri>/logs.txt` respectively (a trailing `/` on the URI is ignored).
With `--dry-run`, nothing is uploaded, the cache is kept, and `--clean-ephemeral` only
lists the datasets it would drop. `log` uploads the run log that dbt-ci writes
to `<cache dir>/dbt-ci.log`.

`--clean-ephemeral` runs before the upload, so a missing manifest or a failed upload
doesn't leave the CI datasets behind. If the cleanup itself fails, the artifacts are
still uploaded, the step fails, and the cache is kept so a re-run can retry the cleanup.

> **Note:** the log file always records at `DEBUG` level, including the resolved
> configuration for each command. Values whose names look sensitive (webhook, token,
> password, secret, credential, api_key) are masked before being written, so uploading
> the log does not publish your Slack webhook or credentials passed via `--docker-env`.
> This is a name-based heuristic — review your artifact bucket's access controls before
> enabling `--files log`.

> All [common options](#common-options) also apply.

## Runners

dbt-ci supports multiple execution environments:

### Local Runner

Execute dbt commands directly on your machine:

```bash
# After init
dbt-ci run \
  --runner local \
  --dbt-project-dir dbt
```

### dbt Runner (Python API)

Uses dbt's Python API (fastest, default):

```bash
# After init - uses dbt Python API
dbt-ci run \
  --runner dbt \
  --dbt-project-dir dbt
```

### Docker Runner

Run dbt commands inside a Docker container. Requires `pip install 'dbt-ci[docker]'`:

```bash
dbt-ci run \
  --runner docker \
  --docker-image ghcr.io/dbt-labs/dbt-duckdb:latest \
  --docker-volumes "$(pwd)/dbt:/dbt:rw"
```

When using the Docker runner, `--project-dir`, `--profiles-dir`, and `--state` are derived from the volume map — if the host path is inside a mounted volume (compared by path, so a `dbt` mount does not cover `dbt_project`), the corresponding container path is passed to dbt inside the container; with nested mounts the most specific one wins. `DBT_PROJECT_DIR`, `DBT_PROFILES_DIR` or `DBT_STATE` in `--docker-env` take priority. If no matching volume is found, `--project-dir` and `--profiles-dir` are omitted and dbt falls back to its own defaults (typically the container's `WORKDIR`), while `--state` is passed through unchanged so an absolute container path can be given directly.

**For Apple Silicon Macs:**

```bash
dbt-ci run \
  --runner docker \
  --docker-platform linux/amd64 \
  --docker-image ghcr.io/dbt-labs/dbt-postgres:latest \
  --docker-volumes "$(pwd)/dbt:/dbt:rw"
```

#### Docker Advanced Options

**Platform (for Apple Silicon compatibility):**
```bash
--docker-platform linux/amd64  # or linux/arm64
```

**Custom Volumes:**
```bash
--docker-volumes "/host/path:/container/path" --docker-volumes "/another:/path:ro"
```

**Environment Variables:**
```bash
--docker-env "DBT_ENV=prod" --docker-env "MY_API_KEY=secret"
```

**Network Mode:**
```bash
--docker-network bridge  # or host, none, container:name
```

**User:**
```bash
--docker-user "1000:1000"  # or leave empty for auto-detect
```

**Additional Docker Args:**
```bash
--docker-args "--memory=2g --cpus=2"
```

dbt-ci drives Docker through the Python SDK rather than the `docker` CLI, so `--docker-args` supports the subset of `docker run` flags that map onto SDK options. Both `--flag value` and `--flag=value` spellings work:

| Supported in `--docker-args` | Effect |
|------------------------------|--------|
| `--memory` / `-m` | Container memory limit |
| `--cpus` | CPU quota |
| `--shm-size` | Size of `/dev/shm` |
| `--env` / `-e` | Extra environment variables (merged with `--docker-env`) |
| `--add-host` | Extra host-to-IP mappings |
| `--workdir` / `-w` | Working directory inside the container |
| `--hostname` | Container hostname |
| `--privileged` | Extended privileges |
| `--platform`, `--network` | Override `--docker-platform` / `--docker-network` |

Anything else is logged as ignored rather than dropped silently.

> **Note:** containers are removed once the command finishes, so repeated CI runs don't accumulate stopped containers.

**Complete Docker Example:**
```bash
dbt-ci run \
  --runner docker \
  --docker-image ghcr.io/dbt-labs/dbt-postgres:1.7.0 \
  --docker-platform linux/amd64 \
  --docker-env "POSTGRES_HOST=host.docker.internal" \
  --docker-network host \
  --docker-volumes "$(pwd):/workspace" \
  --docker-volumes "$HOME/.aws:/root/.aws:ro" \
  --target prod
```

## Common Options

These flags are available on **every** command.

### Configuration File

dbt-ci supports a `dbt-ci.config.yaml` file as an alternative to passing every flag on the command line. It is loaded before any other options, and a flag passed on the command line always wins over it.

**Default location:** `dbt-ci.config.yaml` in the current working directory (override with `--config` / `DBT_CONFIG`).

If the config file is not found in the current directory, dbt-ci will automatically look for it inside `--dbt-project-dir` (or `DBT_PROJECT_DIR`). This means if your dbt project lives in a subdirectory (e.g. `dbt/`), placing `dbt-ci.config.yaml` there and setting `DBT_PROJECT_DIR=dbt` is enough — no `--config` flag needed.

The file uses a **nested style** where top-level keys map to common options, and command-specific options live under their command's key (`init`, `run`, `finalize`, `ephemeral`, `docker`):

Add the following comment to the top of your `dbt-ci.config.yaml` to get autocompletion and validation in editors that support [yaml-language-server](https://github.com/redhat-developer/yaml-language-server) (e.g. VS Code with the [YAML extension](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml)):

```yaml
# yaml-language-server: $schema=https://raw.githubusercontent.com/datablock-dev/dbt-ci/main/dbt-ci.config.schema.json
```

#### dbt-ci.config.yaml
```yaml
# yaml-language-server: $schema=https://raw.githubusercontent.com/datablock-dev/dbt-ci/main/dbt-ci.config.schema.json
project_dir: dbt
profiles_dir: dbt
state: dbt/.dbtstate
runner: docker

notifications:
  provider: slack
  format: compact   # the webhook itself is a secret: pass it via SLACK_WEBHOOK
  pr-comment: true  # dbt-ci report also comments on the pull request

init:
  state-uri: gs://my-bucket/dbt-state/manifest.json
  reference-target: production
  comparison-strategy: hybrid
  base-ref: main

run:
  nodes: models
  downstream-depth: 2

finalize:
  artifacts-uri: s3://my-bucket/dbt-artifacts/
  files:
    - manifest
    - cache

docker:
  image: docker.pkg.dev/my-project/dbt:latest
  volumes:
    - "${PWD}/dbt:/dbt:rw"
    - "${GOOGLE_APPLICATION_CREDENTIALS}:${GOOGLE_APPLICATION_CREDENTIALS}:ro"
  env:
    - "DBT_PROFILES_DIR=/dbt"
    - "GOOGLE_APPLICATION_CREDENTIALS=${GOOGLE_APPLICATION_CREDENTIALS}"
  network: host
```

The legacy flat `DBT_*` key style is also supported:

```yaml
DBT_RUNNER: docker
DBT_PROJECT_DIR: dbt
DBT_STATE: dbt/.dbtstate
DBT_DOCKER_IMAGE: ghcr.io/dbt-labs/dbt-core:latest
```

Other keys accepted in the config file include `dbt-version`, a `report:` section
(`format`, `output`) and a `notifications:` section (`provider`, `format`, `webhook`,
`pr-comment`), matching the flags of the same names.

**Precedence (highest → lowest):**
1. CLI flags
2. `dbt-ci.config.yaml`
3. Shell environment variables
4. Built-in defaults

The config file sits **above** shell environment variables: a value you commit to
`dbt-ci.config.yaml` is deliberate, whereas the environment a CI runner happens to
export is not. To override a config value per-run, pass the flag rather than setting the
environment variable.

`--filters` on `run` is command-line only — it has no environment variable or config
file key.

The config file is validated on load. dbt-ci will exit with a clear error message if it contains unknown keys, invalid enum values (e.g. `runner: kubernetes`), or wrong types (e.g. `defer: "yes"` instead of a boolean).

`${VAR_NAME}` references inside the config file are resolved from the shell environment at load time.

> **Note:** `dbt-ci.config.yaml` is ignored by git by default (it is listed in `.gitignore`). Use it for local developer overrides and commit a `.example` variant for your team.

### Core

| Flag | Aliases | Env Var(s) | Default | Description |
|------|---------|-----------|---------|-------------|
| `--dbt-project-dir` | | `DBT_PROJECT_DIR` | `.` | Path to the dbt project directory |
| `--profiles-dir` | | `DBT_PROFILES_DIR` | Auto-detect | Path to the directory containing `profiles.yml` |
| `--reference-state` | `--state` | `DBT_STATE` | `None` | Local path to the reference state directory (where `manifest.json` is stored) |
| `--target` | `-t` | `DBT_TARGET` | From `profiles.yml` | dbt target to use |
| `--vars` | `-v` | `DBT_VARS` | `""` | YAML string or path to a YAML file with dbt variables |
| `--defer` | | `DBT_DEFER`, `DEFER_FLAG` | `false` | Pass dbt's `--defer` flag (defers unmodified nodes to the production state). `DEFER_FLAG` is a fallback read only when `DBT_DEFER` is unset, and accepts `--defer`, `--no-defer`, an empty value or a boolean |
| `--runner` | `-r` | `DBT_RUNNER` | `dbt` | Runner to use: `dbt`, `local`, `docker`, `bash` |
| `--entrypoint` | | `DBT_ENTRYPOINT` | `dbt` | Command entrypoint for dbt |
| `--dbt-version` | | `DBT_VERSION` | Current | Pin a specific dbt version (e.g. `1.10.13`, or `2.0.8` for dbt v2). **Requires `--runner local`** |
| `--adapter` | `-a` | `DBT_ADAPTER` | `None` | dbt adapter to install (e.g. `dbt-bigquery`, `dbt-duckdb=1.10.0`). **Requires `--runner local`** |
| `--config` | `-c` | `DBT_CONFIG` | `dbt-ci.config.yaml` | Path to a dbt-ci YAML configuration file |
| `--dry-run` | | `DBT_DRY_RUN` | `false` | Print commands without executing them (and without installing a pinned `--dbt-version`) |
| `--quiet` | `-q` | `DBT_QUIET` | `false` | Run in quiet mode with minimal output |
| `--log-level` | | `DBT_LOG_LEVEL` | `INFO` | Logging verbosity: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL` (dbt's `warn` and `none` are accepted too). dbt reads `DBT_LOG_LEVEL` as well, so dbt-ci rewrites it into dbt's lowercase form (`WARNING` → `warn`, `CRITICAL` → `error`) before running dbt |
| `--slack-webhook` | `--slack-webhook-url`, `--notifications-webhook` | `DBT_NOTIFICATIONS_WEBHOOK`, `SLACK_WEBHOOK`, `SLACK_WEBHOOK_URL` | `None` | Webhook URL for CI notifications (`notifications.webhook` in the config file; the top-level `slack-webhook` key still works). `init` posts a summary of the modified, new and deleted nodes, truncated to the provider's message limit |
| `--notifications-provider` | | `DBT_NOTIFICATIONS_PROVIDER` | `slack` | Service the webhook belongs to (`notifications.provider`). Only `slack` is supported today |
| `--notifications-format` | | `DBT_NOTIFICATIONS_FORMAT` | `default` | Layout of the notification (`notifications.format`): `default` lists each changed node by name, `compact` sends only the counts, `json` sends the changed `unique_id`s as a JSON code block. Every layout opens with a header, the counts and, in GitHub Actions, the repository, branch and a link to the run |

### Docker Runner

Only used when `--runner docker` is set.

| Flag | Env Var(s) | Default | Description |
|------|-----------|---------|-------------|
| `--docker-image` | `DBT_DOCKER_IMAGE` | `ghcr.io/dbt-labs/dbt-core:latest` | Docker image to use |
| `--docker-platform` | `DBT_DOCKER_PLATFORM` | Auto-detect | Platform override, e.g. `linux/amd64` or `linux/arm64` |
| `--docker-volumes` | `DBT_DOCKER_VOLUMES` | `[]` | Volume mounts (repeatable): `host:container[:mode]`. In the env var, separate entries with commas or newlines |
| `--docker-env` | `DBT_DOCKER_ENV` | `[]` | Environment variables (repeatable): `KEY=VALUE`. In the env var, separate entries with newlines, or with a comma followed by the next `KEY=`; spaces and other commas stay part of the value |
| `--docker-network` | `DBT_DOCKER_NETWORK` | `host` | Docker network mode |
| `--docker-user` | `DBT_DOCKER_USER` | Invoking user (`uid:gid`) | User to run as inside the container (`UID:GID`). Defaults to the UID and GID of the process running dbt-ci so container-written files are owned by the invoking user. |
| `--docker-args` | `DBT_DOCKER_ARGS` | `""` | Extra arguments appended to `docker run` |

### Pinning a dbt version or adapter

`--dbt-version` and `--adapter` install the requested packages into a cached virtual
environment under `~/.cache/dbt-ci/venvs/` and run dbt from it. Only the **`local`**
runner can execute an arbitrary dbt binary, so the pin applies there:

```bash
dbt-ci run --runner local --dbt-version 1.10.13 --adapter dbt-duckdb=1.10.0
```

Adapters may be given with or without a version (`dbt-bigquery`, `dbt-bigquery=1.10.0`,
`dbt-bigquery==1.10.0`). Each version/adapter combination gets its own cached
environment, and an interrupted install is rebuilt on the next run rather than reused.

The other runners resolve dbt from elsewhere and log a warning if a pin is set:

| Runner | dbt comes from |
|--------|----------------|
| `dbt` | the `dbt-core` installed alongside dbt-ci (runs in-process) |
| `docker` | the configured `--docker-image` |
| `bash` | the script at `--shell-path`, or `--entrypoint` run through it when `--shell-path` is a shell such as `/bin/bash` |

### dbt v2

dbt v2 is a ground-up rewrite of dbt in Rust. It ships on PyPI as `dbt` (or `dbt-oss`)
rather than `dbt-core`, as a self-contained binary with every adapter bundled. dbt-ci
drives it through the CLI, so use the **`local`** or **`docker`** runner:

```bash
# dbt-ci installs dbt v2 into a cached virtual environment
dbt-ci init --runner local --dbt-version 2.0.8 ...
dbt-ci run  --runner local --dbt-version 2.0.8 ...

# ...or point dbt-ci at a dbt v2 binary you installed yourself
dbt-ci run --runner local --entrypoint /path/to/dbt ...

# ...or run dbt v2 from a Docker image
dbt-ci run --runner docker --docker-image my-registry/dbt-v2:2.0.8 ...
```

With the Docker runner, dbt-ci passes the dbt command (`ls ...`, `run ...`) as the
container command, so the image's entrypoint must be `dbt`, as in the official dbt
images. The default `--docker-image` is a dbt-core 1.x image, so pass a dbt v2 image
explicitly. A minimal one:

```dockerfile
FROM python:3.12-slim
RUN pip install --no-cache-dir "dbt==2.0.8"
ENTRYPOINT ["dbt"]
```

Things to know:

- **Keep dbt v2 out of dbt-ci's environment.** dbt-ci depends on dbt-core 1.x, and dbt v2
  and dbt-core both install a `dbt` Python package and `dbt` executable, so installing
  them into one environment overwrites one with the other. Install dbt-ci on its own
  (e.g. `pipx install dbt-ci` or `uv tool install dbt-ci`) and let `--dbt-version`
  or `--entrypoint` supply dbt v2, or run dbt v2 from an image with `--runner docker`.
- **The `dbt` runner is v1 only.** It runs the dbt-core installed alongside dbt-ci
  in-process.
- **`--adapter` is ignored for dbt v2**, because adapters are bundled. A warning is logged.
- **Deferral stays opt-in.** dbt v2 defers to `--state` by default, while dbt v1 only
  defers when asked. dbt-ci passes `--no-defer` whenever it passes `--state` without
  `--defer`, so `run`, `test` and `build` behave the same on both versions. `--defer`
  (or `DBT_DEFER`, or `DEFER_FLAG=--defer`) still turns deferral on.
- **Change detection works unchanged.** dbt v2 still writes `target/manifest.json` in the
  same v12 schema dbt-ci parses. CI checks every manifest field dbt-ci relies on (node
  ids, resource types, project-relative file paths, configs, lineage) against dbt-core
  1.10, 1.11, 1.12 and dbt v2, through both the local and the Docker runner.
- **Upgrading mid-stream is supported.** A reference state built with dbt-core 1.x can be
  compared against a PR that runs dbt v2, so CI keeps working while production is still
  on 1.x. Once production runs dbt v2, refresh the stored state so both sides match.

### Bash Runner

Only used when `--runner bash` is set.

| Flag | Aliases | Env Var(s) | Default | Description |
|------|---------|-----------|---------|-------------|
| `--shell-path` | `--bash-path` | `DBT_SHELL_PATH` | `/bin/bash` | Path to a dbt wrapper script, which receives the dbt arguments directly, or to a shell (`bash`, `sh`, `zsh`, `dash`, `ksh`), which runs `--entrypoint` (default `dbt`) with them |

The bash runner gets the same absolute `--project-dir`, `--profiles-dir` and `--state` paths as the `local` runner.

## Cloud Storage Support

dbt-ci supports storing and retrieving state files from cloud storage (GCS, S3), making it ideal for distributed CI/CD workflows.

> Requires the matching extra: `pip install 'dbt-ci[gcp]'` for `gs://` URIs, `pip install 'dbt-ci[aws]'` for `s3://` URIs.

### GCS/S3 State Storage

Store your dbt reference state in cloud storage for shared access across CI runs:

```bash
# Initialize and download state from GCS
dbt-ci init \
  --dbt-project-dir dbt \
  --state-uri gs://my-bucket/dbt-state/manifest.json \
  --reference-target production \
  --state dbt/.dbtstate

# Run using cached state (no need to specify URI again)
dbt-ci run --dbt-project-dir dbt --mode models
```

**Benefits:**
- 🔄 **Shared State**: Download the same reference state across different CI jobs
- 💾 **Cache-Based**: After init, commands use local cache (no repeated downloads)
- 📦 **No Git Commits**: State files don't need to be committed to version control
- 🚀 **Scalable**: Works seamlessly in containerized and distributed environments
- 🔐 **Secure**: Leverage cloud IAM and bucket policies for access control

**Configuration:**

The tool uses cloud credentials from your environment. Ensure your bucket is accessible:

```bash
# For GCS
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json

# For AWS S3
export AWS_ACCESS_KEY_ID=your_key
export AWS_SECRET_ACCESS_KEY=your_secret
export AWS_DEFAULT_REGION=us-east-1

# Or use IAM roles (recommended in CI/CD)
dbt-ci init --state-uri gs://my-bucket/manifest.json
```

**Supported URI Formats:**
- `gs://bucket-name/path/to/manifest.json` (Google Cloud Storage)
- `s3://bucket-name/path/to/manifest.json` (AWS S3)

## Environment Variables

All CLI options can also be set via environment variables:

```bash
export DBT_PROJECT_DIR=./dbt
export DBT_PROFILES_DIR=./dbt
export DBT_TARGET=production
export DBT_RUNNER=local

# After running init, just use:
dbt-ci run
```

**Common Environment Variables:**
- `DBT_PROJECT_DIR` - Path to dbt project
- `DBT_PROFILES_DIR` - Path to profiles.yml location
- `DBT_TARGET` - Target environment to use
- `DBT_RUNNER` - Runner type (local, docker, bash, dbt)
- `DBT_CI_CACHE_DIR` - Where the state cache and log file live (default: `<temp dir>/dbt-ci`)

**Note:** State management is cache-based. Run `init` once, then subsequent commands automatically use the cached state.

### Settings Inherited From `init`

`init` records the `--target`, `--vars` and `--state` it ran with, and later commands reuse
them, so they only need to be given once:

```bash
dbt-ci init --target ci --vars '{"use_production_data": false}' --state dbt/.dbtstate
dbt-ci run          # runs against target 'ci' with the same vars
dbt-ci delete       # likewise
```

The state path is recorded even when `init` chose it: with `--state-uri` and no `--state`,
the manifest is downloaded to `<dbt-project-dir>/.dbtstate`, and later commands such as
`ephemeral` pass that directory to dbt as `--state` (needed for `dbt clone` and other
deferring commands).

Passing the flag explicitly still wins — the cache only fills in what was left unset.

### Cache Location

`init` writes its cache (state comparison, manifests, run report and log file) to
`<temp dir>/dbt-ci`, and later commands read it from there. Because the location is
fixed, two dbt-ci runs executing at the same time on the same machine — for example two
pull request jobs on a shared self-hosted runner — would overwrite each other's state.

Set `DBT_CI_CACHE_DIR` to give each run its own directory:

```bash
export DBT_CI_CACHE_DIR="/tmp/dbt-ci-${GITHUB_RUN_ID}"
dbt-ci init ...
dbt-ci run ...
```

All commands in the same CI job must see the same value, since that is how `run`,
`delete`, `ephemeral` and `finalize` find the cache written by `init`.

## CI/CD Integration

### GitHub Actions Example

```yaml
name: dbt CI

on: [pull_request]

jobs:
  dbt-ci:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      
      - name: Set up Python
        uses: actions/setup-python@v7
        with:
          python-version: '3.12'
      
      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v2
        with:
          role-to-assume: arn:aws:iam::123456789012:role/GitHubActionsRole
          aws-region: us-east-1
      
      - name: Install dbt-ci
        # The gcp extra provides the GCS client needed for the gs:// state URI below
        run: pip install 'dbt-ci[gcp] @ git+https://github.com/datablock-dev/dbt-ci.git@main'
      
      - name: Initialize dbt-ci with cloud state
        run: |
          dbt-ci init \
            --dbt-project-dir dbt \
            --state-uri gs://my-dbt-state/prod/manifest.json \
            --reference-target production \
            --state dbt/.dbtstate
      
      - name: Run modified models
        run: |
          dbt-ci run --mode models
```

### GitLab CI Example

```yaml
dbt-ci:
  image: python:3.12
  script:
    - pip install 'dbt-ci[gcp] @ git+https://github.com/datablock-dev/dbt-ci.git@main'
    - dbt-ci init --dbt-project-dir dbt --state-uri gs://my-dbt-state/prod/manifest.json --reference-target production --state dbt/.dbtstate
    - dbt-ci run --mode models
  only:
    - merge_requests
```

## Features

- **🎯 Smart Detection**: Automatically identifies modified, new, and deleted models
- **📊 Dependency Tracking**: Generates and traverses dependency graphs for lineage analysis
- **🔄 State Comparison**: Compares current state against production for precise CI
- **☁️ Cloud Storage**: S3 integration for shared state across distributed CI/CD workflows
- **🚀 Multiple Runners**: Supports local, Docker, bash, and dbt Python API execution
- **🐳 Docker-First**: Extensive Docker configuration for containerized workflows
- **⚡ Selective Execution**: Run only what changed, saving time and resources
- **🔌 Adapter Support**: Install specific dbt versions and adapters on-demand
- **💬 Notifications**: Slack webhook integration for CI/CD alerts
- **♻️ Ephemeral Environments**: Test changes in isolated environments
- **🧹 Cleanup**: Automatically remove deleted models from target warehouse
- **🎯 Blast-Radius Control**: Cap how far a change propagates with `--downstream-depth`
- **📝 Run Reports**: Markdown summary of the change set, exposure impact and command status
- **🔀 Partition Migrations**: Rebuild tables whose partitioning configuration changed (BigQuery)

## Use Cases

### Pull Request CI
Only build and test models affected by PR changes:
```bash
# Initialize with reference state
dbt-ci init --state-uri gs://bucket/manifest.json --reference-target production --state dbt/.dbtstate

# Run modified models with defer
dbt-ci run --mode models --defer
```

### Distributed CI with Cloud Storage
Share state across multiple CI jobs:
```bash
# Job 1: Initialize state (downloads from cloud)
dbt-ci init --state-uri gs://my-bucket/manifest.json --reference-target production --state dbt/.dbtstate

# Job 2: Run models (uses cached state)
dbt-ci run --mode models

# Job 3: Run tests (uses cached state)
dbt-ci run --mode tests
```

### Selective Testing
Run tests only for modified models:
```bash
# After init
dbt-ci run --mode tests
```

### Schema Migrations
Clean up deleted models from production:
```bash
# After init
dbt-ci delete --target production
```

### Multi-Environment Testing
Create ephemeral test environments:
```bash
dbt-ci ephemeral --keep-env
```

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

### Development Setup

1. Clone the repository
2. Install dependencies: `pip install -e ".[dev]"`
3. Run tests: `pytest tests/`
4. Run linting: `black dbt_ci/ tests/`

### Commit Message Format

This project uses [Conventional Commits](https://www.conventionalcommits.org/) for automated releases:

- `feat:` New feature (minor version bump)
- `fix:` Bug fix (patch version bump)
- `docs:` Documentation changes
- `refactor:` Code refactoring
- `test:` Adding tests
- `chore:` Maintenance tasks

Example:
```bash
git commit -m "feat: add Docker runner support"
git commit -m "fix: resolve path resolution on Windows"
```

See [RELEASING.md](RELEASING.md) for details on the automated release process.

## License

See [LICENSE](LICENSE) file for details.

## Links

- **PyPI**: [https://pypi.org/project/dbt-ci/](https://pypi.org/project/dbt-ci/)
- **Documentation**: [https://datablock.dev](https://datablock.dev)
- **Issues**: [GitHub Issues](https://github.com/datablock-dev/dbt-ci/issues)
- **Discussions**: [GitHub Discussions](https://github.com/datablock-dev/dbt-ci/discussions)
- **Changelog**: [CHANGELOG.md](CHANGELOG.md)
