Metadata-Version: 2.5
Name: dbt-data-contracts
Version: 0.3.2
Summary: A lightweight data contract control plane for multi-repository dbt ecosystems
Author: dbt_data_contracts contributors
License: MIT
License-File: LICENSE
Keywords: ci-cd,data-contracts,data-mesh,dbt,semver
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Requires-Python: >=3.11
Requires-Dist: pydantic>=2.5.0
Requires-Dist: pyyaml>=6.0.1
Requires-Dist: rich>=13.7.0
Requires-Dist: semver>=3.0.0
Requires-Dist: sqlalchemy>=2.0.0
Requires-Dist: sqlglot>=23.0.0
Requires-Dist: typer>=0.12.0
Provides-Extra: dev
Requires-Dist: httpx>=0.27.0; extra == 'dev'
Requires-Dist: mypy>=1.10.0; extra == 'dev'
Requires-Dist: pytest-cov>=5.0.0; extra == 'dev'
Requires-Dist: pytest>=8.0.0; extra == 'dev'
Requires-Dist: ruff>=0.4.0; extra == 'dev'
Requires-Dist: types-pyyaml>=6.0.12; extra == 'dev'
Provides-Extra: server
Requires-Dist: fastapi>=0.110.0; extra == 'server'
Requires-Dist: uvicorn>=0.28.0; extra == 'server'
Description-Content-Type: text/markdown

# dbt-data-contracts (`dbt_data_contracts`)

[![PyPI version](https://img.shields.io/pypi/v/dbt-data-contracts.svg)](https://pypi.org/project/dbt-data-contracts/)
[![Python versions](https://img.shields.io/pypi/pyversions/dbt-data-contracts.svg)](https://pypi.org/project/dbt-data-contracts/)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
[![Tests](https://img.shields.io/badge/tests-249%20passed%20(100%25)-brightgreen.svg)](tests/)

> An enterprise-grade, lightweight **data contract control plane** for multi-repository dbt ecosystems, mesh architectures, and data platforms.

---

## Overview

dbt model contracts validate what a model produces internally within its own project. However, in modern multi-repository data ecosystems and data mesh topologies, data producers face critical challenges:

- **Invisible Downstream Breakage**: How do producers know if altering a model breaks downstream consumers across separate git repositories?
- **No Heavy Cloning in CI**: Producer CI pipelines should **not** clone, parse, or execute hundreds of downstream repositories just to verify compatibility.
- **Explicit Consumer Commitments**: Downstream teams need a structured, versioned way to declare exact column, type, quality, and SLA requirements.
- **Transitive Impact Simulation**: A single model change can cascade across multiple hops of models, dashboards, and APIs.

`dbt-data-contracts` provides a complete, vendor-neutral control plane to declare, ingest, publish, verify, discover, simulate, and enforce data contracts across distributed data teams.

---

## Core Capabilities

```text
┌─────────────────────────────────────────────────────────────────────────────────────────┐
│                                   dbt Ecosystem & Mesh                                  │
│   dbt manifest.json / static YAML / OpenAPI Specs / Consumer Declarations / Warehouses  │
└────────────────────────────────────────────┬────────────────────────────────────────────┘
                                             │ Ingestion & Auto-Discovery
                                             ▼
┌─────────────────────────────────────────────────────────────────────────────────────────┐
│                         Canonical Data Contract Domain Layer                            │
│           PublishedContract • ColumnContract • SemanticModels • FreshnessSLA            │
└───────────────────────┬─────────────────────────────────────────┬───────────────────────┘
                        │                                         │
                        ▼                                         ▼
┌──────────────────────────────────────┐       ┌──────────────────────────────────────────┐
│       Universal Contract Registry     │       │        Compatibility & Blast Engine      │
│  SQLite • PostgreSQL • GitOps • HTTP  │◄─────►│    SemVer Rules • Transitive DAG Graphs   │
│  Immutability • Promotion • Lifecycle │       │    Quality • SLAs • Warehouse Drift      │
└───────────────────────┬──────────────┘       └──────────────────┬───────────────────────┘
                        │                                         │
                        ▼                                         ▼
┌─────────────────────────────────────────────────────────────────────────────────────────┐
│                              Control Plane & CI/CD Gates                                │
│    dbt-data-contracts CLI • Pre-Commit Hooks • PR Comment Reporters • Web Dashboard     │
│    OpenAPI/ODCS/Avro/Protobuf Exporters • Mock Generators • Slack/Teams Webhooks        │
└─────────────────────────────────────────────────────────────────────────────────────────┘
```

- **Canonical Data Contracts**: Pure domain representations independent of dbt, databases, or cloud vendors.
- **SemVer Compatibility Engine**: Deterministic classification of changes (`PATCH`, `MINOR`, `MAJOR`, `UNKNOWN`) based on column changes, multi-dialect typing, nullability, and quality assertions.
- **Explicit Consumer Expectations**: Downstream teams declare version ranges (e.g., `^1.0.0`) and required columns in `data-contract-consumers.yml`.
- **Automated SQL AST Lineage & Consumer Inference**: SQLGlot-based column-level lineage engine that inspects Jinja `ref()` expressions and SQL ASTs to auto-generate consumer declarations (`dbt-data-contracts consumer infer`).
- **Multi-Hop Transitive Blast-Radius & Dependency DAG Simulator**: Construct complete dependency graphs, discover cascading impact paths across N tiers of consumers, and render ASCII trees or Mermaid diagrams (`dbt-data-contracts impact`).
- **Pre-Commit Git Hook & Local Developer Loop**: Fast local validation of staged dbt files against contracts, governance policies, and waivers before pushing (`dbt-data-contracts hook check`).
- **Automated PR Sticky Comments & CI Gates**: Native sticky PR thread status management on GitHub Actions and Azure DevOps Pipelines.
- **Organization-Scale Auto-Discovery with Activity Filtering**: Discover dbt projects, manifests, and consumer contracts across Azure DevOps, GitHub, GitLab, and Bitbucket with commit age filtering (`--max-commit-age-days 7`) and multi-branch scanning (`--branch`).
- **OpenAPI 3.0 / 3.1 & GraphQL SDL Adapters**: Auto-discover OpenAPI specifications in repositories, import API schemas as contracts (`import openapi`), and export published contracts to OpenAPI 3.1 and GraphQL SDL.
- **Open Data Standards Interoperability**: Bi-directional translation for Open Data Contract Standard (ODCS v3.0), Apache Avro (`.avsc`), Protocol Buffers v3 (`.proto`), JSON Schema (2020-12), OpenLineage assertion facets, and DataHub / OpenMetadata emitters.
- **AI Coding Agent Skills & Rules Generator**: Bootstrap specialized ContractOps rules, instructions, and workflows for Cursor (`.cursor/rules` & `SKILL.md`), GitHub Copilot (`copilot-instructions.md`), Claude Code (`CLAUDE.md`), Cline (`.clinerules`), and Windsurf (`.windsurfrules`) (`dbt-data-contracts codegen agent-skills`).
- **CI/CD Pipeline Generator for Producer & Consumer Checks**: Generate turnkey CI/CD pipeline code for Azure DevOps, GitHub Actions, GitLab CI, and Bitbucket (`dbt-data-contracts codegen pipeline`).
- **Synthetic Test Data & Mock Dataset Generator**: Deterministically generate mock test datasets conforming strictly to contract types, constraints, nullability, allowed values, and patterns in CSV, JSON, or SQL `INSERT` format (`dbt-data-contracts codegen mock`).
- **Automated Backward-Compatibility View & Shim Generator**: Generate dbt SQL shim models mapping new breaking outputs back to legacy schemas, and generate consumer migration YAMLs (`dbt-data-contracts codegen shim` / `codegen migrate-consumer`).
- **Contract Sunset & Deprecation Lifecycle**: Track contract statuses (`ACTIVE`, `DEPRECATED`, `TOMBSTONED`), calculate breaking consumer impact when models disappear, and enforce deprecation/sunset deadlines (`dbt-data-contracts check sunset` / `registry deprecate/tombstone/reconcile`).
- **Multi-Environment Registry Promotion**: Promote contracts and consumer expectations between environments (e.g. dev -> staging -> prod) with conflict detection (`dbt-data-contracts registry promote`).
- **Live Warehouse Schema Introspection & Drift Detection**: Introspect live physical warehouse tables across Snowflake, BigQuery, Databricks, Postgres, Redshift, and DuckDB to detect live schema drift (`dbt-data-contracts check drift`).
- **Control Plane REST API & Web Dashboard**: FastAPI server with interactive web UI at `/` for catalog exploration, version histories, and consumer lineage DAG visualization (`dbt-data-contracts server start`).
- **Real-Time Webhook Notifications**: Event-driven alerts with Slack Block Kit, Microsoft Teams Adaptive Cards, and HMAC-SHA256 signature verification.

---

## Installation

```bash
# Core package with CLI
pip install dbt-data-contracts

# Optional server extensions (FastAPI & Uvicorn for REST API / Web Dashboard)
pip install "dbt-data-contracts[server]"
```

Both `dbt-data-contracts` and `dbt-contracts` are available as CLI aliases.

---

## Quickstart & Core Workflow

### 1. Ingest contracts from a dbt manifest

Inspect public contracts discovered in a dbt project manifest:

```bash
dbt-data-contracts ingest --manifest target/manifest.json
```

### 2. Publish a contract version to the registry

Publish contracts for public dbt models (e.g. version `1.0.0`):

```bash
dbt-data-contracts publish \
  --manifest target/manifest.json \
  --version 1.0.0 \
  --registry .contracts.db
```

### 3. Register a downstream consumer expectation

Downstream teams define `data-contract-consumers.yml` (or generate it automatically with `consumer infer`):

```yaml
consumer: finance_analytics
owner: team-finance@example.com

dependencies:
  - product: core_sales
    model: fct_orders
    version: "^1.0.0"
    expectations:
      columns:
        order_id:
          data_type: bigint
          required: true
        amount:
          data_type: numeric
          required: true
      quality:
        amount:
          min_value: 0
      sla:
        max_staleness: "24h"
```

Register the expectation in the registry:

```bash
dbt-data-contracts consumer register \
  --file data-contract-consumers.yml \
  --registry .contracts.db
```

### 4. Check proposed changes in Producer CI

In producer pull request pipelines, test proposed contracts against published contracts and registered consumers:

```bash
dbt-data-contracts check \
  --manifest target/manifest.json \
  --proposed-version 1.1.0 \
  --registry .contracts.db \
  --post-pr-comment
```

If a breaking change is detected (e.g., removing `amount` while `finance_analytics` requires it), the gate fails with exit code `1` and posts an explanatory breakdown:

```text
BREAKING CHANGE DETECTED: core_sales.fct_orders

Proposed: 1.1.0 (Current: 1.0.0)
Severity: MAJOR

Breaking changes:
  - Column 'amount' was removed

Affected consumers:
  - finance_analytics (expects 'amount: numeric', pinned to ^1.0.0)

SemVer Validation:
  FAIL: Breaking changes require a MAJOR version bump (expected >= 2.0.0).
```

If the proposed version is updated to `2.0.0`, the check passes with exit code `0`, confirming that `1.0.0` remains intact for existing consumers while `2.0.0` introduces the new interface version.

---

## CLI Command Reference

| Command Group | Command | Description |
| :--- | :--- | :--- |
| **Ingest & Publish** | `ingest` | Inspect public models from a dbt `manifest.json`. |
| | `publish` | Publish public dbt models as versioned immutable contracts. |
| **Compatibility & CI**| `check` | Check proposed contract changes against registry and consumers (with `--auto-review`). |
| | `check drift` | Inspect live physical warehouse tables and detect schema drift. |
| | `check sunset` | Evaluate contract deprecation and sunset deadlines against consumers. |
| | `diff` | Diff contract schemas between manifests or against the registry. |
| **Discovery** | `discover` | Discover local nested dbt projects, manifests, and consumer files. |
| | `discover organization` | Scan remote repos (Azure DevOps, GitHub, GitLab, Bitbucket) with recent commit filtering (`--max-commit-age-days`) and branch scanning (`--branch`). |
| **Dependency & Impact**| `impact` / `blast-radius` | Evaluate multi-hop transitive blast radius and export DAG diagrams (ASCII/Mermaid). |
| **Consumers** | `consumer register` | Register a `data-contract-consumers.yml` file into the registry. |
| | `consumer list` | List registered consumer expectations. |
| | `consumer infer` | Auto-generate consumer expectations from SQL AST column lineage and `ref()` calls. |
| **Code Generation** | `codegen shim` | Generate a backward-compatible dbt SQL shim model for major upgrades. |
| | `codegen migrate-consumer` | Generate consumer migration YAML for upgrading contract version pins. |
| | `codegen mock` | Generate synthetic mock datasets (CSV, JSON, SQL `INSERT`) conforming to contract specs. |
| | `codegen dbt-test` | Generate dbt schema test YAML, singular test SQL, or compliance macros from a contract. |
| | `codegen changelog` | Generate Markdown release notes and changelog between contract versions. |
| | `codegen pipeline` | Generate complete CI/CD pipeline code (producer gating or consumer checks) for Azure DevOps, GitHub Actions, GitLab CI, and Bitbucket. |
| | `codegen agent-skills` | Generate AI coding agent rules and skills for Cursor, GitHub Copilot, Claude, Cline, and Windsurf. |
| **Git Hooks** | `hook check` | Pre-commit git hook validation for staged dbt files and schema changes. |
| **Standards & Schemas**| `export` | Export contracts to `openapi`, `graphql`, `odcs`, `avro`, `proto`, `jsonschema`. |
| | `import` | Import contract definitions from `openapi` (3.0/3.1) or `odcs`. |
| **Registry Management**| `registry list` | List published contracts and version histories. |
| | `registry deprecate` | Mark a contract version as deprecated with a sunset date. |
| | `registry tombstone` | Mark a contract version as tombstoned. |
| | `registry reconcile` | Detect disappeared/deleted models and compute breaking consumer impacts. |
| | `registry promote` | Replicate and promote contracts across registry environments (dev -> prod). |
| **Control Plane Server**| `server start` | Launch the FastAPI control plane REST API and interactive web dashboard. |

---

## Developer Workflow & Pre-Commit Hook

Integrate contract validation into your local development workflow using `.pre-commit-config.yaml`:

```yaml
repos:
  - repo: https://github.com/dbt-data-contracts/dbt-data-contracts
    rev: v0.3.2
    hooks:
      - id: dbt-data-contracts-check
        args: ["--registry", ".contracts.db"]
```

Or run locally directly:

```bash
dbt-data-contracts hook check --registry .contracts.db
```

---

## Multi-Hop Transitive Blast Radius

Simulate cascading downstream impacts across complex data mesh graphs:

```bash
dbt-data-contracts impact \
  --product core_sales \
  --model fct_orders \
  --registry .contracts.db \
  --depth 5 \
  --format ascii
```

Output:
```text
TRANSITIVE BLAST RADIUS REPORT: core_sales.fct_orders
Total Direct Consumers:     1
Total Transitive Consumers: 2
Total Impacted Models:      2

IMPACT GRAPH:
core_sales.fct_orders [PRODUCER]
  ├── finance_analytics (direct) [fct_monthly_revenue]
  │     └── executive_dashboard (transitive) [mart_arr_summary]
  └── marketing_attribution (direct) [fct_customer_ltv]
```

Or export as Mermaid diagrams for documentation:

```bash
dbt-data-contracts impact --product core_sales --model fct_orders --format mermaid
```

---

## Standards & API Interoperability

### OpenAPI 3.0/3.1 Import & Export

Export dbt data product contracts to OpenAPI 3.1 JSON/YAML:

```bash
dbt-data-contracts export openapi \
  --product core_sales \
  --model fct_orders \
  --registry .contracts.db \
  --output openapi.json
```

Import existing OpenAPI definitions as data contracts:

```bash
dbt-data-contracts import openapi \
  --file specs/orders-api.yaml \
  --product core_sales \
  --version 1.0.0 \
  --registry .contracts.db
```

### GraphQL SDL Export

```bash
dbt-data-contracts export graphql \
  --product core_sales \
  --model fct_orders \
  --registry .contracts.db \
  --output schema.graphql
```

### Open Data Contract Standard (ODCS v3.0) & Streaming

```bash
# Export to ODCS YAML
dbt-data-contracts export odcs --product core_sales --model fct_orders --registry .contracts.db

# Export to Apache Avro schema (.avsc)
dbt-data-contracts export avro --product core_sales --model fct_orders --registry .contracts.db

# Export to Protocol Buffers v3 (.proto)
dbt-data-contracts export proto --product core_sales --model fct_orders --registry .contracts.db
```

---

## Universal Registry Backends

`dbt-data-contracts` supports SQLite, PostgreSQL, GitOps repositories, and HTTP REST endpoints:

```bash
# Local SQLite file
dbt-data-contracts check --registry .contracts.db

# Central PostgreSQL database
export CONTRACTS_DATABASE_URL="postgresql://user:password@pg.company.internal:5432/contracts"
dbt-data-contracts check --manifest target/manifest.json

# GitOps-backed repository catalog
dbt-data-contracts check --registry "git+https://github.com/company/data-contracts-catalog.git"

# Remote Control Plane REST API
dbt-data-contracts check --registry "http://contracts.internal.net:8000" --token "$CONTRACTS_TOKEN"
```

---

## Running the Control Plane Server

Launch the REST API server and interactive web dashboard:

```bash
pip install "dbt-data-contracts[server]"
dbt-data-contracts server start --host 0.0.0.0 --port 8000 --registry .contracts.db
```

Open `http://localhost:8000` in your browser to explore the catalog, inspect contract schemas and version histories, visualize consumer lineage DAGs, and verify compatibility interactively. Use `http://localhost:8000/graphql` for the live GraphiQL playground.

---

## Static Documentation Portal & Markdown Docs

Generate standalone zero-dependency static documentation sites (e.g. for GitHub Pages, S3, or Azure Blob):

```bash
# Build single-page static HTML documentation portal with client-side search
dbt-data-contracts docs build --registry .contracts.db --output ./site

# Export structured Markdown folder tree for MkDocs / Docusaurus
dbt-data-contracts docs export-markdown --registry .contracts.db --output docs/contracts
```

---

## Automated Consumer Version Upgrades & PRs

Synchronize consumer expectation files when upstream data products publish new versions:

```bash
dbt-data-contracts consumer upgrade \
  --file data-contract-consumers.yml \
  --product core_sales \
  --model fct_orders \
  --target-version "^2.0.0" \
  --registry .contracts.db
```

---

## AI Coding Agent Skills & Rules Generator

Equip AI coding agents (Cursor, GitHub Copilot, Claude Code, Cline, Windsurf) with specialized knowledge about data contracts, SemVer breaking change rules, consumer expectations, and migration workflows:

```bash
# Generate rules and skills for a specific framework
dbt-data-contracts codegen agent-skills --framework cursor
dbt-data-contracts codegen agent-skills --framework copilot
dbt-data-contracts codegen agent-skills --framework claude

# Generate skills for all supported frameworks at once
dbt-data-contracts codegen agent-skills --framework all --output-dir .

# Or include automatically during repository initialization
dbt-data-contracts init --agent-skills cursor
```

Generated files include:
- **Cursor**: `.cursor/rules/dbt-data-contracts.mdc` and `.cursor/skills/dbt-data-contracts/SKILL.md`
- **GitHub Copilot**: `.github/copilot-instructions.md`
- **Claude Code**: `CLAUDE.md`
- **Cline / Roo-Code**: `.clinerules`
- **Windsurf Cascade**: `.windsurfrules`

---

## CI/CD Pipeline Generator for Producer & Consumer Checks

Generate turnkey CI/CD pipeline definitions across Azure DevOps, GitHub Actions, GitLab CI, and Bitbucket Pipelines:

```bash
# Generate Producer Gate (checks PRs against downstream consumers)
dbt-data-contracts codegen pipeline \
  --platform azure-devops \
  --role producer \
  --manifest target/manifest.json \
  --registry "$CONTRACTS_DATABASE_URL" \
  --post-pr-comment \
  --auto-review \
  --output azure-pipelines-contracts.yml

# Generate Consumer Verification Pipeline (checks upstream dependencies & sunsets)
dbt-data-contracts codegen pipeline \
  --platform github-actions \
  --role consumer \
  --consumer-file data-contract-consumers.yml \
  --registry .contracts.db \
  --output .github/workflows/consumer-contracts.yml
```

---

## Runtime Model Execution Guard

Prevent orchestrators (Airflow, Dagster, Prefect, GitHub Actions) from executing `dbt run`/`dbt build` against tombstoned or expired sunset models:

```bash
dbt-data-contracts exec -- dbt build --select +fct_orders
```

---

## Warehouse Data Diff & Value Distribution Validator

Run database pushdown SQL queries to validate actual table data against quality constraints (nullability, min/max bounds, allowed enums, uniqueness):

```bash
dbt-data-contracts check data-diff \
  --product core_sales \
  --model fct_orders \
  --connection-url "snowflake://user:pass@account/db/schema" \
  --fail-on-violation
```

---

## Governance Policies & Breaking-Change Waivers

Enforce organizational governance policies using `.dbt-contracts-policy.yml`:

```yaml
require_owner: true
require_model_description: true
require_column_descriptions: true
min_column_description_coverage: 0.8
allowed_data_types:
  - bigint
  - numeric
  - text
  - timestamp
  - boolean
```

Temporarily authorize breaking changes during migrations using `.waivers.yml`:

```yaml
waivers:
  - product: core_sales
    model: fct_orders
    rule: ColumnRemovedRule
    reason: "Decommissioning legacy amount column in Q3 refactor (Jira: DATA-1234)"
    expires_at: "2026-12-31T23:59:59Z"
    approved_by: "data-gov-board"
```

---

## Development & Testing

```bash
# Clone the repository
git clone https://github.com/your-org/dbt_contracts.git
cd dbt_contracts

# Set up virtual environment
python -m venv .venv
source .venv/bin/activate  # Or .venv\Scripts\Activate.ps1 on Windows

# Install in editable mode with dev dependencies
pip install -e ".[dev,server]"

# Run tests and linting
pytest
ruff check .
ruff format --check .
mypy src tests
```

---

## License

MIT License. See [LICENSE](LICENSE) for details.
