Metadata-Version: 2.4
Name: databricks-schema
Version: 0.7.0
Summary: Databricks Unity Catalog schema extractor
Project-URL: Homepage, https://github.com/frizzleqq/databricks-schema
Project-URL: Repository, https://github.com/frizzleqq/databricks-schema
Project-URL: Issues, https://github.com/frizzleqq/databricks-schema/issues
Project-URL: Changelog, https://github.com/frizzleqq/databricks-schema/releases
Author: frizzleqq
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: databricks,schema,sql,unity-catalog
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.11
Requires-Dist: databricks-sdk>=0.64.0
Requires-Dist: pydantic>=2.0
Requires-Dist: pyyaml>=6.0
Description-Content-Type: text/markdown

# databricks-schema

A CLI tool and Python library that uses the Databricks SDK to extract and diff Unity Catalog schemas as YAML files. It can also generate Databricks Spark SQL to apply schema changes across catalogs.

## Getting Started

Install from PyPI (requires Python 3.11+):

```bash
pip install databricks-schema
```

Configure your Databricks credentials (see [Authentication](#authentication) for all supported methods).

Then extract, diff, and generate SQL for your Unity Catalog schemas:

```bash
# 1. Find the catalog you want to snapshot
databricks-schema list-catalogs

# 2. Extract its schemas to YAML files (one file per schema)
databricks-schema extract prod_catalog --output-dir ./schemas/

# 3. Diff those files against a catalog (same or different)
databricks-schema diff test_catalog ./schemas/

# 4. Generate SQL to bring that catalog in line with the YAML files
databricks-schema generate-sql test_catalog ./schemas/ --output-dir ./migrations/
```

The YAML files act as a version-controllable snapshot of your schema. The `diff` command exits with code `1` when differences are found, making it suitable for CI pipelines.

## Output Format

Each schema is written to `{output-dir}/{schema-name}.yaml`. Fields with no value (null comments, empty tag dicts, empty FK lists) are omitted. Use `--format json` to write `.json` files with the same structure.

```yaml
name: main
comment: Main production schema
tags:
  env: prod
tables:
  - name: users
    table_type: MANAGED
    comment: User accounts
    tags:
      domain: identity
    columns:
      - name: id
        data_type: bigint
        nullable: false
        comment: Primary key
      - name: email
        data_type: string
      - name: org_id
        data_type: bigint
    primary_key:
      name: pk_users
      columns:
        - id
    foreign_keys:
      - name: fk_org
        columns:
          - org_id
        ref_schema: orgs
        ref_table: organizations
        ref_columns:
          - id
```

Without `--output-dir`, `extract` prints a single `Catalog` document to stdout instead — the same schema shape nested under a `schemas` list, with the catalog name as a top-level field:

```yaml
name: prod_catalog
schemas:
  - name: main
    comment: Main production schema
    tags:
      env: prod
    tables: [...]
```

## Authentication

The tool uses the [Databricks SDK](https://github.com/databricks/databricks-sdk-py) for auth. Configure it via environment variables:

```bash
export DATABRICKS_HOST=https://<databricks-url>
export DATABRICKS_TOKEN=<your-personal-access-token>
```

Or use a [Databricks CLI profile](https://docs.databricks.com/dev-tools/cli/profiles.html) (`~/.databrickscfg`) — the SDK will pick it up automatically.

You can also pass credentials directly as flags (see `--host` / `--token` below).

## Agent Skill

This repo ships an [Agent Skill](https://code.claude.com/docs/en/skills) at
[`.claude/skills/databricks-schema-cli/SKILL.md`](.claude/skills/databricks-schema-cli/SKILL.md)
that teaches an AI coding agent (e.g. Claude Code) how to use the `databricks-schema` CLI to
explore, snapshot, diff, and generate migration SQL for a live Unity Catalog. Point an agent at
this repo (or copy the skill into your own project) and it can use the CLI directly instead of
writing ad-hoc Databricks SDK calls.

The skill assumes the `databricks-schema` command is on `PATH`. Install it as a standalone tool
with [uv](https://github.com/astral-sh/uv):

```bash
uv tool install databricks-schema
```

To use the skill in your own project without cloning this repo, download it directly:

```bash
mkdir -p .claude/skills/databricks-schema-cli
curl -o .claude/skills/databricks-schema-cli/SKILL.md \
  https://raw.githubusercontent.com/frizzleqq/databricks-schema/main/.claude/skills/databricks-schema-cli/SKILL.md
```

## Development

Requires Python 3.11+ and [uv](https://github.com/astral-sh/uv).

```bash
git clone https://github.com/frizzleqq/databricks-schema
cd databricks-schema
uv sync --all-groups  # includes pytest and ruff
```

```bash
# Run tests
uv run pytest

# Lint
uv run ruff check databricks_schema/ tests/

# Format
uv run ruff format databricks_schema/ tests/
```

## CLI Usage

```
databricks-schema [OPTIONS] COMMAND [ARGS]...
```

`extract`, `diff`, `generate-sql`, and `diff-files` print progress messages (e.g. `Extracting
catalog 'X'...`) to stderr as they run. Pass `--quiet` / `-q` to suppress them; results and
errors are unaffected.

### `extract`

Extract all schemas from a catalog to YAML files:

```bash
databricks-schema extract <catalog> --output-dir ./schemas/
```

Use `--format json` to write `.json` files instead of `.yaml`.

Extract specific schemas only:

```bash
databricks-schema extract <catalog> --schema main --schema raw --output-dir ./schemas/
```

Or extract a single schema or table by naming it directly, dotted onto the catalog (output keeps the same `Catalog`/`Schema` shape, just filtered down). This cannot be combined with `--schema`:

```bash
databricks-schema extract <catalog>.<schema>
databricks-schema extract <catalog>.<schema>.<table>
```

Print to stdout instead of writing files (no `--output-dir`) — the catalog and its matching schemas are printed as a single `Catalog` document (`name` + `schemas: [...]`):

```bash
databricks-schema extract <catalog> --schema main
```

Include Unity Catalog tags in the output (extra API calls per entity; excluded by default):

```bash
databricks-schema extract <catalog> --output-dir ./schemas/ --include-tags
```

Include additional metadata (`owner`, `storage_location`) in the output:

```bash
databricks-schema extract <catalog> --output-dir ./schemas/ --include-metadata
```

Control the number of parallel workers (default: 4):

```bash
databricks-schema extract <catalog> --output-dir ./schemas/ --workers 8
```

### `diff`

Compare the live catalog against previously extracted schema files (format auto-detected from the directory — YAML or JSON, not mixed):

```bash
databricks-schema diff <catalog> ./schemas/
```

Or compare two live catalogs directly, e.g. dev against prod — no local files needed. The second argument is treated as a directory if one exists at that path, otherwise as a catalog name:

```bash
databricks-schema diff dev_catalog prod_catalog
```

Compare specific schemas only:

```bash
databricks-schema diff <catalog> ./schemas/ --schema main --schema raw
```

Include Unity Catalog tags in the comparison (excluded by default):

```bash
databricks-schema diff <catalog> ./schemas/ --include-tags
```

Include `owner` in the comparison (excluded by default):

```bash
databricks-schema diff <catalog> ./schemas/ --include-metadata
```

Exits with code `0` if no differences are found, `1` if there are — making it suitable for CI pipelines. Output example:

```
~ Schema: main [MODIFIED]
  ~ Table: users [MODIFIED]
    ~ Column: score [MODIFIED]
        data_type: 'int' -> 'double'
    + Column: phone [ADDED]
  + Table: events [ADDED]
- Schema: legacy [REMOVED]
```

Markers: `+` added in catalog, `-` removed from catalog, `~` modified.

### `generate-sql`

Generate Databricks Spark SQL statements to bring the live catalog in line with local schema files (format auto-detected, YAML or JSON, not mixed). Statements are printed to stdout by default:

```bash
databricks-schema generate-sql <catalog> ./schemas/
```

Write one `.sql` file per schema to a directory instead:

```bash
databricks-schema generate-sql <catalog> ./schemas/ --output-dir ./migrations/
```

Destructive statements (`DROP SCHEMA`, `DROP TABLE`, `DROP COLUMN`) are emitted as SQL comments by default. Pass `--allow-drop` to emit them as executable statements:

```bash
databricks-schema generate-sql <catalog> ./schemas/ --allow-drop
```

Filter to specific schemas:

```bash
databricks-schema generate-sql <catalog> ./schemas/ --schema main --schema raw
```

Include Unity Catalog tags in the comparison (excluded by default):

```bash
databricks-schema generate-sql <catalog> ./schemas/ --include-tags
```

Include `owner` in the comparison (excluded by default):

```bash
databricks-schema generate-sql <catalog> ./schemas/ --include-metadata
```

### `validate`

Validate local schema files for structural integrity (no Databricks connection needed):

```bash
databricks-schema validate ./schemas/
```

### `diff-files`

Compare two local directories of schema files (no Databricks connection needed). Same output,
exit codes, and `--schema` / `--include-metadata` flags as `diff`:

```bash
databricks-schema diff-files ./schemas-prod/ ./schemas-test/
```

### `list-catalogs`

List all accessible catalogs:

```bash
databricks-schema list-catalogs
```

### `list-schemas`

List schemas in a catalog:

```bash
databricks-schema list-schemas <catalog>
```

## Python Library Usage

See also the [`examples/`](examples/) directory for standalone scripts.

```python
from pathlib import Path
from databricks_schema import CatalogExtractor, catalog_to_yaml, schema_from_yaml
from databricks_schema import diff_catalog_with_dir, diff_catalogs, diff_schemas, schema_diff_to_sql

# Extract using configured auth (max_workers controls parallel table extraction)
extractor = CatalogExtractor(max_workers=4)
catalog = extractor.extract_catalog("my_catalog", schema_filter=["main", "raw"])

# Include Unity Catalog tags (extra API calls per entity; excluded by default)
catalog = extractor.extract_catalog("my_catalog", include_tags=True)

# Include additional metadata (owner, storage_location)
catalog = extractor.extract_catalog("my_catalog", include_metadata=True)

# Serialise to YAML
yaml_text = catalog_to_yaml(catalog)

# Deserialise from YAML
schema = schema_from_yaml(open("schemas/main.yaml").read())
print(schema.tables[0].columns)

# Compare live catalog against local YAML files
result = diff_catalog_with_dir(catalog, Path("./schemas/"))
if result.has_changes:
    for schema_diff in result.schemas:
        print(schema_diff.name, schema_diff.status)

# Compare two live catalogs directly
prod_catalog = extractor.extract_catalog("prod_catalog")
result = diff_catalogs(live=catalog, stored=prod_catalog)

# Compare two Schema objects directly
stored = schema_from_yaml(open("schemas/main.yaml").read())
diff = diff_schemas(live=catalog.schemas[0], stored=stored)

# Generate SQL to bring live in line with stored
sql = schema_diff_to_sql("my_catalog", diff, stored_schema=stored, allow_drop=False)
print(sql)
```
