Metadata-Version: 2.4
Name: databricks-schema
Version: 0.8.0
Summary: Databricks Unity Catalog schema extractor
Project-URL: Homepage, https://github.com/frizzleqq/databricks-schema
Project-URL: Repository, https://github.com/frizzleqq/databricks-schema
Project-URL: Issues, https://github.com/frizzleqq/databricks-schema/issues
Project-URL: Changelog, https://github.com/frizzleqq/databricks-schema/releases
Author: frizzleqq
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: databricks,schema,sql,unity-catalog
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Requires-Python: >=3.11
Requires-Dist: databricks-sdk>=0.64.0
Requires-Dist: pydantic>=2.0
Requires-Dist: pyyaml>=6.0
Description-Content-Type: text/markdown

# databricks-schema

A CLI tool and Python library that uses the Databricks SDK to extract and diff Unity Catalog schemas in YAML format. It can also generate Databricks Spark SQL to apply schema changes across catalogs.

## Getting Started

Install from PyPI (requires Python 3.11+):

```bash
pip install databricks-schema
```

Configure your Databricks credentials (see [Authentication](#authentication) for all supported methods).

Then extract, diff, and generate SQL for your Unity Catalog schemas:

```bash
# 1. Find the catalog you want to snapshot
databricks-schema list-catalogs

# 2. Extract its schemas to YAML files (one file per schema)
databricks-schema extract prod_catalog --output-dir ./schemas/

# 3. Diff those files against a catalog (same or different)
databricks-schema diff test_catalog ./schemas/

# 4. Generate SQL to bring that catalog in line with the YAML files
databricks-schema generate-sql test_catalog ./schemas/ --output-dir ./migrations/
```

## Databricks Authentication

The tool uses the [Databricks SDK](https://github.com/databricks/databricks-sdk-py) for auth. Configure it via environment variables:

```bash
export DATABRICKS_HOST=https://<databricks-url>
export DATABRICKS_TOKEN=<your-personal-access-token>
```

Or use a [Databricks CLI profile](https://docs.databricks.com/dev-tools/cli/profiles.html) (`~/.databrickscfg`) — the SDK will pick it up automatically.

You can also pass credentials directly as flags using `--host` / `--token` below.

## Development

Requires Python 3.11+ and [uv](https://github.com/astral-sh/uv).

```bash
git clone https://github.com/frizzleqq/databricks-schema
cd databricks-schema
uv sync --all-groups  # includes pytest and ruff
```

```bash
# Run tests
uv run pytest

# Lint
uv run ruff check databricks_schema/ tests/

# Format
uv run ruff format databricks_schema/ tests/
```

## Agent Skill

This repo ships an [Agent Skill](https://code.claude.com/docs/en/skills) at
[`skills/databricks-schema-cli/SKILL.md`](skills/databricks-schema-cli/SKILL.md)
that teaches an AI coding agent (e.g. Claude Code) how to use the `databricks-schema` CLI to
explore, snapshot, diff, and generate migration SQL for a live Unity Catalog. Copy the skill into
your own project and it can use the CLI directly instead of writing ad-hoc Databricks SDK calls
or queries.

The skill assumes the `databricks-schema` command is on `PATH`. Install it as a standalone tool
with [uv](https://github.com/astral-sh/uv):

```bash
uv tool install databricks-schema
```

To use the skill in your own project without cloning this repo, download it directly into your
project's `.claude/skills/`:

```bash
mkdir -p .claude/skills/databricks-schema-cli
curl -o .claude/skills/databricks-schema-cli/SKILL.md \
  https://raw.githubusercontent.com/frizzleqq/databricks-schema/main/skills/databricks-schema-cli/SKILL.md
```

For agentic harnesses other than Claude Code that look under `.agents/skills/` instead:

```bash
mkdir -p .agents/skills/databricks-schema-cli
curl -o .agents/skills/databricks-schema-cli/SKILL.md \
  https://raw.githubusercontent.com/frizzleqq/databricks-schema/main/skills/databricks-schema-cli/SKILL.md
```

## Output Format

Each schema is written to `{output-dir}/{schema-name}.yaml` if `--output-dir` is specified. Fields with no value (null comments, empty tag dicts, empty FK lists) are omitted. Use `--format json` to write `.json` files with the same structure.

```yaml
name: main
comment: Main production schema
tags:
  env: prod
tables:
  - name: users
    table_type: MANAGED
    comment: User accounts
    tags:
      domain: identity
    columns:
      - name: id
        data_type: bigint
        nullable: false
        comment: Primary key
      - name: email
        data_type: string
      - name: org_id
        data_type: bigint
    primary_key:
      name: pk_users
      columns:
        - id
    foreign_keys:
      - name: fk_org
        columns:
          - org_id
        ref_schema: orgs
        ref_table: organizations
        ref_columns:
          - id
```

Without `--output-dir`, `extract` prints a single `Catalog` document to stdout instead — the same schema shape nested under a `schemas` list, with the catalog name as a top-level field:

```yaml
name: prod_catalog
schemas:
  - name: main
    comment: Main production schema
    tags:
      env: prod
    tables: [...]
```

## CLI Usage

```
databricks-schema [OPTIONS] COMMAND [ARGS]...
```

`extract`, `diff`, `generate-sql`, and `diff-files` print progress messages (e.g. `Extracting catalog 'X'...`) to stderr as they run. Pass `--quiet` / `-q` to suppress them;
results and errors are unaffected.

### `list-catalogs`

List all accessible catalogs:

```bash
databricks-schema list-catalogs
```

### `list-schemas`

List schemas in a catalog:

```bash
databricks-schema list-schemas <catalog>
```

Both print one name per line by default. Pass `--format json` / `-f json` (or `--format yaml`) to
get the names as a JSON array or YAML list instead.

### `extract`

Extract all schemas from a catalog to YAML files:

```bash
databricks-schema extract <catalog> --output-dir ./schemas/
```

Print to stdout instead of writing files (no `--output-dir`) — the catalog and its schemas are printed as a single `Catalog` document (`name` + `schemas: [...]`):

```bash
databricks-schema extract <catalog>
```

Extract specific schemas only:

```bash
databricks-schema extract <catalog> --schema main --schema raw --output-dir ./schemas/
```

Or extract a single schema or table by naming it directly, dotted onto the catalog (output keeps the same `Catalog`/`Schema` shape, just filtered down). This cannot be combined with `--schema`:

```bash
databricks-schema extract <catalog>.<schema>
databricks-schema extract <catalog>.<schema>.<table>
```

Include Unity Catalog tags in the output (extra API calls per entity; excluded by default):

```bash
databricks-schema extract <catalog> --output-dir ./schemas/ --include-tags
```

Include additional metadata (`owner`, `storage_location`) in the output:

```bash
databricks-schema extract <catalog> --output-dir ./schemas/ --include-metadata
```

Control the number of parallel workers (default: 4), useful for large schemas with many tags:

```bash
databricks-schema extract <catalog> --output-dir ./schemas/ --workers 8
```

### `diff`

Compare the live catalog against previously extracted schema files (format auto-detected from the directory — YAML or JSON, not mixed):

```bash
databricks-schema diff <catalog> ./schemas/
```

Exits with code `0` if no differences are found, `1` if there are — making it suitable for CI pipelines. Output example:

```
~ Schema: main [MODIFIED]
  ~ Table: users [MODIFIED]
    ~ Column: score [MODIFIED]
        data_type: 'int' -> 'double'
    + Column: phone [ADDED]
  + Table: events [ADDED]
- Schema: legacy [REMOVED]
```

Markers: `+` added in catalog, `-` removed from catalog, `~` modified.

Or compare two live catalogs directly, e.g. dev against prod. The second argument is treated as a directory if one exists at that path, otherwise as a catalog name:

```bash
databricks-schema diff dev_catalog prod_catalog
```

Compare specific schemas only:

```bash
databricks-schema diff <catalog> ./schemas/ --schema main --schema raw
```

Include Unity Catalog tags in the comparison (excluded by default):

```bash
databricks-schema diff <catalog> ./schemas/ --include-tags
```

Include `owner` in the comparison (excluded by default):

```bash
databricks-schema diff <catalog> ./schemas/ --include-metadata
```

Get the same comparison as structured JSON or YAML instead of the `+`/`-`/`~` tree (exit codes unchanged) — useful for scripts or agents:

```bash
databricks-schema diff <catalog> ./schemas/ --format json
databricks-schema diff <catalog> ./schemas/ --format yaml
```

### `generate-sql`

Generate Databricks Spark SQL statements to bring the live catalog in line with local schema files (format auto-detected, YAML or JSON, not mixed). Statements are printed to stdout by default:

```bash
databricks-schema generate-sql <catalog> ./schemas/
```

Write one `.sql` file per schema to a directory instead:

```bash
databricks-schema generate-sql <catalog> ./schemas/ --output-dir ./migrations/
```

Destructive statements (`DROP SCHEMA`, `DROP TABLE`, `DROP COLUMN`) are emitted as SQL comments by default. Pass `--allow-drop` to emit them as executable statements:

```bash
databricks-schema generate-sql <catalog> ./schemas/ --allow-drop
```

Filter to specific schemas:

```bash
databricks-schema generate-sql <catalog> ./schemas/ --schema main --schema raw
```

Include Unity Catalog tags in the comparison (excluded by default):

```bash
databricks-schema generate-sql <catalog> ./schemas/ --include-tags
```

Include `owner` in the comparison (excluded by default):

```bash
databricks-schema generate-sql <catalog> ./schemas/ --include-metadata
```

### `validate`

Validate local schema files for structural integrity (no Databricks connection needed):

```bash
databricks-schema validate ./schemas/
```

Pass `--format json` or `--format yaml` for an `{"issues": [...]}` document instead of `OK`/`ERROR:` text.

### `diff-files`

Compare two local directories of schema files (no Databricks connection needed). Same output,
exit codes, `--format json`/`yaml`, and `--schema` / `--include-metadata` flags as `diff`:

```bash
databricks-schema diff-files ./schemas-prod/ ./schemas-test/
```

### `json-schema`

Print the JSON Schema for one of the CLI's structured output shapes — no Databricks connection
needed. Useful before parsing `--format json`/`yaml` output (or `extract`'s output)
programmatically. Add `--format yaml` to render the schema itself as YAML:

```bash
databricks-schema json-schema catalog    # extract's stdout Catalog document
databricks-schema json-schema schema     # one per-schema extract file
databricks-schema json-schema diff       # diff / diff-files --format json/yaml
databricks-schema json-schema validate   # validate --format json/yaml
```

## Python Library Usage

See also the [`examples/`](examples/) directory for standalone scripts.

```python
from pathlib import Path
from databricks_schema import CatalogExtractor, catalog_to_yaml, schema_from_yaml
from databricks_schema import diff_catalog_with_dir, diff_catalogs, diff_schemas, schema_diff_to_sql

# Extract using configured auth (max_workers controls parallel table extraction)
extractor = CatalogExtractor(max_workers=4)
catalog = extractor.extract_catalog("my_catalog", schema_filter=["main", "raw"])

# Include Unity Catalog tags (extra API calls per entity; excluded by default)
catalog = extractor.extract_catalog("my_catalog", include_tags=True)

# Include additional metadata (owner, storage_location)
catalog = extractor.extract_catalog("my_catalog", include_metadata=True)

# Serialise to YAML
yaml_text = catalog_to_yaml(catalog)

# Deserialise from YAML
schema = schema_from_yaml(open("schemas/main.yaml").read())
print(schema.tables[0].columns)

# Compare live catalog against local YAML files
result = diff_catalog_with_dir(catalog, Path("./schemas/"))
if result.has_changes:
    for schema_diff in result.schemas:
        print(schema_diff.name, schema_diff.status)

# Compare two live catalogs directly
prod_catalog = extractor.extract_catalog("prod_catalog")
result = diff_catalogs(live=catalog, stored=prod_catalog)

# Compare two Schema objects directly
stored = schema_from_yaml(open("schemas/main.yaml").read())
diff = diff_schemas(live=catalog.schemas[0], stored=stored)

# Generate SQL to bring live in line with stored
sql = schema_diff_to_sql("my_catalog", diff, stored_schema=stored, allow_drop=False)
print(sql)
```
