Metadata-Version: 2.4
Name: tablevalidator-databricks
Version: 0.1.4
Summary: Generates a widget-driven Databricks notebook UI on top of the table-validator package - no code required to run a comparison.
License: MIT
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Requires-Dist: typer>=0.12
Requires-Dist: table-validator>=0.1.19
Provides-Extra: dev
Requires-Dist: pytest>=7.0; extra == "dev"
Requires-Dist: ruff>=0.4; extra == "dev"
Requires-Dist: black>=24.0; extra == "dev"

# tablevalidator-databricks

A widget-driven Databricks notebook UI for [`table-validator`](https://pypi.org/project/table-validator/)
- generates a notebook with `dbutils.widgets` for picking source/target
tables and validation checks, so you can run a comparison without writing
any Python.

This package contains **no validation logic of its own**. It only writes a
notebook file whose cells call directly into `table-validator`'s own
`validate_tables()` API - the exact same engine the CLI (`tablevalidator`)
and the notebook-native Python API already use.

## Install

```bash
pip install tablevalidator-databricks
```

Installing this package also installs `table-validator` (a dependency), so
there's nothing else to install separately.

## Usage

```bash
tablevalidator-databricks init
```

Writes `./TableValidator.py` - a ready-to-use Databricks notebook. Options:

```bash
tablevalidator-databricks init --output MyValidation.py --mode full
```

- `--mode basic` - only the check-group Yes/No dropdowns and table pickers.
- `--mode full` (default) - also shows optional filter widgets (only/
  ignore columns, row filter, primary key). Every check defaults to "Yes".
- `--mode schema` - same widgets as `basic`, but the Row checks default to
  "No" - a lightweight schema-shape-only comparison out of the box (still
  changeable per-run).

> **Note on the check groups**: "Catalog & Schema" only verifies that
> catalogs/schemas/tables exist and match by name - it produces no
> per-table PASS/FAIL by itself, so running it alone reports every table
> as `SKIPPED`. The per-table verdict comes from the **Column** checks
> (names, types, nullability, statistics) and **Row** checks (row counts,
> row-level data).

Then, in your Databricks workspace: **Workspace -> Import**, format
"Source", and select the generated file. Databricks recognizes the
`# Databricks notebook source` header and `# COMMAND ----------` cell
markers and opens it as a real notebook.

## What the generated notebook does

1. Installs `table-validator` (`%pip install table-validator`).
2. Queries Unity Catalog (`SHOW CATALOGS` / `SHOW SCHEMAS IN ...` /
   `SHOW TABLES IN ...` via the notebook's own ambient Spark session) to
   populate Source/Target Catalog, Schema, and Table dropdowns.
3. Shows a Yes/No dropdown per check group (Catalog & Schema / Column /
   Row - the three independently-selectable check groups the engine
   actually supports) plus, in `full` mode, optional text widgets for
   column filtering, a row filter, and a primary key.
4. On run, reads every widget and calls
   `validate_tables(source, target, enabled_validations=..., ...)`,
   then prints a summary and displays the Table Validation sheet.

**Cascading dropdowns**: Databricks widgets don't auto-refresh when an
upstream selection changes. After changing Source/Target Catalog or
Schema, re-run the notebook's "Create / refresh widgets" cell once to
repopulate the dropdowns below it before running the rest of the notebook.
This is a Databricks widget limitation, not a bug in the generated
notebook. If you skip this and run validation anyway with a stale Schema/
Table selection, the notebook fails with a clear error telling you which
cell to re-run, rather than silently comparing the wrong table.

**Can't find a name in a dropdown?** Every Catalog/Schema/Table dropdown
has a matching "(manual override)" text widget next to it. Type the exact
name there and it's used instead of the dropdown - useful if something
isn't showing up (a permissions lag, sort order, anything). It's used
as-is, not checked against the live catalog listing. Leave every override
blank to just use the dropdowns as normal.

**Validate every table in a schema, not just one**: pick
"(all tables in schema)" as the Table dropdown (Source and/or Target
independently) to compare every identically-named table in that schema in
one run - the same schema-wide sweep `validate_tables()` itself supports.
A primary key can't be set in this mode (leave the Primary Key widget
blank). Comparing every schema in a catalog isn't supported yet - Schema
must be a specific choice, even when Table is left as "(all ... in ...)".

## Why a separate package

`table-validator` stays focused on the actual comparison engine (CLI +
notebook API + report generation). This package is purely a Databricks-
specific UI layer on top of it - keeping the two independently versioned
and installable means CLI-only or code-only users never need Databricks-
specific tooling pulled in, and this package can add more Databricks UX
(e.g. richer widgets) without touching the core engine at all.

## Development

```bash
pip install -e ".[dev]"
pytest
```

## License

MIT - see the `table-validator` repository's [LICENSE](../table_validator/LICENSE).
