Metadata-Version: 2.1
Name: dscribe-dq
Version: 1.1.3
Summary: Run dScribe data quality rules against Databricks, MSSQL, or SAP HANA/Data Warehouse Cloud sources and write the results back to dScribe.
License: Proprietary
Requires-Python: >=3.10,<3.14
Classifier: License :: Other/Proprietary License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Dist: databricks-sdk (>=0.20.0)
Requires-Dist: databricks-sql-connector (>=4.0.0,<5.0.0)
Requires-Dist: databricks-sqlalchemy (>=2.0.9,<3.0.0)
Requires-Dist: great-expectations (==1.18.2)
Requires-Dist: hdbcli (>=2.29.25,<3.0.0)
Requires-Dist: pyodbc (>=5.0.0,<6.0.0)
Requires-Dist: pyspark (>=3.5.0,<5.0.0)
Requires-Dist: pyyaml (>=6.0.1,<7.0.0)
Requires-Dist: requests (>=2.31.0,<3.0.0)
Requires-Dist: sqlalchemy (>=2.0.0,<3.0.0)
Requires-Dist: sqlalchemy-hana (>=4.6.0,<5.0.0)
Description-Content-Type: text/markdown

# dscribe-dq

Run dScribe data quality rules against your Databricks or MSSQL databases and write the results back to dScribe.

> For library internals, architecture, and contributing, see [DEVELOPMENT.md](DEVELOPMENT.md).

## Prerequisites

- A dScribe account with at least one asset that has data quality rules defined in its ODCS spec
- Your dScribe API key (Settings → API keys in the dScribe UI)
- The asset UUID you want to validate
- Access to the database the rules target (Databricks or MSSQL)

## Installation

```bash
pip install dscribe-dq
```

## How it works

Initialize `DScribeDQ` once with your credentials, then call `run_validation` with a list of **post-processors**. Post-processors are small pipeline steps that run in order after validation — writing results back to dScribe, uploading failed-row CSVs, generating reports, etc.

```
DScribeDQ(credentials) → dq.run_validation(connector_config, postprocessors=[...])
```

The only post-processor built into the SDK is `write_back_to_dscribe`, which posts pass/fail results to dScribe so the asset's quality status updates in the UI.

## Quickstart

### 1. Find your asset ID and API key

In the dScribe UI, open the asset you want to validate. The asset ID is the UUID in the URL:

```
https://app.dscribe.cloud/catalog/assets/337eaa9e-47ed-4b37-a124-050d4932a520
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
```

Your API key is under **Settings → API keys**.

### 2. Run validation and write back to dScribe

```python
from dscribe_dq import DScribeDQ

dq = DScribeDQ(
    api_key="<your-api-key>",
    base_url="https://app.dscribe.cloud/catalog/api",
    asset_id="337eaa9e-47ed-4b37-a124-050d4932a520",
)

ctx = dq.run_validation(
    source_configs={
        # key must match the server id in the ODCS servers block
        "09bcc0f9-9d21-460d-9cb9-942b00e360bf": {
            "type": "databricks",
            "host": "adb-858283489583940.0.azuredatabricks.net",
            "client_id": "<client-id>",
            "client_secret": "<client-secret>",
            "tenant_id": "<tenant-id>",
            "http_path": "/sql/1.0/warehouses/abc123def456",
            "catalog": "hive_metastore",   # optional
            "schema": "default",           # optional
        }
    },
    postprocessors=[dq.write_back_to_dscribe()],
)
```

Each rule in dScribe gets a `lastCheckStatus` (`passed` or `failed`), `lastCheckTimestamp`, and failure metrics added to its `customProperties` after the pipeline runs.

## Databricks notebook

```python
%pip install dscribe-dq
```

```python
from dscribe_dq import DScribeDQ

dq = DScribeDQ(
    api_key=dbutils.secrets.get(scope="dscribe-dq", key="DSCRIBE_API_KEY"),
    base_url="https://app.dscribe.cloud/catalog/api",
    asset_id="<your-asset-uuid>",
)

ctx = dq.run_validation(
    source_configs={
        "<server-id>": {
            "type": "databricks",
            "host": spark.conf.get("spark.databricks.workspaceUrl"),
            "client_id": dbutils.secrets.get(scope="dscribe-dq", key="CLIENT_ID"),
            "client_secret": dbutils.secrets.get(scope="dscribe-dq", key="CLIENT_SECRET"),
            "tenant_id": dbutils.secrets.get(scope="dscribe-dq", key="TENANT_ID"),
            "http_path": "/sql/1.0/warehouses/<warehouse-id>",
        }
    },
    postprocessors=[dq.write_back_to_dscribe()],
)
```

> The `http_path` can be found in the Databricks UI under **SQL Warehouses → your warehouse → Connection details**.

## Connecting to MSSQL

**SQL Server authentication:**

```python
source_configs={
    "<server-id>": {
        "type": "sqlserver",
        "host": "your-server.database.windows.net",
        "database": "your-db",
        "schema": "SalesLT",
        "authentication": "SQL Server",
        "username": "your-user",
        "password": "your-password",
    }
}
```

**Entra ID (service principal) authentication:**

```python
source_configs={
    "<server-id>": {
        "type": "sqlserver",
        "host": "your-server.database.windows.net",
        "database": "your-db",
        "schema": "SalesLT",
        "authentication": "Entra ID",
        "tenant_id": "<tenant-id>",
        "client_id": "<client-id>",
        "client_secret": "<client-secret>",
    }
}
```

## Multiple sources in one call

If your asset has rules targeting both Databricks and MSSQL, pass both in `source_configs`. Rules are automatically grouped by source and run independently:

```python
source_configs={
    "<mssql-server-id>": {"type": "sqlserver", ...},
    "<databricks-server-id>": {"type": "databricks", ...},
}
```

## CI/CD usage (env vars)

For automated pipelines, set env vars and call the module-level `run_validation` directly:

```bash
DSCRIBE_API_KEY=...
DSCRIBE_ASSET_ID=...
DSCRIBE_BASE_URL=...
DATABRICKS_HOST=...
DATABRICKS_CLIENT_ID=...
DATABRICKS_CLIENT_SECRET=...
DATABRICKS_TENANT_ID=...
DATABRICKS_HTTP_PATH=...
```

```python
from dscribe_dq import run_validation

ctx = run_validation()  # reads all config from env vars
```

## Post-processors

Post-processors provide a way to perform additional processing after validation has completed. They receive the validation context (`ctx`) and can inspect or modify the results before the run finishes.

Pass them to `run_validation` in the order you want them to run:

```python
ctx = dq.run_validation(
    source_configs={...},
    postprocessors=[
        dq.write_back_to_dscribe(),
    ],
)
```

### Built-in: `write_back_to_dscribe`

`write_back_to_dscribe` is the built-in post-processor. It sends each rule's validation result back to dScribe so the latest status is visible in the dScribe UI.

```python
postprocessors=[
    dq.write_back_to_dscribe(),
]
```

For each rule, it sends:

- `lastCheckStatus`
- `lastCheckTimestamp`
- `lastCheckDetails` — a structured object with `success_percent`, `unexpected_percent`, `unexpected_count`, and `element_count`, generated automatically from `ctx.results`. You do not need to set these yourself.

`lastCheckDetails` can also carry two optional links, but only if an earlier post-processor sets them first:

| `lastCheckDetails` field | Set by a post-processor via | Used for                                     |
| ------------------------ | --------------------------- | -------------------------------------------- |
| `failed_rows_url`        | `res["csv_url"]`            | Link to the failing rows for a rule          |
| `report_url`             | `ctx.report_url`            | Link to a report for the full validation run |

If they are not set, `write_back_to_dscribe` still writes the validation results normally — add a custom post-processor before it that uploads the relevant files and sets these values, as shown in [Custom post-processors](#custom-post-processors).

### The `ctx` object

Every post-processor receives the same `DQContext`.

The most useful fields are:

| Field            | Description                                             |
| ---------------- | ------------------------------------------------------- |
| `ctx.results`    | Validation results, one entry per rule                  |
| `ctx.asset_id`   | ID of the asset being validated                         |
| `ctx.odcs`       | Parsed ODCS contract used for validation                |
| `ctx.timestamp`  | UTC timestamp for the start of the run                  |
| `ctx.audience`   | Contract audience, if one was selected                  |
| `ctx.profile`    | Profiling results when `enable_profiling=True`          |
| `ctx.report_url` | Optional report URL that can be set by a post-processor |

Each entry in `ctx.results` looks like this:

```python
{
    "rule_id": "675dbbcd-ee5c-45bc-85b7-34514beaea73",
    "expectation": "unexpected_rows_expectation",
    "column": "NATIONALITY_ID",
    "success": False,
    "result": {...},
    "failed_rows": [
        {"ID": "...", "NATIONALITY_ID": None},
        ...
    ],
}
```

For most custom post-processors, these fields are enough:

- `rule_id` — ID of the dScribe rule
- `column` — column checked by the rule, or `"table"` for table-level rules
- `success` — whether the rule passed
- `failed_rows` — collected rows that failed the rule

`result` contains the raw Great Expectations validation result and is available when you need more detailed validation information.

When profiling is enabled with `enable_profiling=True`, profiling data is available through `ctx.profile`:

```python
{
    "__table__": {
        "row_count": 2672186
    },
    "NATIONALITY_ID": {
        "type": "categorical",
        "null_count": 12,
        "distinct_count": 4,
        "value_counts": [...]
    },
    "AGE": {
        "type": "numeric",
        "null_count": 0,
        "distinct_count": 90,
        "mean": 41.2,
        "min": 0,
        "max": 120,
        "quantiles": [0, 22, 41, 60, 120]
    },
}
```

### Custom post-processors

A custom post-processor is simply a function that accepts `ctx`.

For example, to log the failing rows:

```python
def log_failed_rows(ctx):
    for res in ctx.results:
        if not res["success"]:
            for row in res["failed_rows"]:
                print(row)
```

This is also how you add your own reporting or storage. For example, a post-processor that uploads failing rows to your storage provider and sets `res["csv_url"]`, so `write_back_to_dscribe` picks it up as `failed_rows_url`:

```python
def upload_failed_rows(ctx):
    for res in ctx.results:
        if not res["success"] and res["failed_rows"]:
            res["csv_url"] = my_storage.upload_csv(res["rule_id"], res["failed_rows"])
```

Add both to `postprocessors`, before `write_back_to_dscribe`:

```python
ctx = dq.run_validation(
    source_configs={...},
    postprocessors=[
        log_failed_rows,
        upload_failed_rows,
        dq.write_back_to_dscribe(),
    ],
)
```

Post-processors run in the order they are listed, so `write_back_to_dscribe` must come last to see `csv_url` (or `ctx.report_url`) once an earlier post-processor sets it.

## Options

### `DScribeDQ.__init__`

| Parameter   | Type | Default | Description                                        |
| ----------- | ---- | ------- | -------------------------------------------------- |
| `api_key`   | str  | —       | dScribe API key (or set `DSCRIBE_API_KEY` env var) |
| `base_url`  | str  | —       | dScribe API base URL (or set `DSCRIBE_BASE_URL`)   |
| `asset_id`  | str  | —       | Asset UUID to validate (or set `DSCRIBE_ASSET_ID`) |
| `log_level` | str  | `INFO`  | `DEBUG`, `INFO`, `RESULT`, `WARNING`, `ERROR`      |

### `DScribeDQ.run_validation`

| Parameter             | Type | Default | Description                                                                                                                                                                             |
| --------------------- | ---- | ------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `source_configs`      | dict | `{}`    | Per-source connection settings keyed by server ID from ODCS spec                                                                                                                        |
| `connector_config`    | dict | `{}`    | Default connection settings used when no per-source config found                                                                                                                        |
| `collect_failed_rows` | bool | `True`  | Fetch the actual failing rows for each failed rule                                                                                                                                      |
| `max_failed_rows`     | int  | `1000`  | Max failing rows to fetch per rule. See [Known limitations](#known-limitations) — does not apply to every rule type.                                                                    |
| `enable_profiling`    | bool | `False` | Compute descriptive statistics (row count, null counts, etc.)                                                                                                                           |
| `postprocessors`      | list | `[]`    | Pipeline steps to run after validation in order                                                                                                                                         |
| `audience`            | str  | `None`  | Restrict to this audience's contract (or set `DSCRIBE_CONTRACT_AUDIENCE`). Defaults to the unlabeled "Default" contract if one exists, else whichever audience has the highest version. |
| `contract_version`    | str  | `None`  | Restrict to this exact contract version (or set `DSCRIBE_CONTRACT_VERSION`). Defaults to the latest version.                                                                            |

## Known limitations

- **`max_failed_rows` does not apply to all rule types.** It is respected for all built-in library rules (`nullValues`, `missingValues`, `duplicateValues` — including compound/multi-column rules — and `invalidValues` when defined using allowed or excluded values) and for SQL rules. It is not respected for regex-based `invalidValues` rules (`expect_column_values_to_match_regex` and `expect_column_values_to_not_match_regex`) or for expectations defined using a custom Great Expectations YAML block — for those, the number of returned failing rows is controlled by Great Expectations and capped at 200 regardless of `max_failed_rows`. This limit comes from Great Expectations' internal `MAX_RESULT_RECORDS` setting and cannot be configured by dscribe-dq.

## Supported ODCS metrics

| ODCS `metric`     | What it checks                                    |
| ----------------- | ------------------------------------------------- |
| `rowCount`        | Row count within expected bounds                  |
| `nullValues`      | No NULL values in a column                        |
| `missingValues`   | No missing/empty values in a column               |
| `duplicateValues` | All values in a column (or column set) are unique |
| `invalidValues`   | Values match an allowed list or regex pattern     |

## Environment variable reference

| Variable                    | Description                                                                                   |
| --------------------------- | --------------------------------------------------------------------------------------------- |
| `DSCRIBE_API_KEY`           | dScribe API key                                                                               |
| `DSCRIBE_ASSET_ID`          | Asset UUID to validate                                                                        |
| `DSCRIBE_BASE_URL`          | dScribe API base URL                                                                          |
| `DSCRIBE_CONTRACT_AUDIENCE` | Audience to restrict validation to (e.g. "sales", "finance")                                  |
| `DSCRIBE_CONTRACT_VERSION`  | Contract version to restrict validation to                                                    |
| `DATABRICKS_HOST`           | Databricks workspace hostname                                                                 |
| `DATABRICKS_CLIENT_ID`      | Azure AD service principal client ID                                                          |
| `DATABRICKS_CLIENT_SECRET`  | Azure AD service principal client secret                                                      |
| `DATABRICKS_TENANT_ID`      | Azure AD tenant ID                                                                            |
| `DATABRICKS_HTTP_PATH`      | SQL warehouse HTTP path                                                                       |
| `DATABRICKS_WAREHOUSE_ID`   | SQL warehouse ID (alternative to HTTP path)                                                   |
| `DATABRICKS_CATALOG`        | Default Unity Catalog catalog name                                                            |
| `DATABRICKS_SCHEMA`         | Default schema name                                                                           |
| `MSSQL_HOST`                | MSSQL server hostname                                                                         |
| `MSSQL_DATABASE`            | MSSQL database name                                                                           |
| `MSSQL_USER`                | SQL Server username                                                                           |
| `MSSQL_PASSWORD`            | SQL Server password                                                                           |
| `MSSQL_AUTH`                | `SQL Server` or `Entra ID`                                                                    |
| `MSSQL_TENANT_ID`           | Azure tenant ID (Entra ID auth only)                                                          |
| `MSSQL_CLIENT_ID`           | Azure client ID (Entra ID auth only)                                                          |
| `MSSQL_CLIENT_SECRET`       | Azure client secret (Entra ID auth only)                                                      |
| `HANA_HOST`                 | SAP HANA / Data Warehouse Cloud hostname                                                      |
| `HANA_PORT`                 | SQL port — `443` for SAP Data Warehouse Cloud, tenant SQL port (e.g. `3<instance>15`) on-prem |
| `HANA_DATABASE`             | Tenant database name (on-prem MDC routing; not needed for DWC)                                |
| `HANA_SCHEMA`               | Schema containing the target table                                                            |
| `HANA_TABLE`                | Table name to validate                                                                        |
| `HANA_USER`                 | HANA username                                                                                 |
| `HANA_PASSWORD`             | HANA password                                                                                 |
| `HANA_ENCRYPT`              | `true` or `false` (default `true`)                                                            |
| `HANA_VALIDATE_CERT`        | `true` or `false` (default `true`)                                                            |

