Metadata-Version: 2.5
Name: vibedata-dlt-databricks-notebook
Version: 0.3.0
Summary: VibeData dlt runtime — Databricks notebook execution unit (runs inside a Databricks notebook): dlt providers for the Domain's Unity Catalog secrets and (legacy) Databricks secret scope, and a databricks destination that stages load files through the Files API. dbutils is provided by the Databricks runtime.
Author-email: Vibedata <eng@acceleratedata.ai>
License: MIT
Requires-Python: >=3.11
Requires-Dist: dlt[databricks]<1.31,>=1.30.0
Requires-Dist: vibedata-dlt-core>=0.11.0
Provides-Extra: dev
Requires-Dist: pytest-mock>=3.12; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Description-Content-Type: text/markdown

# vibedata-dlt-databricks-notebook

Databricks notebook execution unit for the VibeData dlt runtime — the installed distribution for Databricks domains running a dlt pipeline inside a Databricks notebook. `dbutils` is provided by the Databricks runtime, not a pip dependency.

```python
from vibedata.dlt.databricks import DatabricksUcSecretsProvider, databricks_files_api, setup_environment
import dlt

setup_environment()
dlt.config.register_provider(DatabricksUcSecretsProvider())
pipeline = dlt.pipeline(pipeline_name="<name>", destination=databricks_files_api(), dataset_name="<schema>")
```

- **Import order.** Import this package before anything imports `dlt`: on Databricks compute `import dlt` otherwise binds Databricks' own Lakeflow Declarative Pipelines module.
- **`setup_environment()`** makes the Domain's `ingestion/` (the notebook's `ingestion_dir` widget) the working directory and an import root, and caps load files below the SDK's single-upload limit.
- **`DatabricksUcSecretsProvider`** resolves dlt secrets from the Unity Catalog schema that `[providers.databricks_uc_secrets] catalog` and `schema` name in the committed `.dlt/config.toml`. Each name is the dotted dlt key with every `.` turned into `__` (`sources.crm.credentials.password` → `sources__crm__credentials__password`); a key whose segment is empty, holds `__`, starts or ends with `_`, or is uppercase is refused rather than risk two keys sharing one secret. It lists the schema once, through the Unity Catalog secrets API, and reads values with `dbutils.secrets.get(catalog=, schema=, key=)`, which needs Databricks Runtime 17.3 LTS+ or serverless environment 4+ and READ SECRET on each secret. An unlisted name is missing; a listed one the run cannot read, as under a REFERENCE SECRET grant, raises. It resolves nothing when no schema is configured.
- **`DatabricksSecretScopeProvider`** (legacy) resolves dlt secrets from the Databricks secret scope that `[providers.databricks_secret_scope] scope` names in the committed `.dlt/config.toml`. Each key is the dotted dlt key, lowercased. It resolves nothing when no scope is configured.
- **`databricks_files_api`** is dlt's `databricks` destination with one change: it stages load files on the volume through the Files API instead of the SQL connector's `PUT`. For a catalog on Databricks-managed storage, `PUT` uploads to the region's storage gateway, whose name resolves to a refusing private address on serverless compute. dlt has no hook for the upload, so the class overrides dlt's private staging methods; raise the package's dlt cap only after re-checking them against the new release.
- **No `finalize()`.** The unit writes no audit tables yet, so a Databricks pipeline makes the one `setup_environment()` call; the other units' audit pair arrives with Databricks run history.

Install this package; it pulls in `vibedata-dlt-core` and `dlt[databricks]`.
