Metadata-Version: 2.5
Name: vibedata-dlt-databricks-notebook
Version: 0.2.0
Summary: VibeData dlt runtime — Databricks notebook execution unit (runs inside a Databricks notebook): a dlt provider for the Domain's Databricks secret scope, and a databricks destination that stages load files through the Files API. dbutils is provided by the Databricks runtime.
Author-email: Vibedata <eng@acceleratedata.ai>
License: MIT
Requires-Python: >=3.11
Requires-Dist: dlt[databricks]<1.31,>=1.30.0
Requires-Dist: vibedata-dlt-core>=0.11.0
Provides-Extra: dev
Requires-Dist: pytest-mock>=3.12; extra == 'dev'
Requires-Dist: pytest>=8.0; extra == 'dev'
Description-Content-Type: text/markdown

# vibedata-dlt-databricks-notebook

Databricks notebook execution unit for the VibeData dlt runtime — the installed distribution for Databricks domains running a dlt pipeline inside a Databricks notebook. `dbutils` is provided by the Databricks runtime, not a pip dependency.

```python
from vibedata.dlt.databricks import DatabricksSecretScopeProvider, databricks_files_api, setup_environment
import dlt

setup_environment()
dlt.config.register_provider(DatabricksSecretScopeProvider())
pipeline = dlt.pipeline(pipeline_name="<name>", destination=databricks_files_api(credentials={...}), dataset_name="<schema>")
```

- **Import order.** Import this package before anything imports `dlt`: on Databricks compute `import dlt` otherwise binds Databricks' own Lakeflow Declarative Pipelines module.
- **`setup_environment()`** makes the Domain's `ingestion/` (the notebook's `ingestion_dir` widget) the working directory and an import root, and caps load files below the SDK's single-upload limit.
- **`DatabricksSecretScopeProvider`** resolves dlt secrets from the Databricks secret scope that `[providers.databricks_secret_scope] scope` names in the committed `.dlt/config.toml`. Each key is the dotted dlt key, lowercased. It resolves nothing when no scope is configured.
- **`databricks_files_api`** is dlt's `databricks` destination with one change: it stages load files on the volume through the Files API instead of the SQL connector's `PUT`. For a catalog on Databricks-managed storage, `PUT` uploads to the region's storage gateway, whose name resolves to a refusing private address on serverless compute. dlt has no hook for the upload, so the class overrides dlt's private staging methods; raise the package's dlt cap only after re-checking them against the new release.
- **No `finalize()`.** The unit writes no audit tables yet, so a Databricks pipeline makes the one `setup_environment()` call; the other units' audit pair arrives with Databricks run history.

Install this package; it pulls in `vibedata-dlt-core` and `dlt[databricks]`.
