Metadata-Version: 2.4
Name: databricks360
Version: 0.15.2
Summary: Install Databricks course lab environments: notebooks, catalogs, datasets and governance objects.
Project-URL: Homepage, https://github.com/databrickslms/dbxdemos
Project-URL: Source, https://github.com/databrickslms/dbxdemos
Project-URL: Issues, https://github.com/databrickslms/dbxdemos/issues
Author: Lakehouse Academy
License: MIT
License-File: LICENSE
Keywords: databricks,genie,lakehouse,training,unity-catalog
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Education
Classifier: Programming Language :: Python :: 3
Classifier: Topic :: Education
Requires-Python: >=3.9
Requires-Dist: databricks-sdk>=0.38.0
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == 'dev'
Provides-Extra: documents
Requires-Dist: fpdf2>=2.7; extra == 'documents'
Description-Content-Type: text/markdown

# databricks360

Installs Databricks course lab environments — notebooks, catalogs, datasets and
governance objects — into your own workspace.

Modelled on `dbdemos`: you run it **inside a Databricks notebook**, so
`databricks-sdk` picks up the notebook's own identity. There is no host, token or
profile to configure.

## Install

Not on PyPI yet, so install from the repo. In a **Databricks notebook**:

```python
%pip install git+https://github.com/databrickslms/dbxdemos.git
dbutils.library.restartPython()
```

Once published, that becomes `%pip install databricks360`.

To pin a version, append a tag or commit:

```python
%pip install git+https://github.com/databrickslms/dbxdemos.git@v0.1.0
```

## Usage

```python
import databricks360 as academy

academy.list_courses()
academy.install('genie-agents')
```

The bare call uses the course's own naming — a single `genie_agent` schema in your
current catalog, with objects named `mfg_core_*`, `mfg_ref_*`, `mfg_staging_*`:

```
genie_agent.mfg_core_dim_date
genie_agent.mfg_core_fct_transactions
genie_agent.mfg_ref_documents
```

Override with `schema=`, `table_prefix=` or `catalog=` — see **Regulated
environments** below.

`install` writes the lab notebooks into your workspace and prints the run order.
Then you open them and run each in turn.

```
Installed 'genie-agents' → /Workspace/Users/you@corp.com/databricks360/genie-agents
  catalog: mfg    tier: small

  Run these in order:
    1. 01_catalog_and_schemas
    2. 02_dimensions
    3. 03_facts
```

## Cleaning up

When a cohort finishes, remove what the course installed:

```python
academy.cleanup('genie-agents')                # show what would go
academy.cleanup('genie-agents', confirm=True)  # remove it
```

Dry run by default, and `confirm` is keyword-only so it cannot be passed by
accident. It removes the schema and its objects, the documents volume, the Genie
Agents and the notebooks — and only what the course created. An agent you renamed,
or a table someone else left in the lab schema, is reported and left alone.

Notebook `100_cleanup` does the same thing, with the destructive call commented
out so that running every cell top to bottom does not delete your lab.

If you installed with non-default arguments, pass the same ones back:

```python
academy.cleanup('genie-agents', catalog='training', confirm=True)
academy.cleanup('genie-agents', schema='large_tier', title_suffix='(large tier)', confirm=True)
```

## Documents

Module 3's Agent-mode exercises read unstructured files alongside the tables:

```python
academy.create_documents('genie-agents')
```

Forty Meridian documents — investment committee memos, advisor call notes and
client complaint resolutions — written into the course's Unity Catalog volume.
Each restates a planted flaw in the words someone would use in a meeting, so
answering a question means reconciling prose against numbers rather than reading
one or the other. PDFs where `fpdf` is installed, plain text otherwise.

## Genie Agents

Courses that need Genie Agents ship them as `serialized_space` definitions, so an
agent is never built by hand:

```python
academy.create_agents('genie-agents', dry_run=True)   # check, create nothing
academy.create_agents('genie-agents')                 # create them
```

Notebook `08_agents` does exactly this, so it is usually easier to run that in
order with the rest. Object names come from the same layout as the notebooks, so
the agents follow whichever catalog and schema you installed into.

This is the one call that creates something without you running a notebook. It is
still cheap: an agent is metadata, and nothing touches a warehouse until someone
asks it a question. It refuses to create an agent whose tables do not exist yet,
because an agent pointed at a missing table fails on every question and the
failure looks like a Genie problem rather than a notebook you skipped.

### Options

```python
academy.install(
    'genie-agents',
    path='/Workspace/Shared/labs',   # default: your home folder
    catalog='training_v2',           # default: the course's own catalog
    tier='large',                    # default: 'small'
    overwrite=True,                  # replace existing notebooks
)
```

## Why it does not run the notebooks for you

`dbdemos` starts a job and loads the data on your behalf. This deliberately does
not. Generating the data is the substance of Module 0 — the point is to watch a
warehouse chew through 20M rows and see the flaws appear, not to have a finished
catalog materialise. It also means nothing consumes your DBUs without you asking.

## Regulated environments

A locked-down Unity Catalog is normal in a governed organisation, so **the default
assumes the least privilege that can still work**: the lab lands in whatever
`current_catalog()` returns, and nothing is created above schema level.

Notebook 01 opens by showing you exactly where that is:

```sql
SELECT current_catalog() AS default_catalog, current_schema() AS default_schema;
```

### The four shapes

```python
# 1. Default — current catalog, schemas core / ref / staging.
#    No CREATE CATALOG attempted. Table names are two-level (core.dim_date),
#    so SQL resolves the catalog itself.
academy.install('genie-agents')

# 2. A specific catalog you were granted. Adds USE CATALOG, still creates no catalog.
academy.install('genie-agents', catalog='main')

# 3. Create a catalog — opt in, only if you hold the privilege.
academy.install('genie-agents', catalog='mfg', create_catalog=True)

# 4. One schema you already own. Everything lands there.
academy.install('genie-agents', schema='training_you')
academy.install('genie-agents', catalog='main', schema='training_you')
```

`create_catalog` is **off** by default and raises if you pass it without a catalog
name — `current_catalog()` exists by definition, so there would be nothing to create.

Passing `schema` assumes you cannot create that schema either, since that is the
reason to reach for it. Override with `create_schema=True` if you can.

### Which of these have actually been run

The spoiler sweep renders all six shapes at build time, but rendering is not running.
These have been executed end to end against a real workspace:

| Layout | Executed |
|---|---|
| Course default — one schema, `mfg_` prefix | all nine notebooks |
| Multi-schema — `core` / `ref` / `staging`, no prefix | all nine notebooks |
| Single schema, named catalog, no prefix | schemas + dimensions |
| `create_catalog=True` | catalog, schemas + dimensions |

`scripts/run_lab.py` is what does it — it executes a notebook's SQL against a workspace and
reports the first failure in each file, which is how six bugs in `06_curated` and
`07_metric_view` were found after they had shipped.

### One schema shared with other content

If your schema also holds other things, prefix the object names so their
provenance is visible:

```python
academy.install('genie-agents', schema='genie_agent', table_prefix='mfg_')
```

The logical group folds into the name:

| Default | With `table_prefix='mfg_'` |
|---|---|
| `core.dim_date` | `genie_agent.mfg_core_dim_date` |
| `ref.documents` | `genie_agent.mfg_ref_documents` |
| `staging.fct_txn_legacy` | `genie_agent.mfg_staging_fct_txn_legacy` |

The prefix names objects, never the schema. `table_prefix` requires `schema` —
without one, the schemas already namespace the objects and a prefix would just be
noise.

> **Worth weighing before you use it.** Genie reads table and column names as
> context, and Module 7 of the course is partly about *removing* noise so an agent
> has less to wade through. `mfg_core_dim_date` is measurably noisier than
> `dim_date`. Use a prefix when the schema is genuinely shared and provenance
> matters more; skip it when the schema is yours.

### No CREATE VOLUME either

```python
academy.install('genie-agents', schema='training_you', create_volume=False)
```

The volume only serves Agent mode over unstructured files (Modules 3 and 16).
Skipping it costs those exercises and nothing else, and the notebook says so.

### Why the single-schema case renames nothing

`core` holds the dimensions and facts, `ref` holds only the documents volume, and
`staging` holds only the two decoy tables. The three sets of object names do not
overlap, so collapsing them into one schema is safe — and notebooks 02 onward
resolve their own paths, so they need no changes whichever layout you chose.

### What you get back

`install` reports the resolved layout and anything it declined to create, so a
restricted install is visible rather than quietly partial:

```
Installed 'genie-agents' → /Workspace/Users/you@corp.com/databricks360/genie-agents
  tier: small
  single schema: current_catalog().training_you
  skipping: CREATE SCHEMA, CREATE VOLUME
```

> **Unity Catalog is required.** Genie Agents read Unity Catalog objects, so
> `hive_metastore` will not work as the target catalog.

## Tiers

| Tier | Transactions | Use |
|---|---|---|
| `small` | 20M | default. Every module except 13 |
| `large` | 900M | Module 13 (latency) only. Left unclustered on purpose |

Start small. The large tier exists because you cannot measure query latency on a
toy dataset, and nowhere else needs it.

## Adding a course

Each course is a subpackage under `databricks360/courses/`:

```
databricks360/courses/<course_id>/
  __init__.py
  manifest.json      # title, default catalog, tiers, notebooks in run order
  *.sql              # synced from content/courses/<id>/assets/lab/
```

Placeholders available in the SQL: `{{CATALOG}}`, plus anything declared under a
tier's `values` (currently `{{TXN_COUNT}}`). An unresolved placeholder raises rather
than silently rendering empty.

Dataset documentation lives in [`docs/`](docs/).

## Publishing

Releases go to PyPI via **Trusted Publishing** — GitHub Actions authenticates to
PyPI with a short-lived OIDC identity, so no API token exists in repo secrets or on
anyone's laptop. A leaked token is the usual way a package supply chain gets
compromised; the safest token is one that was never created.

### One-time PyPI setup

1. Sign in at [pypi.org](https://pypi.org) → **Your account → Publishing**
2. Under *Add a new pending publisher*, choose **GitHub** and enter exactly:

   | Field | Value |
   |---|---|
   | PyPI Project Name | `databricks360` |
   | Owner | `databrickslms` |
   | Repository name | `dbxdemos` |
   | Workflow name | `publish.yml` |
   | Environment name | `pypi` |

3. In GitHub → **Settings → Environments → New environment** → name it `pypi`.
   Add yourself as a required reviewer if you want to approve each release.

"Pending" publisher is correct — the project does not exist on PyPI yet, and the
first successful run creates it.

### Cutting a release

```bash
# bump version in pyproject.toml, commit, then:
git tag v0.1.0
git push origin v0.1.0
```

The workflow runs the tests, builds, checks the tag matches `pyproject.toml`, and
publishes. A mismatched tag fails before anything reaches the index — versions on
PyPI are immutable, so a wrong number cannot be taken back, only yanked.

### Publishing by hand instead

```bash
python -m pip install build twine
python -m build
twine check dist/*
twine upload dist/*          # prompts for an API token
```

Test it against TestPyPI first if you want a dry run:
`twine upload --repository testpypi dist/*`.

## Tests

```bash
python3 run_tests.py
```

Eleven tests, no workspace required: manifest loading, notebook cell structure,
catalog substitution, tier switching, unresolved-placeholder detection, and a
`dry_run` install. It also asserts the flaw-teaching column comments survive into
the generated notebooks — those comments are the curriculum, so losing them in
rendering would be a silent failure.
