Metadata-Version: 2.4
Name: dagster-dex
Version: 0.6.4
Summary: Let dex read your Dagster asset graph the way it reads a dbt project: keys, joins, sources, semantics, drift
Author: David Anaya
License-Expression: Apache-2.0
Project-URL: Homepage, https://github.com/catincloud-labs/dagster-dex
Project-URL: Repository, https://github.com/catincloud-labs/dagster-dex
Project-URL: Issues, https://github.com/catincloud-labs/dagster-dex/issues
Project-URL: Changelog, https://github.com/catincloud-labs/dagster-dex/blob/main/CHANGELOG.md
Keywords: dagster,dex,dbt,data-quality,drift,project-format
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Database
Classifier: Topic :: Software Development :: Libraries :: Python Modules
Classifier: Typing :: Typed
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: pyyaml>=6.0
Provides-Extra: dagster
Requires-Dist: dagster>=1.13; extra == "dagster"
Provides-Extra: dex
Requires-Dist: exmergo-dex-core~=1.8; extra == "dex"
Provides-Extra: test
Requires-Dist: pytest>=8.4; extra == "test"
Dynamic: license-file

# dagster-dex

Let [dex](https://github.com/exmergo/dex) read your Dagster asset graph the way
it reads a dbt project: keys, joins, sources, semantics, drift.

**dex** reports where a warehouse has drifted from the structure a project
declares. It reads dbt projects natively. This package makes a Dagster asset
graph readable the same way, without the graph having to pretend to be a dbt
project first.

Copyright 2026 David Anaya. Apache-2.0.

```bash
pip install dagster-dex          # the core: no orchestrator, no engine
pip install 'dagster-dex[dex]'   # with the dex-core boundary
```

Or from source, for `main` ahead of the last release:

```bash
pip install 'git+https://github.com/catincloud-labs/dagster-dex'
pip install 'dagster-dex[dex] @ git+https://github.com/catincloud-labs/dagster-dex'
```

Python 3.10+, matching Dagster's own floor. The `[dex]` extra needs 3.11+,
because the engine does: on 3.10 you can still reduce a graph and write an
artifact for a machine that has the engine to read it.

## What it finds

`examples/walk_the_whole_loop.py` puts a small Dagster asset graph in front of a
local DuckDB warehouse, breaks the warehouse and one of the project's
definitions by hand, and lets dex find the drift through this package. It needs no cloud account, no credential and
nothing private. From the root of a clone of this repository:

```bash
uv run --no-project --with-editable . --with 'dagster>=1.13' \
  --with exmergo-dex-core==1.8.0 --with sqlglot==30.13.0 --with duckdb \
  python examples/walk_the_whole_loop.py
```

Some of the lines it prints:

<!-- excerpt: examples/walk_the_whole_loop.py, held to the run by scripts/check_readme_excerpt.py -->
```text
leg 5 ok          : 2 finding(s); dim_date impacts ['dim_date', 'fact_sales']
leg 7/8 ok        : wrote it, and the project gained [('dim_date', 'date_key')]
leg 11 ok         : dangling_source on raw.sales_events, declared_in sources/fact_sales.yml
leg 12 ok         : definition_changed, and sales.amount reported as uncheckable
```

- **Leg 5.** One duplicate row in each of `dim_date` and `fact_sales` breaks
  both keys, hence two findings, and the `dim_date` finding names `fact_sales`
  among the models it impacts. Nothing in the warehouse connects those two
  tables; the Dagster asset graph does.
- **Legs 7 and 8.** The `unique` test dex proposed for the broken key is written
  into the hand-written declaration YAML through this package, and reading the
  project back shows the key declared.
- **Leg 11.** A source table the project declares is dropped, and the finding
  names the file that declared it.
- **Leg 12.** A measure is redefined as `amount * 1.2`. dex reports
  `definition_changed`, which no inspection of the warehouse finds because
  nothing in the warehouse changed, and says the new expression no longer maps
  to a column it can check, rather than passing it.

CI runs this example on every pull request and refuses the change if any line
quoted above is not one the run printed.

## Pointing dex at your graph

Once the package is installed, name the format in dex's own config:

```yaml
# .dex/config.yml
project:
  format: dagster
  options:
    assets: my_project.definitions:all_assets
```

## What this is

A working sketch of a contract, not a finished integration. It exists to make a
design argument concrete enough to disagree with: an asset graph is a perfectly
good source of project truth, and reading one does not require it to pretend to
be a dbt project first.

One boundary, stated up front: this format describes **one orchestrated asset
graph**. It does not describe, and will not absorb, a sibling project in
another format - two formats each describing the same warehouse table is
exactly the duplication this package's design argument rejects. Where
two projects meet, the warehouse is the join: each describes what it builds,
and lineage across them is read from the tables, not from either project.

## The seam, in three tiers

| Tier | Adds | A format implements it when |
| --- | --- | --- |
| `ProjectSource` | `declarations()` | it can say what it declares |
| `FingerprintedProject` | `fingerprint()` | it can be a drift baseline |
| `EditableProject` | `propose_edits()` | its source of truth can receive an edit |

**Which tier you get depends on how the project was built, and that is the
point.** `DagsterProject` reaches tier 2. `EditableDagsterProject` reaches tier
3, and the factory returns one only when the project points at a declarations
directory on disk - somewhere an edit can land. Read one from an `artifact:`
alone and you get tier 2, because an artifact is a JSON file with no directory
behind it.

**The graph is no longer the only way to have one.** A `declarations:` directory
named beside an `artifact:` reaches tier 3 as well; see
[Two ways to name the project](#two-ways-to-name-the-project-and-the-reason-there-are-two)
for what that directory supersedes and why it has to.

The split is a class, not a flag, and it has to be. `EditableProject` is
`runtime_checkable`, so it matches on the method being present: put
`propose_edits` on the shared class and every project claims the write tier and
then refuses at call time. `isinstance(project, EditableProject)` is the truth
for each instance, so a caller finds out by asking rather than by receiving an
empty result that looks like success.

What can receive an edit is the **declarations**, never the models. The models
are a reduction of a running asset graph, whose source of truth is the code that
built it; the declared keys and joins are hand-written YAML that nothing
regenerates. Those are different claims about different files.

## Four model decisions worth arguing about

**A key is always a tuple of columns.** A single-column grain is the `n == 1`
case, not a different type. Modelling it the other way is what makes a composite
grain inexpressible until someone widens the type - and after widening, every
reader handles two shapes forever.

**There is no project directory.** A path is one format's way of locating
itself, not a property every project has.

**Freshness is a tri-state.** `FRESH` / `STALE` / `NOT_APPLICABLE`. A boolean
cannot tell "current" from "there is no such artifact here", so a format with
nothing to compile reports itself identically to one that just compiled.

**A required field is a refusal to invent.** `ExternalSource` requires a source
system because the consumer requires one; making it optional would only move the
moment of invention into the boundary code, where a fabricated value cannot be
told from a declared one. Where the consumer requires something a project may
genuinely lack, such as a file path, the field is optional here and the loss is
reported instead of filled in.

## External sources: the edge a graph cannot see

Everything in `models` is something the project builds. `ExternalSource` is
where it stops and depends on a warehouse object somebody else keeps honoring.

These **cannot be derived from the asset graph**. A model that reads a table in
its own SQL, because the source system offers no other interface, draws no
dependency edge, so the read is invisible to any amount of traversal. It is
genuinely extra information and has to be declared, keyed by the model that
reads it. A format that inferred sources from dependencies would find none and
look like it had checked.

They are not decorative on the consumer side either: they drive a
`dangling_source` finding and the set of tables a project is understood to
cover. A format that reports none silently narrows its own blast radius.

## Layout, and why the dependencies sit where they do

```text
model.py         the neutral model          stdlib only
protocol.py      the tiered seam            stdlib only
declarations.py  input parsing              + pyyaml
project.py       the reduction, the writer  stdlib only
conformance.py   the importable contract    + pytest
dex.py           the dex-core boundary      + exmergo-dex-core, lazily
```

The engine coupling is one file. Everything else, including the conformance
suite and the whole write path, runs with dex-core uninstalled, which is what
keeps the design from being shaped by whatever the engine happens to look like
today. `dex.py` imports **public API only**; `tests/test_dex_bridge.py` asserts
that with an AST walk rather than trusting it.

The write path is in `project.py` rather than at the boundary for that reason.
The conflict handshake is the one part of this package where getting it wrong
costs somebody their work rather than an inaccurate report, so it is asserted in
the step that runs with no engine installed - `tests/test_write_tier.py`.

## Running it

From the repository root:

```bash
# the core: no orchestrator, no engine
uv run --no-project --with-editable . \
    --with pytest==8.4.1 \
    python -m pytest tests \
    --ignore=tests/test_dex_bridge.py \
    --ignore=tests/test_upstream_contract.py

# with the engine, to exercise the boundary
uv run --no-project --with-editable . \
    --with pytest==8.4.1 --with exmergo-dex-core==1.8.0 --with sqlglot==30.13.0 \
    python -m pytest tests \
    --ignore=tests/test_upstream_contract.py
```

`--with-editable` is not optional: without it the package is never installed and
every test errors at collection on `No module named 'dagster_dex'`. Both run
in CI, and the first is a **control**: it is what holds
the engine coupling to one file, so an `exmergo_dex_core` import anywhere else
turns it red at collection.

The engine is pinned to `exmergo-dex-core==1.8.0` here and in CI, which is
**not** what the `[dex]` extra publishes. The extra is
`exmergo-dex-core~=1.8`, so consumers are not forced onto
one release; the exact pin is where the demonstration lives, because a
claim about what passed should name the version it passed against.

### Against dex-core's own contract

`tests/test_upstream_contract.py` runs this format against the conformance suite
**dex-core ships**, which is upstream's acceptance criterion for a second project
format (`exmergo/dex#144`). That suite ships in **v1.5.2**, released 2026-08-05,
via [`exmergo/dex#192`](https://github.com/exmergo/dex/pull/192) - our own PR, so
the criterion is now judged by upstream's released code rather than by a branch:

```bash
DEX_UPSTREAM_CONTRACT_REQUIRED=1 \
uv run --no-project --with-editable . \
    --with pytest==8.4.1 --with exmergo-dex-core==1.8.0 --with sqlglot==30.13.0 \
    python -m pytest tests/test_upstream_contract.py
```

`DEX_UPSTREAM_CONTRACT_REQUIRED=1` is what makes it a signal, and the release did
not change that. Without the variable the file skips itself when the contract is
unimportable, which is right at a developer's desk and wrong in CI, where an unrun
file and a passing file look the same. With it, a missing contract is a collection
error - which still matters against a release, because a yank or a bad bump lands
in exactly the same place a rewritten branch used to.

## Using the contract on another format

```python
from dagster_dex.conformance import FingerprintedProjectContract

class TestMyFormat(FingerprintedProjectContract):
    def make_project(self, declarations, semantics, sources):
        return MyProject(declaration_sources=declarations,
                         semantic_sources=semantics,
                         source_declarations=sources)
```

The assertions that matter most are behavioural: `declarations()` must not raise
on an absent, empty, or malformed project. That is the property a second
implementation is most likely to get wrong, and a suite checking only shapes
will pass it.

`make_project` took two arguments before external sources landed. The third
is a **breaking change to this contract**, taken deliberately while the suite has
no outside implementers rather than carried forever as an optional hook that
could be skipped - and a skipped assertion is not a passing one.

## Status

**Alpha, and the contract may move before 1.0.** The tiered seam is a proposal,
not a settled interface: it exists to be argued with, and the argument may change
it. Pin the minor if you depend on it.

[CHANGELOG.md](https://github.com/catincloud-labs/dagster-dex/blob/main/CHANGELOG.md) names what changed between releases and which
changes were breaking, which is the question this section cannot answer for a
consumer already on an older version.

**dex finds this format by name.** From dex-core 1.6.0 the engine resolves the
`exmergo_dex_core.projects` entry-point group, which this package registers in,
so a format can be named from `.dex/config.yml`, from a dotted
`mypkg.projects:my_project` path, or from that group, with `--project-format`
overriding on the CLI. The entry point names `dex:project_from_context`, which
takes a context and returns a `DexProject`, and a regression test asserts the
registration *and* that it loads to the callable. How it came to be registered
that way is in the changelog.

| Verified in CI, every commit | Where |
| --- | --- |
| The suite passes with the engine uninstalled | control step |
| The suite passes against the engine | boundary step |
| The format passes dex-core's own project contract | criterion step |
| The reduction works on real Dagster objects | reduction step |
| The annotations type-check at the floor | type check |
| The built wheel imports, declares its entry point, ships `py.typed` | release workflow |
| A reconcile proposal becomes a plan, is written through this format, and changes what the project declares | release workflow |
| A second plan is refused over a human's edit rather than written | release workflow |

Every row runs from a clean checkout with no access to anything private:
[CONTRIBUTING.md](https://github.com/catincloud-labs/dagster-dex/blob/main/CONTRIBUTING.md)
gives the commands, and the workflows under
`.github/workflows/` run the rest.

### Two ways to name the project, and the reason there are two

`assets:` reduces a live asset graph in the calling process. That is the honest
form, and its cost is dominated by **importing the code location** rather than by
the reduction - which is not something laziness or caching can reach, because a
host that builds a project per command never holds one long enough to amortize
it. How long that import takes is a property of your project rather than of this
package, so it is the thing to measure before choosing between the two forms:
time `dagster definitions list` against your own code location and you have the
number, on your machine, where it is true.

`artifact:` is the answer for a host that cannot pay that: the side that already
has the graph reduces it once and writes the result down, and the side that
answers requests reads it back.

```yaml
project:
  format: dagster
  options:
    artifact: project/my_project.json   # written by dagster_dex.artifact.dump
```

Exactly one of the two is required, and naming both is refused rather than
resolved: a snapshot and a live graph disagree by design. An artifact carries its
own semantics, its own sources and its own name, so `semantics:`, `sources:` and
`name:` are refused beside it rather than silently ignored - each would modify
nothing while reading as though the files on disk were live, which is the
ordinary shape of a config quietly a day out of date.

**A missing artifact is refused, not read as an empty project.** An empty
project is a valid one, so the tolerant reading would report a broken deploy as a
warehouse with nothing declared, quietly, and for as long as it lasted.

#### Reaching the write tier over the artifact transport

`declarations:` is the one option admitted beside `artifact:`, and it
**supersedes** the declaration text the artifact carries:

```yaml
project:
  format: dagster
  options:
    artifact: project/my_project.json
    declarations: declared            # the live source, and where edits land
```

The obvious reading of that option - the artifact still declares, and the
directory only says where an edit may land - cannot be built, and the reason is
worth knowing before proposing it again:

- An artifact keys its declarations by a **bare stem**, and must keep doing so.
  It has no directory to have come from, and inventing one would be fabricated
  provenance.
- An editing surface is checked segment-wise, so `declarations` admits
  `declarations/dim_date.yml` and refuses a bare `dim_date`. A bare stem is
  inside no surface.
- So an edit view built from the artifact's text is **empty**. Every edit pins
  against a file it believes absent, and every apply is then a conflict on a file
  that plainly exists - refused on every write, forever, which is worse than
  declining the tier.

The directory has to be read, so it is the declarations. That costs nothing the
artifact was protecting: `artifact:` exists because reducing a live graph imports
a code location, and hand-written YAML needs neither an import nor an
orchestrator. What the artifact still answers for is the graph.

Because the artifact's copy of that text loses, `notes()` says so - a mapping
that drops something without disclosing it is the failure `notes` exists to
prevent. A missing directory is refused rather than read as "nothing is
editable": a declined write tier is indistinguishable from a format that never
had one, so a typo would be silent.

**An empty directory is accepted, and the asymmetry is a contract rather than
an accident.** A consumer's provisioning order can legitimately create the
directory before its first content arrives, so first boot depends on empty
being admitted; the project reaches the write tier with an empty editing view,
and the view fills as files do. A missing directory means a wrong path; an
empty one means a true state. Tightening the empty arm into a refusal would be
a breaking change to that provisioning order, not a tidier default.

**The directory's parent becomes the project root, and that derivation is a
guarantee (stable at least until 1.0).** Keys are `<directory-name>/<file>`,
relative to the parent, and downstream tooling stores plans relative to that
root - so a plan made against one checkout applies against another only
because both derive the same root from the same configured path. A change here
relocates every stored plan's `project_dir`; it is pinned by a test in this
repository so the rule goes red where it lives.

### Writing one, from the side that has the graph

`dagster_dex.artifact.dump` writes the file. It is deliberately
orchestrator-free - the reduction reads asset definitions structurally - so the
scheduling around it is yours. This is what that looks like in Dagster, and it is
**user code**: copy it, do not import it.

```python
from datetime import datetime, timezone

import dagster as dg
from dagster_dex import DagsterProject
from dagster_dex.artifact import dump

@dg.op
def write_project_artifact(context) -> int:
    from my_project.definitions import all_assets   # deferred: see below

    project = DagsterProject.from_asset_graph(all_assets, name="my_project")
    declarations = project.declarations()
    dump(
        "/shared/project/my_project.json",
        name="my_project",
        models=declarations.models,
        generated_at=datetime.now(timezone.utc).isoformat(timespec="seconds"),
        declaration_sources=my_declaration_yaml,      # {filename: text}
    )
    context.log.info("wrote %s models", len(declarations.models))
    return len(declarations.models)

@dg.job
def write_project_artifact_job() -> None:
    write_project_artifact()

write_project_artifact_schedule = dg.ScheduleDefinition(
    job=write_project_artifact_job,
    cron_schedule="0 6 * * *",
    execution_timezone="UTC",
    default_status=dg.DefaultScheduleStatus.RUNNING,
)
```

Four things in that sketch are not incidental:

- **The asset import is deferred into the op body.** A module-scope import of
  the graph is a cycle if your `definitions.py` also imports this module, and a
  failure in it takes down the whole code location rather than one job.
- **`generated_at` is yours to supply.** The library has no clock on purpose: a
  default would make every artifact look freshly written, including one produced
  by a replay.
- **The path's parent must already exist.** `dump` will not create it - a
  missing parent is a configuration mistake, and creating it writes the artifact
  somewhere nobody reads.
- **Schedule it.** An artifact that regenerates when someone remembers is a
  stale artifact, and a stale project is a wrong drift report rather than an
  absent one.

**The sketch's mechanics are executed by CI** -
`examples/write_project_artifact.py` runs the same op body against a real
graph, executes the job in process, reads the artifact back, and constructs
the schedule, so the code above cannot silently rot. What CI deliberately does
not decide is the schedule *values*: scheduling is a decision, and a library
that made it for you would be wrong in every deployment that differs - which
is why the sketch stays here, with placeholders, as user code.

## Verifying a release

Each file on PyPI carries a PEP 740 attestation, which PyPI checks and shows:
it names this repository's `publish.yml`, run at the release's `v*` tag. From
the first release after `v0.6.2`, that tag is also signed by the maintainer, so
the chain ends at a public key rather than at repository settings you cannot
read. `v0.6.2` and every tag before it are unsigned and stay that way: a
signature added now would vouch for them after the fact, not at their release.

To check one, with git 2.34 or newer and OpenSSH 8.8 or newer, put the version
you installed in place of `vX.Y.Z`:

```bash
git clone https://github.com/catincloud-labs/dagster-dex
cd dagster-dex
cat .github/allowed_signers
curl -s https://api.github.com/users/catincloudlabs/ssh_signing_keys
git -c gpg.ssh.allowedSignersFile=.github/allowed_signers tag -v vX.Y.Z
```

- **The key comes before the file.** Every key in
  [`.github/allowed_signers`](https://github.com/catincloud-labs/dagster-dex/blob/main/.github/allowed_signers)
  has to be one GitHub lists as a signing key of the maintainer's account,
  `catincloudlabs`, which is what `curl` prints. The file travels in the same
  repository as the tag it vouches for, so on its own it proves nothing.
- **Then the tag.** `tag -v` prints the tag, then `Good "git" signature for`
  the principal in that file, and exits 0. Anything else (`no signature
  found`, `No principal matched`) means the tag is not one the maintainer
  signed.
- **Then two lines of what it printed.** `tag` has to name the version you
  asked for: a signature covers the name written inside the tag, so a tag
  signed under another name still verifies wherever it is moved. `object` is
  the commit the release was built from, the same commit its PyPI attestation
  names.
- **The release workflow runs the same checks first.** Its `build` job
  verifies the tag, and the name inside it, before it runs a test or builds
  anything, and a tag that fails stops the release before anything is
  uploaded.
