Sharing

Publish and inspect history

dml log | jq -e 'length > 0'
dml show | jq -e '.dags["course-execution"] and .dags["course-artifacts"]'
dml tag create course-v1 >/dev/null
dml tag list | jq -e '.[] | select(.name == "course-v1")'
dml config show | jq -e --arg root "$DML_REMOTE_ROOT" '.remote.root == $root'
dml push >/dev/null
dml push --revision @course-v1 >/dev/null
dml branch list --remote | jq -e '.[] | select(.name == "main")'

true
true
{
  "commit": "commit:ab3ff6b36aa2e02be3e59d902b08b1f7ccd251272b512a18b9ab2d077925dfbb",
  "name": "course-v1"
}
true
{
  "commit": "commit:ab3ff6b36aa2e02be3e59d902b08b1f7ccd251272b512a18b9ab2d077925dfbb",
  "name": "main"
}

dml log, show, and diff inspect history; branches can have one upstream, while tags do not. push publishes HEAD to the current attached branch’s upstream by default. push --revision BRANCH publishes that local branch to its upstream without checking it out; push --revision @NAME publishes a local tag once without replacing an existing remote tag. A commit ref or ancestor revision publishes to the current attached branch’s upstream. Endpoint listing is read-only: it does not fetch commits or update local tracking refs.

Reuse from satellite projects

The satellites are deliberately outside research-demo, so fetch, dependency, and shallow-history exercises cannot disturb the course project.

reuse_home="$DOCS_WORKSPACE_ROOT/course-reuse"
dep_home="$DOCS_WORKSPACE_ROOT/course-dependency"
shallow_home="$DOCS_WORKSPACE_ROOT/course-shallow"
mkdir "$reuse_home" "$dep_home" "$shallow_home"

(
  cd "$reuse_home"
  dml init
  dml config set remote.root "$DML_REMOTE_ROOT" >/dev/null
  dml fetch main >/dev/null
  dml dag checkout main course-execution --remote --name reused-execution
  dml show | jq -e '.dags["reused-execution"]'
)

(
  cd "$dep_home"
  dml init
  dml dep add tutorial "$DML_REMOTE_ROOT"
  dml fetch --dep tutorial main >/dev/null
  dml dag checkout main course-artifacts --dep tutorial --name dependency-artifacts
  dml show | jq -e '.dags["dependency-artifacts"]'
)

(
  cd "$shallow_home"
  dml --remote-root "$DML_REMOTE_ROOT" clone main --depth 1 --project-home .
  dml show | jq -e '.dags["course-execution"]'
  dml fetch --unshallow main >/dev/null
)

{"ahead":null,"behind":null,"branch":"main","branches":[],"commit":null,"mode":"attached","num_indexes":0,"upstream":null}
commit:cea7821041c36822754451717fb9d6475292f85dd0a19b8011656e631acb4738
"dag:8257f6ba072dd2a94d0be446688848cd0a54e5fe4ceb6cde07f73b9598773611"
{"ahead":null,"behind":null,"branch":"main","branches":[],"commit":null,"mode":"attached","num_indexes":0,"upstream":null}
tutorial
commit:3c2260f9c4c167a0586b889e212fc84bf7f2e7e5a3c030c733773ca6603aaf2e
"dag:bd8e5e5b71213031bbd1f7017bbc10f8fe90706d29ccd2a1a5f4dc8cd76d65ce"
{"ahead":0,"behind":0,"branch":"main","branches":["main"],"commit":"commit:ab3ff6b36aa2e02be3e59d902b08b1f7ccd251272b512a18b9ab2d077925dfbb","mode":"attached","num_indexes":0,"upstream":"main"}
"dag:8257f6ba072dd2a94d0be446688848cd0a54e5fe4ceb6cde07f73b9598773611"

dml.load("name") reads a committed DAG and dag.require("name") imports one into new work. A shallow clone contains the selected commit’s complete tree, DAGs, nodes, data, and imports; only older parents may be unavailable. Use a larger fetch --depth N to deepen history or fetch --unshallow to retrieve it all. Dependencies are import-only endpoints, not the project’s cache or execution remote. A normal push can advance an observed shallow branch; creating a remote branch or forcing publication requires complete ancestry.

show preserves the current snapshot at a shallow boundary and reports diff: null when its parent is unavailable. diff itself requires the parent; deepen the clone to compare changes.

Migrate an existing remote

Migration is permanently restricted to the previous version → current version only. If you are N versions behind, install each intermediate version in order and run its migration tool. Complete every migration step; the latest tool does not support skipping versions or migrating directly from older ones.

An older individual-object remote must be converted before use with this client. Use a new, empty S3 prefix. Stop all source writers, workers, and garbage collection until conversion and configuration cutover are complete. Execution and adapter records are copied unchanged, regardless of lifecycle or cleanup status; migration does not invoke adapters or perform their cleanup.

python -m daggerml._core.migrate_remote \
  --source s3://YOUR-BUCKET/old-root \
  --destination s3://YOUR-BUCKET/new-root --dry-run

python -m daggerml._core.migrate_remote \
  --source s3://YOUR-BUCKET/old-root \
  --destination s3://YOUR-BUCKET/new-root --quiescent

dml config set remote.root s3://YOUR-BUCKET/new-root

Use normal AWS credentials. The migration verifies the converted graphs before making the destination usable; it never changes or deletes the source. Switch every participating client and worker to the new root after success. A failed attempt leaves a staging prefix; use another empty destination for a retry. Dry-run checks metadata and prerequisites without copying or validating all data.

Embedded artifact URIs are preserved, not rewritten. Even though auxiliary files are copied, old URIs can still point to the source: retain that data while those artifacts are referenced. Remote graph garbage collection likewise requires quiescent writers and does not collect external artifact bytes.