DAGs

A DAG is a durable record of a computation. Create one first, then add work to it. Until commit(), dag is the live computation we are building.

import daggerml as dml
from daggerml.contrib.funks import docker_build

dag = dml.new("docs-image", message="build the tutorial image")

Put values in the DAG

dag.put() records a value and returns its node. Naming important nodes makes them easy to find later without changing their identity.

count = dag.put(3, name="count")
labels = dag.put(["fresh", "local", "seasonal"], name="labels")

print(count.value())
3
print(labels.value())
['fresh', 'local', 'seasonal']

The values look like ordinary Python values when materialized with .value(), but count and labels are nodes. They retain where they live in the DAG and can be used as inputs without discarding that provenance.

Access collections

Indexing a collection node creates another node. This matters when a downstream result uses only part of a large or structured input: the graph records which part was selected.

first_label = labels[0]

print(first_label.value())
fresh
print(first_label.ref)
Ref(node-fn:19f1809270bb832ea9b1b25315d5b222ea2a292a7c672cd6975c7788d39c7035)

Dictionary access works the same way, including nested access.

settings = dag.put(
    {"mode": "local", "limits": {"retries": 2}},
    name="settings",
)
mode = settings["mode"]
retries = settings["limits"]["retries"]

print(mode.value())
local
print(retries.value())
2

Nodes can be placed inside new collections. DaggerML records those relationships; it does not replace the nodes with unrelated copies of their Python values.

summary = dag.put(
    {
        "count": count,
        "first_label": first_label,
        "mode": mode,
        "retries": retries,
    },
    name="summary",
)

print(summary.value())
{'count': 3, 'first_label': 'fresh', 'mode': 'local', 'retries': 2}
print(dag.keys())
['count', 'labels', 'settings', 'summary']

Call a node

Functions are data in DaggerML. Putting docker_build in the DAG produces a callable node; calling that node records both the inputs and the produced image. For this live build, the harness quietly packages the current checkout into the same S3-backed context used by this tutorial’s Docker build.

context = dag.put(context_uri, name="context-tarball")
build = dag.put(docker_build, name="build-image")
image = build(
    context,
    ["-f", "./docs/Dockerfile"],
    name="image",
)

print(image.value().uri.startswith("s3://"))
True

image is the result we want another DAG to reuse, so we commit it last. A commit closes the live DAG and makes its named nodes, result, and provenance durable.

dag.commit(image)

The Funks lesson now can import this exact result with dag.require() and run code inside the image.

References