DaggerML

DaggerML makes a computation a durable research artifact. Its inputs, functions, results, execution boundaries, and provenance are recorded as a DAG, so a result can lead back to the computation that produced it and repeated work can reuse a known result.

It is a good fit when research needs repeatable derived results, reusable intermediate work, auditable inputs and outputs, or execution beyond one local Python process. It combines Python authoring with a CLI for project history, configuration, remotes, and runtime operations.

It is not a general workflow scheduler, a replacement for object storage, or a promise that arbitrary Python state can be replayed. Large data normally stays in external storage and appears in a DAG as a tracked artifact URI.

Start here

Follow the live course in order. Each page builds on the project and results created by the pages before it:

  1. Get started: install DaggerML and initialize a project.
  2. DAGs: create, query, trace, and commit nodes, then build a reusable Docker image.
  3. Funks: write containerized functions and inspect their execution DAGs and cache reuse.
  4. Dagclasses: package reusable parameters and funks behind a conventional entry point.

Use DaggerML

Continue with Projects, then follow the executable Use course through artifacts, execution, inspection, runtimes, and sharing.

Build integrations

Codecs, Adapters, and Executors form the executable extension course.

Contribute to DaggerML

Contributing explains local setup, testing, and the repository workflow. Co-located package READMEs provide architecture orientation.

Keep close at hand

References