Metadata-Version: 2.1
Name: workflow-canvas
Version: 0.6.1
Summary: Reproducible computational pipelines with containerized methods, content-addressed caching, run lineage, and a visual pipeline builder; compiles to Snakemake
Home-page: https://workflow-canvas.readthedocs.io
License: BSD-3-Clause
Keywords: workflow,pipeline,reproducibility,provenance,lineage,snakemake,dvc,containers,bioinformatics,scientific-computing
Author: Dante
Author-email: dantepoe@noboundssolutions.com
Requires-Python: >=3.12,<3.13
Classifier: Development Status :: 3 - Alpha
Classifier: Environment :: Console
Classifier: Environment :: Web Environment
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: License :: OSI Approved :: BSD License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering
Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
Classifier: Topic :: System :: Distributed Computing
Provides-Extra: azure
Provides-Extra: gcs
Provides-Extra: remotes
Provides-Extra: s3
Provides-Extra: ssh
Requires-Dist: axiom-annotations (>=0.1.0,<0.2.0)
Requires-Dist: dvc (>=3.67.1,<4.0.0)
Requires-Dist: dvc-azure ; extra == "azure" or extra == "remotes"
Requires-Dist: dvc-gs ; extra == "gcs" or extra == "remotes"
Requires-Dist: dvc-s3 ; extra == "s3" or extra == "remotes"
Requires-Dist: dvc-ssh ; extra == "ssh" or extra == "remotes"
Requires-Dist: fastapi (>=0.110)
Requires-Dist: numpy (>=1.24)
Requires-Dist: pandas (>=2.0)
Requires-Dist: psutil (>=5.9)
Requires-Dist: pyarrow (>=14.0)
Requires-Dist: pydantic (>=2.0.0)
Requires-Dist: snakemake (>=8.0)
Requires-Dist: sqlmodel (>=0.0.14)
Requires-Dist: uvicorn[standard] (>=0.27)
Requires-Dist: wfc-client (>=0.2.0,<0.3.0)
Project-URL: Changelog, https://github.com/ddpoe/workflow-canvas/blob/main/CHANGELOG.md
Project-URL: Documentation, https://workflow-canvas.readthedocs.io
Project-URL: Issues, https://github.com/ddpoe/workflow-canvas/issues
Project-URL: Repository, https://github.com/ddpoe/workflow-canvas
Description-Content-Type: text/markdown

# Workflow Canvas

[![PyPI](https://img.shields.io/pypi/v/workflow-canvas)](https://pypi.org/project/workflow-canvas/)
[![CI](https://github.com/ddpoe/workflow-canvas/actions/workflows/ci.yml/badge.svg)](https://github.com/ddpoe/workflow-canvas/actions/workflows/ci.yml)
[![Documentation](https://readthedocs.org/projects/workflow-canvas/badge/?version=latest)](https://workflow-canvas.readthedocs.io)
[![License: BSD-3-Clause](https://img.shields.io/badge/license-BSD--3--Clause-blue)](LICENSE)

Workflow Canvas (`wfc`) is a Python tool for reproducible computational pipelines,
managed from the command line or a browser-based visual Canvas. It is built for
researchers and computational scientists who need to know exactly which code,
container, parameters, and inputs produced every result.

![The Canvas Builder with the demo pipeline wired: three samples fanned across five methods](https://raw.githubusercontent.com/ddpoe/workflow-canvas/main/userdocs/guide/_images/builder-demo-pipeline.png)

Workflow Canvas tracks your methods, their input/output contracts, run lineage, and
content-addressed caching in a local SQLite database, then compiles pipelines to
[Snakemake](https://snakemake.readthedocs.io/) Snakefiles so Snakemake handles
execution, parallelism, and dependency resolution. Every method runs inside a
container, and every output is stored in a content-addressed archive (DVC), so
results are reproducible and re-running an unchanged pipeline does no duplicate work.

## Key Features

| Feature | What it does |
|---|---|
| **Core pipeline** | Register modules, methods, and samples; build a pipeline in the Canvas; generate and execute a Snakefile (`wfc run-pipeline`). |
| **Contracts** | Two-tier validation (module + method) declared in `method.yaml` — wiring errors and missing outputs are caught before a run starts. |
| **Caching** | A clean-git run gate plus a SHA-256 cache key over code, inputs, parameters, and the resolved container image — unchanged steps are skipped automatically; changing one method re-runs only what it touched. |
| **Lineage** | Every run is recorded in SQLite; trace any output back through its full DAG ancestry (cache hits appear as audit rows). |
| **Environments** | Methods run in containers built from a `pixi`, `conda`, or bring-your-own (`byo`) backend via `wfc register-env`; image digests are pinned in `.wfc/envs.json`. |
| **Parameter sweeps & fan-out** | Sweep parameters, fan out across variants and per-sample overrides, and reuse values with pipeline variables. |
| **Canvas** | A browser-based UI (`wfc canvas`) with three views: a drag-and-drop Builder, a run-History viewer with lineage and artifacts, and a Registry browser. |

## Requirements

- **Python 3.12** — required (the config parser uses stdlib `tomllib`).
- **Docker** — required. Methods always run inside a container; nothing runs without a working Docker installation. `wfc init` pre-flights it and `wfc doctor` checks it.
- **Git** — required, but **local only**. wfc records a commit per run and refuses to run on uncommitted code. No GitHub account, login, or network is involved; wfc never pushes.

DVC and Snakemake ship with Workflow Canvas — you do not install them separately.

## Installation

```bash
# Installs the wfc CLI, plus bundled DVC and Snakemake
pip install workflow-canvas

# Verify
wfc --help
```

## Quick Start

New to Workflow Canvas? Start with the **Getting Started** tutorial in the
[documentation](#documentation).

For the fastest first look, `wfc demo` populates a freshly initialized project
with a complete runnable example pipeline and opens it in the Canvas
(`wfc demo --remove` takes it back out):

```bash
wfc init --dir ./demo_project && cd demo_project && wfc demo
```

The steps below show the same setup done manually with your own code and data:

```bash
# 1. Initialize a project (guided wizard: scaffolds dirs, configures an output
#    archive, sets up a local git repo, and pre-flights Docker)
wfc init --dir ./my_project
cd my_project

# Optional: check that the project is ready to run (git, archive, Docker)
wfc doctor

# 2. Build and register the container environment your methods will run in.
#    Docs → "Registering an Environment"
wfc register-env analysis conda:analysis

# 3. Register a module (from a module.yaml, recommended).
#    Docs → "Registration"
wfc register-module --name cell_analysis --module-dir modules/cell_analysis

# 4. Register a method (AST-scans the script for its inputs, outputs, and params).
#    Docs → "Authoring a Method Script" and "Writing Contracts"
wfc register-method modules/cell_analysis/preprocess --module cell_analysis

# 5. Register input data (content-hashed into the DVC store).
wfc register-sample --name CFPAC_ERKi --source /data/raw/cfpac_erki.csv

# 6. Build a pipeline in the Canvas, then run it.
#    Docs → "Running & Inspecting Results"
wfc run-pipeline --cores 4
```

## Visual Canvas

```bash
wfc canvas        # serves the UI at http://127.0.0.1:8500 by default
```

Use the **Builder** to assemble pipelines by dragging and wiring method nodes, the
**History** view to browse past runs (lineage chains, params, metrics, artifacts),
and the **Registry** to inspect everything registered in the project. See the
**Canvas** how-to in the documentation for a walkthrough.

## Project Structure

```
my_project/
  .wfc/
    wf-canvas.toml        # Project config (DB path, env root, DVC archive URL)
    wfc.db                # SQLite database — single source of truth (not in git)
    envs.json             # Registered container environments (tracked in git)
  modules/                # Nested module/method hierarchies (module.yaml + methods)
  methods/                # Flat standalone methods
  data/samples/           # Ephemeral workspace; samples restored lazily at run time
  .runs/
    sentinels/            # Zero-byte Snakemake DAG-wiring sentinels
    {run_id}/             # Transient staging; bytes are moved to the DVC archive
  .dvc/cache/             # Local content-addressed cache
  .gitignore
```

Output bytes are not kept in `.runs/`; they live in the content-addressed DVC
archive (default `~/.wfc/archives/<project>`, outside the repo), indexed by
`.wfc/wfc.db`. Back up both `.wfc/` and the archive folder to keep results recoverable.

## Documentation

Full user documentation is published at
[workflow-canvas.readthedocs.io](https://workflow-canvas.readthedocs.io) and covers:

- **Tutorials** — Getting Started, Exploring the Demo, Authoring a Method Script, Registering an Environment, Writing Contracts
- **How-to** — Registration, the Canvas, Running & Inspecting Results, Sweeping Parameters & Fanning Out
- **Reference** — CLI Reference, Python API (`wfc-client`), `method.yaml` schema, `wf-canvas.toml`
- **Explanation** — How a Run Executes, Caching & Reproducibility, Project Anatomy, Storage & Provenance

## Contributing

Bug reports, questions and pull requests are welcome. See
[CONTRIBUTING.md](CONTRIBUTING.md) for how to set up a development environment,
run the tests and build the docs, and the [Code of Conduct](CODE_OF_CONDUCT.md)
for how we work together.

## License

See [LICENSE](LICENSE).

