Metadata-Version: 2.4
Name: wofl
Version: 0.6.0
Summary: Composable task graphs for Python workflows with caching and remote execution.
License-Expression: CC0-1.0
Project-URL: Homepage, https://github.com/ast-al/wofl
Project-URL: Source, https://github.com/ast-al/wofl
Project-URL: Issues, https://github.com/ast-al/wofl/issues
Keywords: workflow,pipeline,dag,caching,remote,hpc
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Classifier: Programming Language :: Python :: 3.15
Classifier: Operating System :: POSIX
Requires-Python: >=3.11
Description-Content-Type: text/markdown
License-File: LICENSE.md
Provides-Extra: format
Requires-Dist: black; extra == "format"
Provides-Extra: dev
Requires-Dist: pytest; extra == "dev"
Requires-Dist: pytest-xdist; extra == "dev"
Requires-Dist: mypy; extra == "dev"
Dynamic: license-file

# wofl: composable tasks for Python workflows

wofl adds persistent caching, retries, resource limits, and local or remote
execution to Python functions. Build parallel pipelines by passing task results
to other tasks; use ordinary Python loops, conditionals, and `await` to control
the workflow. Each feature is available through a small, composable decorator.

For complete examples, see the [cache demo](examples/persistent_cache_demo.py),
[async task demo](examples/async_task_demo.py), and
[protein-alignment pipeline](examples/align_prots.py).

## Contents

- [Installation](#installation)
- [Complete example](#complete-example)
- [Command helpers](#command-helpers)
  - [`cmd.run`: run a command or script](#cmdrun-run-a-command-or-script)
  - [`cmd.popen`: connect and read commands](#cmdpopen-connect-and-read-commands)
- [Decorators](#decorators)
  - [`@retry`: retry transient failures](#retry-retry-transient-failures)
  - [`@cache`: persist results on disk](#cache-persist-results-on-disk)
  - [`@throttle`: limit concurrency and attempt rate](#throttle-limit-concurrency-and-attempt-rate)
- [Asynchronous pipelines](#asynchronous-pipelines)
  - [`@task`: define a pipeline task](#task-define-a-pipeline-task)
  - [`buffer`: bound eager task creation](#buffer-bound-eager-task-creation)
  - [`run_workflow`: execute a pipeline](#run_workflow-execute-a-pipeline)
  - [Passing static configuration parameters to tasks](#passing-static-configuration-parameters-to-tasks)
  - [Inspecting a run](#inspecting-a-run)
  - [Rerunning a task out of band](#rerunning-a-task-out-of-band)
  - [Inspecting task inputs and outputs](#inspecting-task-inputs-and-outputs)
  - [Remote execution](#remote-execution)
  - [Containerization](#containerization)
- [Comparison with other libraries](#comparison-with-other-libraries)
- [Comparison with Nextflow](#comparison-with-nextflow)

## Installation

Requires Python 3.11+, a POSIX system, and the `zstd` command-line program for
caching and task execution. There are no required third-party Python packages.

```sh
# run from a source checkout without installing:
PYTHONPATH=src python3 -c 'import wofl'

# install from PyPI:
python3 -m pip install wofl

# install the latest GitHub version:
python3 -m pip install git+https://github.com/ast-al/wofl.git
```

With [uv](https://docs.astral.sh/uv/), add wofl to a project with:

```sh
uv add wofl
```

Or use it in a standalone script without creating a project:

```python
#!/usr/bin/env -S uv run --script
# /// script
# dependencies = [
#   "wofl",
# ]
# ///

import wofl
```

wofl is a single public-domain module; you can also copy
[`src/wofl.py`](src/wofl.py) into your project.

## Complete example

This runnable pipeline uses every public API with every parameter shown explicitly.
It chunks an input file, sorts the chunks concurrently with GNU `sort`,
and merges them into one sorted file. For example, `python pipeline.py records.tsv 2,2n`
sorts numerically by the second field. The key uses GNU `sort --key` syntax.
Cached results are reused when the pipeline runs again.

```python
#!/usr/bin/env -S uv run --script --with wofl==0.6.0

import shutil
import sys
from pathlib import Path

import wofl

throttle_workers = wofl.throttle(capacity=2, min_delay=0.1)
executor_kwargs = {}  # e.g. container_overrides_json for AWS or GCP

# Define reusable decorator by composing wofl-decorators.
# This can go in a config file for generic "run-anywhere" pipelines.
def task(fn):
    fn = wofl.cache(
        workspace_root="./work",
        expires_in=None,
        ignore_file_meta=False,
        environment_fingerprint="local-example-v1",
    )(fn)
    fn = wofl.task(
        executor="local",   # local|qsub|bsub|sbatch|...
        clear_taskdir=True, # clear-out taskdir from a previous run
        timeout=30,

        # Resource requirements for remote schedulers.
        # these can be static values or dynamic callables
        # that compute the requirement based on task's args
        # and/or retry-number.
        cpus=None,
        gpus=None,
        mem_mb=None,
        queue=None,

        remote_cwd=None,
        python3=None,       # can be containerized, e.g. 'apptainer run ... python3 -B'
        staging_url=None,   # file exchange bucket for cloud-backed schedulers
        staging_transport=None, # required when staging_url is set
        **executor_kwargs,
    )(fn)
    return fn

@task
def t_chunk(source: Path, lines_per_chunk: int) -> list[Path]:
    wofl.cmd.run(["split", "-l", str(lines_per_chunk), source, "chunk-"])
    return sorted(Path(".").glob("chunk-*"))

@task
@wofl.retry(max_retries=1, delay=1)
@throttle_workers(slots=1)
def t_sort(chunk: Path, key: str) -> Path:
    output = Path("sorted.txt")
    wofl.cmd.run(["sort", f"--key={key}", f"--output={output}", chunk])
    return output

@task
def t_merge(chunks: list[Path], key: str) -> Path:
    output = Path("sorted.txt")
    if not chunks:
        output.touch()
        return output

    with (
        wofl.cmd.popen(["sort", "--merge", f"--key={key}", *chunks], text=False) as command,
        output.open("wb") as merged,
    ):
        shutil.copyfileobj(command.stdout, merged)
    return output

async def wf_sort(source: Path, key: str):
    chunks = await t_chunk(source, lines_per_chunk=100_000)
    sorted_chunks = [t_sort(chunk, key) for chunk in chunks]
    return t_merge(sorted_chunks, key)

if __name__ == "__main__":
    if len(sys.argv) != 3:
        raise SystemExit(f"usage: {sys.argv[0]} INPUT_FILE GNU_SORT_KEY")
    source, key = Path(sys.argv[1]), sys.argv[2]
    output = wofl.run_workflow(wf_sort(source, key))

    with output.open(encoding="utf-8") as result:
        shutil.copyfileobj(result, sys.stdout)
```

## Command helpers

Both helpers default to UTF-8 text (`text=True`, `encoding="utf-8"`). Pass
`text=False` for binary input/output, or set `encoding` for another text encoding.
Use strings for text input and bytes for binary input.
With `shell=True`, both helpers default to `executable="strict_bash"`.

### `cmd.run`: run a command or script

`wofl.cmd.run()` runs a command and raises `subprocess.CalledProcessError` on
failure. Pass an argument list to keep filenames and other values intact:

```python
from pathlib import Path
import wofl

input_file = Path("reads.fa")
wofl.cmd.run(["wc", "-l", str(input_file)])
```

A plain string is split with `shlex.split`; no shell runs unless you request
one. On Python 3.14+, t-strings quote interpolated values automatically:

```python
wofl.cmd.run(t"wc -l {input_file}")
```

For shell pipelines or scripts, `shell=True` uses Bash with strict error handling:

```python
wofl.cmd.run(
    "sort input.txt | gzip > output.gz",
    shell=True,
)
```

`strict_bash` enables `errexit`, `nounset`, `pipefail`, and `noclobber`, sets
the locale to `C`, and limits field splitting to newlines and tabs. Pass an
explicit `executable`, such as `"/bin/sh"`, to use a shell without these settings.
Other keyword arguments go to `subprocess.run`; `check=True` is the default.
`cmd.run` returns a `subprocess.CompletedProcess` with the exit status and any
captured output. Use `capture_output=True` to collect stdout and stderr:

```python
result = wofl.cmd.run(["wc", "-l", str(input_file)], capture_output=True)
print(result.stdout, end="")
```

### `cmd.popen`: connect and read commands

`wofl.cmd.popen()` connects commands with Unix-style pipes and process
substitution, without a shell or intermediate files. It yields a
`subprocess.Popen` object with text output, waits for the command on context
exit, and checks its status.

If no standard stream is set to `subprocess.PIPE`, stdout is piped by default.
Otherwise, only the requested streams are piped. To write stdin and read stdout,
set both to `subprocess.PIPE`.

Pass a process as `stdin` to pipe its output into the next command.
For example, `cat input.txt | sort -u` becomes:

```python
import wofl

with (
    wofl.cmd.popen(["cat", "input.txt"]) as source,
    wofl.cmd.popen("sort -u", stdin=source) as unique,
):
    lines = [line.rstrip("\n") for line in unique.stdout]
```

Put processes in an argument list to use their output as file inputs.
For example, `diff -u <(sort left.txt) <(sort right.txt)` compares two sorted
streams. Accept status `1` because `diff` uses it to report differences:

```python
with (
    wofl.cmd.popen("sort left.txt") as left,
    wofl.cmd.popen("sort right.txt") as right,
    wofl.cmd.popen(["diff", "-u", left, right], ok_retcodes=(0, 1)) as diff,
):
    result = diff.stdout.read()
```

For output substitution, open the destination with `stdin=subprocess.PIPE`.
A process in an argument list must have exactly one remaining pipe. A successful
connection closes and detaches the parent endpoint so EOF and SIGPIPE can propagate.
The following command compresses a copy of the data while counting its lines,
like `seq 1 100 | tee >(gzip > out.gz) | wc -l`:

```python
import subprocess

with (
    open("out.gz", "wb") as out,
    wofl.cmd.popen("gzip", stdin=subprocess.PIPE, stdout=out) as gzip,
    wofl.cmd.popen("seq 1 100") as source,
    wofl.cmd.popen(["tee", gzip], stdin=source) as tee,
    wofl.cmd.popen("wc -l", stdin=tee) as count,
):
    print(count.stdout.read(), end="")
```

See the [`cmd.popen` docstring](src/wofl.py) for command options, pipe
selection and ownership, cleanup, and exit-status rules.

## Decorators

All decorator factories require parentheses. `@cache()`, `@retry()`, and
decorators created by `throttle()` work on ordinary synchronous functions.
Add `@task()` to schedule calls as asynchronous pipeline tasks.

wofl combines these policies into one configuration, so their order does not
change the execution stages. Cache hits skip execution; on a miss, each attempt
acquires throttles before execution and releases them before retry backoff.

### `@retry`: retry transient failures

Use retries for idempotent work: repeating a failed call should be safe.
`@wofl.retry()` is equivalent to `@wofl.retry(max_retries=1, delay=1)`:
one retry after one second. `max_retries` excludes the initial execution, so
`max_retries=3` permits at most four executions and `max_retries=0` disables
retries. Both arguments are keyword-only. The retry limit must be a non-negative
integer; a fixed delay must be a finite non-negative number of seconds.

Supply a `delay` callback to select errors or calculate backoff:

```python
@wofl.retry(
    max_retries=3,
    delay=lambda _attempt, _error: (
        2 ** (_attempt - 1)
        if isinstance(_error, (ConnectionError, TimeoutError))
        else None
    ),
)
def fetch_data(url: str) -> bytes:
    ...
```

`_attempt` is `1` for the first retry, and `_error` is the exception just
raised. The callback may also request function arguments by name. Return
seconds of retry backoff, `0` for no backoff, or `None` to stop. Fatal exceptions
and programming errors such as `KeyboardInterrupt`, `SystemExit`,
`AssertionError`, and `TypeError` are never passed to `delay` or retried.

The retry limit is checked before calling the callback. When the limit is
reached or the callback returns `None`, the last exception is propagated
without another wait. A callback does not remove the default limit of one
retry; set `max_retries` explicitly to permit more retries.

### `@cache`: persist results on disk

Like `functools.cache`, `@wofl.cache()` reuses a function's result, but persists
it across script runs. The wrapped function retains ordinary Python working-
directory and `Path` semantics:

```python
@wofl.cache()
def normalize(input_file: Path) -> Path:
    output = input_file.with_suffix(".normalized.txt")
    output.write_text(input_file.read_text().lower())
    return output

normalized = normalize(Path("input.txt"))
```

Each call's result is stored in
`./work/<module>/<function>/<args_hash>/.wofl/cache.pkl.zst`, without changing
the function's current working directory. The caller's current working directory
is not implicitly part of cache identity. If the function depends on it, pass
`Path.cwd()` as an argument or represent it in `environment_fingerprint`.
Relative `Path` arguments are associated with the directory that resolves them.
Arguments and results must be picklable.

Pass files as **`Path` objects** to track their contents; strings are ordinary
values. Cache-only functions receive and return paths unchanged. Directory
inputs are tracked recursively, excluding `.wofl/` metadata. Returned files are
tracked by content; returned directories track only their existence and type,
not their contents.

Cache validation automatically includes the workflow process's `sys.version`.
It also tracks function source, directly referenced helper functions, picklable
globals and captured closure values, and input/output files. A source or file
change invalidates the result without necessarily changing the call directory.
Cached functions and referenced Python helpers must have readable source.
Only wofl configuration wrappers are removed before source and runtime
dependencies are collected. User wrapper code and captured configuration are
part of the identity. Source identities ignore comments and formatting.
String contents and nested decorators remain part of the identity.
Module contents reached through module objects or lazy imports, unpicklable
object state, API data, installed tools, and the payload's runtime environment
need an explicit version argument, an expiry, or an environment fingerprint.

The executor, backend, `python3` command, resources, and executor-specific
options control dispatch and are not cache identity. Use
`environment_fingerprint` to identify a remote interpreter, container image,
Conda environment, loaded modules, or other external runtime. If it is omitted,
changing those settings may reuse an existing result.

| Option | Purpose |
| --- | --- |
| `workspace_root="./work"` | Choose where call directories and cache records live. |
| `expires_in=None` | Limit cache age in seconds; `None` never expires, `0` disables cache lookup and storage. |
| `environment_fingerprint=None` | Declare a stable identity for the payload runtime or external state, such as a tool version, image digest, or lock-file hash. |
| `ignore_file_meta=False` | Set to `True` to ignore persisted checksum hints. The checksum LRU in the current process still applies. |

A call directory has a `.running.<id>` suffix during fresh execution and a
`.failed` suffix if that execution fails. Errors during cache setup or after
publication leave the published directory and its execution metadata unchanged.

For outputs with irrelevant, nondeterministic metadata, the producing task can
write a `<file>.wofl_checksum` sidecar with a normalized checksum. wofl uses
that checksum in place of hashing the file's bytes.

### `@throttle`: limit concurrency and attempt rate

`wofl.throttle()` creates a reusable decorator factory. Functions decorated
through the **same factory** share its limit:

```python
api_limit = wofl.throttle(capacity=4, min_delay=0.5)

@wofl.task()
@api_limit()
def fetch_page(url: str) -> bytes:
    ...
```

This permits up to four concurrent attempts and spaces attempt starts by at least
half a second, including retries. Defaults are `capacity=1`, `min_delay=0`, and
one slot per attempt. Slots remain held while waiting for other throttle
resources, the rate limit, and execution cleanup, but are released before retry
backoff. Each retry competes for capacity again.

For weighted resources, calculate the required slots from arguments:

```python
memory_limit = wofl.throttle(capacity=32768)

@wofl.task()
@memory_limit(slots=lambda requested_mb: requested_mb)
def analyze(input_file: Path, requested_mb: int) -> Path:
    ...
```

Callable `slots` resolve on each attempt after dependencies resolve. They can
also request `_attempt`, which is `0` for the initial attempt and increases on
each retry, to match executor resource escalation.

Each new `wofl.throttle()` call creates an independent limiter.

Set an integer priority on a decoration, for example
`@memory_limit(slots=8192, priority=10)`. The default priority is `0`;
higher values go first. Only requests at the highest waiting priority can
acquire slots. Within that priority, the largest request that fits goes first,
with arrival order breaking ties. Lower priorities wait even when they fit.
This can leave slots idle or delay lower priorities indefinitely. Running calls
continue, and calls with `slots=0` bypass capacity admission. Each limiter
applies this order independently.

## Asynchronous pipelines

### `@task`: define a pipeline task

`@wofl.task()` makes a function call return an eagerly scheduled `asyncio.Task`.
By default, each task runs in an isolated local process and working directory.
Add `@cache()` to reuse results; `expires_in=3600` limits cache age to one hour.
Pass files as `Path` objects to track their contents. wofl adjusts paths between
task directories; see the [path-rebasing demo](examples/task_path_rebasing_demo.py).

Pass task handles to other tasks to express dependencies, including handles
nested in lists, tuples, or dictionaries. wofl waits for them and passes their
resolved values to the payload. Independent calls can run concurrently.
Use `await` when the workflow itself needs a value to branch or build more work.

The synchronous decorators preserve the function's parameter and return types.
A task with a payload annotated `-> int` returns `wofl.Task[int]`; both `await`
and `.result()` produce `int`. `Task` extends `asyncio.Task` and exposes
`task_dir: Path | None`. Reapplying decorators preserves these types.

`run_workflow()` returns `Any` to type checkers because it resolves task handles,
including handles inside returned containers. Individual tasks remain typed.

Task payload arguments are not statically checked, so nested task handles need
no ignores. `_depends_on` is checked as an iterable of wofl task handles.

For mypy, enable `follow_untyped_imports` to read wofl's annotations. CI runs
the [typing checks](tests/typing/decorators.py) with `python -m mypy`; install
the `dev` extra to run them locally.

Save this example as `pipeline.py` and run it twice to see cache reuse:

```python
from pathlib import Path
import wofl

@wofl.task()
@wofl.cache()
def uppercase(text: str) -> Path:
    output = Path("text.txt")
    output.write_text(text.upper())
    return output

@wofl.task()
@wofl.cache()
def combine(files: list[Path]) -> Path:
    output = Path("combined.txt")
    output.write_text("\n".join(path.read_text() for path in files))
    return output

async def workflow():
    parts = [uppercase(word) for word in ("hello", "world")]
    return combine(parts)

if __name__ == "__main__":
    result = wofl.run_workflow(workflow())
    print(result.read_text())
```

Define tasks at module level in a `.py` file and guard workflow startup as above.
Return simple values or paths to output files. Copy outputs to a separate
results directory if they must survive reruns or workspace cleanup.
See the [`task()` docstring](src/wofl.py#L6087) for serialization, path, cleanup,
and artifact-lifetime rules.

Task payloads can use user decorators applied in source. Decorators must use
`functools.wraps`, return synchronous functions, and preserve the payload's call
signature. Put wofl decorators above user decorators:

```python
import functools
import os

import wofl

def set_env(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        os.environ["MY_VAR"] = "MY_VAL"
        return func(*args, **kwargs)
    return wrapper

@wofl.task()
@set_env
def compute():
    return os.environ["MY_VAR"]
```

The wrapper sets the environment variable in the worker process before the
payload runs. Cache hits skip this work; each retry runs it again.

Workers import the source and apply these decorators there. Decorators imported
from local modules follow the same source-shipping rules as other helpers.
Wrappers and captured state added only at runtime are not shipped.

wofl finds nested task handles and `Path` values in dataclass instances and
`dict`, `list`, `tuple`, `set`, `frozenset`, `deque`, `ChainMap`, `UserDict`, and
`UserList` containers, including subclasses. Reconstruction uses
`copy.deepcopy` and preserves cycles and shared references. Other objects,
including custom iterables, are opaque: their contents are not resolved or
rebased, and the objects retain their identity during replacement.

#### Task results and dependency provenance

Awaiting a task, or calling `result()` on a completed task, returns a
plain payload value with its ordinary Python type and operations.

Task handles and paths to task outputs record DAG dependencies automatically.
Use `_depends_on` to retain DAG provenance in `.wofl/execution.json` after awaiting or
transforming a result:

```python
@wofl.task()
def split_words(text: str) -> list[str]:
    return text.split()

async def workflow():
    source = split_words("hello world")
    words = await source  # an ordinary list of strings
    outputs = [uppercase(word, _depends_on=(source,)) for word in words]
    return combine(outputs)
```

### `buffer`: bound eager task creation

`@task()` calls start eagerly. Use `wofl.buffer()` to consume a task generator
without creating every task at once. It yields tasks in input order and creates
one replacement each time the consumer advances:

```python
async def workflow(lines, size):
    tasks = wofl.buffer(
        (uppercase(line) for line in lines),
        size,
    )

    for task in tasks:
        print(await task)
```

A positive `size` keeps up to that many tasks in flight. `0` and `None`
disable look-ahead, so the input is consumed directly. The consumer must await
each task before it advances to retain the bound. Use `@throttle` instead when
all task handles can be created immediately and only concurrent execution needs
a limit.

### `run_workflow`: execute a pipeline

`wofl.run_workflow()` is the synchronous entry point. It accepts an awaitable
and resolves tasks nested in the returned result:

```python
async def workflow():
    return uppercase("hello")

result = wofl.run_workflow(workflow())
```

Create task calls inside the async workflow, where an event loop is running.
Calling `workflow()` creates a coroutine; its body runs inside the runner.
Passing a function or lambda to `run_workflow()` raises `TypeError`.
Returning a list or dictionary of terminal tasks is sufficient;
there is no need to await each leaf separately.

Await or return every task whose outcome must affect the workflow. Tasks passed
as dependencies are awaited automatically. A task error must escape the workflow or
its returned task graph to fail the run. You can catch task errors and continue.
Other task failures are logged but do not change the run's result.

To finish a batch after individual samples fail, use
`asyncio.gather(..., return_exceptions=True)` and retain the sample IDs.
The [batch-outcome example](examples/batch_outcomes_demo.py) returns a dictionary
with one result or exception per sample, including multiple failures. Collected
failures do not fail the workflow; the caller can use the report to choose an
exit status. Cancellation of the workflow still propagates.

Progress goes to standard error; pass `log=None` to silence it or a text stream
to redirect it. By default, an error escaping the workflow or returned task graph
cancels other executing tasks. Set `termination_policy="finish"` to let them
finish before raising an ordinary error. Normal exit and Ctrl-C cancel remaining
tasks under either policy. The runner waits for all owned tasks to finish cleanup
before returning or raising.

Worker and scheduler log reporting, and remote staging cleanup, are best effort.
Failures produce warnings; they do not fail or retry the payload. Result
retrieval and output validation must still succeed.

Structured event logging is controlled separately by `structured_log`. Its
default, `"auto"`, writes a run-specific JSON Lines file under the workspace,
compresses it at the end of the run, and prints the final `.jsonl.zst` path as
`Logfile: <path>`. Pass `None` to disable structured logging, or pass a text
stream to write JSON Lines directly without creating a structured-log file; the
caller retains ownership of that stream.

### Passing static configuration parameters to tasks

Wofl does not provide any special facilities for this.

Load static configuration from a JSON or YAML file in user code. Pass each
task's configuration explicitly as an ordinary parameter. The task defines
the structure of its configuration and validates its contents.

Note that any paths passed as strings in `config` dict do not participate
in content-based cache invalidation; for that, they must be cast to `Path`
in-place before passing the config to the task.

`config.yaml`:
```yaml
task_configs:
  t_my_task:
    foo: 42
```

pipeline.py
```python
from pathlib import Path
from typing import Any

import yaml
import wofl


@wofl.task()
def t_my_task(*, config: dict[str, Any]) -> int:
    # Validate task-specific configuration at the task boundary.
    assert isinstance(config["foo"], int)
    return config["foo"]


async def wf_workflow(task_configs: dict[str, Any]) -> int:
    # wire task-specific args from `task_configs` to respective task's params as necessary.
    return await t_my_task(config=task_configs["t_my_task"])


def main() -> None:
    # Load configuration before starting the workflow.
    config = yaml.safe_load(Path("config.yaml").read_text())
    result = wofl.run_workflow(wf_workflow(config["task_configs"]))
    print(result)


if __name__ == "__main__":
    main()
```

### Inspecting a run

In the default `structured_log="auto"` mode, structured logs live in
`<workspace>/<module>/.wofl_logs/` and record task states, cache decisions,
retries, exceptions, and worker output. Each line contains one JSON event.
Read a completed log with `zstd -dc path/to/log.jsonl.zst`.
Worker host diagnostics and resource usage are stored in
`<taskdir>/.wofl/worker/host_info.json.zst`.

To inspect the dependency graph leading to an output task, use its directory:

```sh
taskdir='work/pipeline/combine/<args_hash>'  # replace with the actual directory
python3 -m wofl show-dag "$taskdir"
python3 -m wofl show-dag "$taskdir" --out-fmt=tsv
```

The default YAML output includes a text timeline, execution IDs, task timings,
and input/output files. Input paths are relative to the pipeline module directory
(`<workspace_root>/<module>/`); output paths are relative to the task directory.

When a worker resource report is available, YAML also includes `cpu-time-s`,
`wall-time-s`, and `peak-rss-mib`. CPU time is user plus system time for the
worker and its waited-for children. Wall time is the measured worker interval,
including setup and cleanup. Peak RSS is the larger of the final worker and
child-process RSS peaks, in MiB; it is not the combined peak of the process tree.
Existing resource reports retain their original RSS values; rerun a task to
collect the corrected peak.

`show-dag` uses restricted unpickling. If a cache contains unsupported values,
it reports a warning and omits that cache's file details. The execution graph
and metadata remain available.

Execution IDs, timings, status, and dependencies are stored in
`<taskdir>/.wofl/execution.json`, including failed and uncached executions.
Cache hits leave this file unchanged. Its dependencies describe the execution
that produced the cached result. Dependencies are written before the payload runs,
so failed runs can also be inspected. Worker stdout, stderr, and host/resource
diagnostics are in `<taskdir>/.wofl/worker/` as compressed files.
Worker launch settings are in `<taskdir>/.wofl/worker/task.json`. This file is
recreated for each attempt.

### Rerunning a task out of band

Run a previously dispatched task locally with its saved arguments:

```sh
python3 -m wofl run pipeline.py "$taskdir"
```

Use the existing directory, including its `.failed` suffix if present. This
executes the payload in place and writes `.wofl/worker/out.pkl.zst`; it bypasses
caching, retries, throttling, and scheduler submission, and does not refresh
the workflow cache. The local environment must provide the payload's tools.

`run` returns status `0` for a task result and status `1` for a task exception.
Automatic launches use `run --worker`, which returns status `0` after reporting
either outcome in `out.pkl.zst`. `run-supervised` starts the worker in this mode
and handles its deadline and logs. Only the submission host decodes the response
and raises task exceptions. Schedulers see reported task exceptions as successful
jobs; Wofl controls task status and retries. Worker and transfer failures still
produce a nonzero status, even if an output file exists.

Failed tasks retain a source snapshot under `.wofl/worker/source/`, which
takes precedence over the supplied pipeline file. Successful tasks remove that
snapshot and use the available source when rerun. For normal resumption of
cached tasks, run the workflow again and let cache validation select the tasks
to execute.

### Inspecting task inputs and outputs

`inspect-pickle` renders saved values as JSON:

```sh
python3 -m wofl inspect-pickle "$taskdir/.wofl/worker/inp.pkl.zst"
python3 -m wofl inspect-pickle "$taskdir/.wofl/worker/out.pkl.zst"
python3 -m wofl inspect-pickle "$taskdir/.wofl/cache.pkl.zst"
```

These contain the saved `(args, kwargs)`, the worker result or exception, and
the cache record respectively. File outputs remain in the task directory.

### Remote execution

Set `executor` to a command prefix; task bodies and dependency wiring stay the
same:

| Execution target | `executor` |
| --- | --- |
| Local process (default) | `"local"` |
| SSH | `"ssh user@host"` |
| Slurm / SGE | `"srun"`, `"sbatch"` / `"qrsh"`, `"qsub"` |
| LSF / Flux / HTCondor | `"bsub"` / `"flux"` / `"condor_submit"` |
| AWS Batch / ECS | `"aws batch submit-job ..."` / `"aws ecs run-task ..."` |
| GCP Batch / Cloud Run Jobs | `"gcloud batch jobs submit ..."` / `"gcloud run jobs execute ..."` |

Executor strings can include scheduler or provider flags. Common resource
options are `cpus`, `gpus`, `mem_mb`, `queue`, and `timeout` (seconds per
attempt, excluding queue wait). Options can be computed from arguments and the
zero-based attempt number:

```python
@wofl.retry(max_retries=2, delay=30)
@wofl.task(
    executor="sbatch",
    cpus=8,
    mem_mb=lambda _attempt: 4096 * 2 ** _attempt,
    timeout=1800,
)
def align(input_file: Path) -> Path:
    ...
```

`executor`, resource options, and backend-specific keyword arguments accept
synchronous callbacks selecting payload arguments by name. They are evaluated
once per execution attempt. Cache hits skip these callbacks and backend setup.
Resource requests go to the scheduler; configure the tool itself to use the
requested CPUs or GPUs.

HPC jobs expect a shared filesystem. SSH also uses shared paths by default;
set `remote_cwd="/scratch"` to stage the task and its input files to that host.
Cloud jobs require a writable `staging_url`, such as `"s3://bucket/wofl/"` or
`"gs://bucket/wofl/"`, and a `staging_transport`. Set both options together.
The submitting host needs the provider's submission flags and credentials,
and uses `aws s3` for S3 staging or `gsutil` for GCS staging.

SSH and cloud return archives exclude the reserved `.wofl/worker/inp` subtree.
After a failure or timeout, they retain logs, partial outputs, and scratch files.
Output symlinks are dereferenced so their contents remain available locally.

Set `staging_transport` to `"rclone"`, `"s5cmd"`, `"aws s3 cp"`,
`"gcloud storage cp"`, or `"gsutil cp"` to use a command installed in the worker
image. Use a command that supports the staging bucket. For example:

```python
@wofl.task(
    executor="aws batch submit-job --job-queue myqueue --job-definition myjob",
    staging_url="s3://bucket/wofl/",
    staging_transport="s5cmd",
)
def process(input_file: Path) -> Path:
    ...
```

To download `rclone` or `s5cmd` in the worker, set `staging_transport` to an
HTTP(S) URL for the executable. Its filename must contain `rclone` or `s5cmd`.
Provide a direct binary URL for the image's OS and CPU architecture, not an
archive URL. The worker downloads it with a Python one-liner that uses
`urllib.request`, makes it executable, and removes it when the job exits.
The binary download makes one attempt with a 60-second socket timeout.
Archive transfers use the selected CLI's native retry policy.
Workers need credentials or an assigned role that permits reads and writes to
the staging bucket. Rclone uses runtime credentials for S3 and GCS.

wofl ships its module and a snapshot of the task's Python source. Workers need
Python 3.11+, `zstd`, and the payload's packages and command-line tools;
installing wofl on workers is unnecessary. Staged SSH execution also needs
Bash and tar; cloud images need these tools and the selected staging transport.
Pass data files as `Path` arguments; package data is not shipped automatically.

Source collection follows `import` and `from ... import` statements throughout
the task module and its local dependencies under the same import root. It
includes imports inside functions and conditional branches, and required
package `__init__.py` files. Computed imports through `importlib.import_module`
or `__import__` are not discovered. Modules needed only by pickled argument
types must be imported by the task's source or installed on the worker.

### Containerization

The `python3` option accepts a full interpreter command, allowing tasks to run
through Docker, Apptainer, Singularity, or Conda:

```python
@wofl.cache(environment_fingerprint="my-image-sha256:...")
@wofl.task(
    executor="sbatch",
    python3="docker run --rm -v $PWD:$PWD my-image python3",
)
def process(input_file: Path) -> Path:
    ...
```

`$PWD` expands to the work root; mount it at the same absolute path and make
input paths outside it accessible too. The environment needs Python, `zstd`,
and the task's dependencies. Pin the environment and set
`environment_fingerprint` to its digest or lock-file hash so changes invalidate
cached results. The `python3` command controls execution only; changing it does
not invalidate the cache by itself. Cloud executors use the image configured
for the cloud job.

See [Running containerized workloads](docs/running_containerized_workloads.md)
for further examples.

## Comparison with other libraries

Related Python libraries include
[redun](https://insitro.github.io/redun/design.html), with lazy task expressions
and database-backed provenance;
[Pydra](https://nipype.github.io/pydra/explanation/hashing-caching.html), with
hashed task directories and file-aware caching; and
[Parsl](https://parsl.readthedocs.io/en/stable/userguide/workflows/workflow.html),
with futures and parallel Python workflows.

wofl combines eager `asyncio` tasks, persistent file-aware caching, and direct
job submission in one vendorable module. State lives in task directories and
logs, with no daemon or metadata database. It provides no web UI, recurring
schedule service, or worker-pool autoscaler. See the
[detailed comparison](docs/comparison_to_other_workflow_libs.md) for differences
in execution, persistence, and scope.

## Comparison with Nextflow

Both support cached, parallel scientific pipelines with cluster and cloud
execution. Nextflow expresses work through
[processes](https://docs.seqera.io/nextflow/process) and
[workflows connected by channels](https://docs.seqera.io/nextflow/workflow).
wofl expresses it through Python functions and task results:

| Concern | Nextflow | wofl |
| --- | --- | --- |
| Task inputs and outputs | Process input/output declarations | Function arguments and return values; `Path` for files |
| Scatter and gather | Channels and operators | Loops, comprehensions, lists of tasks, and `await` |
| Execution policy | Directives and [configuration profiles](https://docs.seqera.io/nextflow/config#config-profiles) | Decorator options and reusable Python decorator functions |
| Workflow composition | Named workflows with `take` / `emit` | Async functions returning task results |

Compare the same protein-alignment pipeline in
[Python](examples/align_prots.py) and [Nextflow](examples/align_prots.nf).
The [feature comparison](docs/comparison_to_nextflow.md) maps further Nextflow
constructs to Python and describes facilities wofl omits.
