Metadata-Version: 2.4
Name: libpipe
Version: 0.9
Summary: Pipeline orchestration and node management utilities
License-Expression: GPL-3.0-or-later
License-File: LICENSE
Author: Florent Mertens
Author-email: florent.mertens@gmail.com
Requires-Python: >=3.10,<4.0
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Programming Language :: Python :: 3.14
Provides-Extra: dev
Provides-Extra: docs
Requires-Dist: asyncssh (>=2.0)
Requires-Dist: click (>=8.0)
Requires-Dist: myst-parser (>=2) ; extra == "docs"
Requires-Dist: progressbar2 (>=3.40)
Requires-Dist: pydata-sphinx-theme (>=0.15) ; extra == "docs"
Requires-Dist: pytest (>=8.0) ; extra == "dev"
Requires-Dist: pytest-cov (>=5.0) ; extra == "dev"
Requires-Dist: ruff (==0.16.1) ; extra == "dev"
Requires-Dist: sphinx (>=7) ; extra == "docs"
Requires-Dist: sphinx-click (>=6) ; extra == "docs"
Requires-Dist: toml (>=0.10)
Project-URL: Documentation, https://libpipe.readthedocs.io
Project-URL: Homepage, https://gitlab.com/flomertens/libpipe
Project-URL: Issues, https://gitlab.com/flomertens/libpipe/-/issues
Project-URL: Repository, https://gitlab.com/flomertens/libpipe
Description-Content-Type: text/markdown

Libpipe
=======

Common pipeline framework and library

Installation
------------

libpipe can be installed via pip:

    $ pip install libpipe

and requires Python 3.10 or higher.

Worker pools on a Slurm cluster
-------------------------------

A `WorkerPool` runs its tasks locally, on other hosts over SSH, or through Slurm with the
host `slurm`: each task is then started with `srun`, as a job of its own when the pool runs
on a login node, as a job step inside an allocation (`salloc`, `sbatch`).

    pool = worker.WorkerPool(['slurm'], name='Imaging', max_tasks_per_worker=8,
                             slurm_options='-p regular --mem=200G --time=24:00:00')

`max_tasks_per_worker` is the number of jobs in flight. The srun options (partition,
account, resources) come from `slurm_options`, or from the `LIBPIPE_SLURM_OPTIONS`
environment variable; `max_time` becomes the job time limit when the options set none.
Host constraints (`run_on_host`) do not apply: Slurm places the tasks, so the data must be
on a file system shared with the compute nodes.

With `slurm_nodes`, a pool on a login node requests one allocation of up to that many nodes
when its first task starts, runs each task as a job step in it (`srun --jobid --exact`), and
releases it at the end, or after 10 minutes without step. Slurm packs the steps on the nodes
from their CPUs and memory, given by the category of the task; a step starts when the
allocation has its CPUs free, so that no srun waits polling the controller:

    pool = worker.WorkerPool(['slurm'], name='Cubes', slurm_nodes=4,
                             resources={'heavy': {'cpus': 12, 'mem': '40GB'}})
    pool.add('pstool gen_vis_cube ...', category='small', label='gen_vis_cube')

The categories (`DEFAULT_RESOURCES`) are `small` (1 CPU, 2 GB), `medium` (4, 8 GB),
`high_mem` (2, 16 GB), `cpu` (12, 8 GB) and `heavy` (12, 40 GB, the default), the others capped
at `heavy`; a step also gets the matching `OMP_NUM_THREADS`. The options apply to the jobs and
the allocation, not to the steps. Inside an allocation started by hand, the tasks are job
steps with these resources as well.

Slurm does not necessarily limit the memory of a job step: the memory of a category must stay
above the peak memory of its tasks. The local and Slurm pools measure each task (wall time,
CPU time, peak RSS, in its log file), and `WorkerPoolReport.usage_lines()` summarises them per
task type (`label`).

