Metadata-Version: 2.4
Name: ntrun
Version: 0.1.0
Summary: CSV/TOML task dispatcher for local execution and Slurm
Author: shengda zhao
License-Expression: MIT
Keywords: slurm,csv,toml,batch,hpc,workflow
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Science/Research
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3 :: Only
Requires-Python: >=3.10
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: tomli>=2.0; python_version < "3.11"
Provides-Extra: progress
Requires-Dist: tqdm>=4.64; extra == "progress"
Dynamic: author
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: keywords
Dynamic: license-file
Dynamic: license-expression
Dynamic: provides-extra
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

# ntrun

A small CSV/TOML task dispatcher for local execution and Slurm clusters.
One CSV row describes one task; a TOML file defines its command pipeline.
Scheduling stays separate from scientific code.

Version: **0.1.0**. Author: **shengda zhao**. License: **MIT**.

The source archive includes a detailed Chinese guide (`README.zh-CN.md`),
release instructions (`PUBLISH.md`), tests and runnable examples. Installing
the wheel does not copy those examples into your scientific project.

## Install

After publication to PyPI:

```bash
python -m pip install ntrun
ntrun --version
```

Python 3.10+ is required. Python 3.10 uses `tomli`; Python 3.11+ has no
mandatory third-party runtime dependencies. Optional progress bars:
`python -m pip install "ntrun[progress]"`. Without tqdm, progress is plain text.
Local execution and script generation work on Windows/Linux; actual submission
requires a configured Slurm cluster.

## Quick start

Run from **your project root**, not the installation or TOML directory:

```bash
ntrun -p tasks/run.toml
```

Project layout:

```text
project/
  worker.py
  CSVs/parameters.csv
  tasks/run.toml
  @BaseFiles/base.sh       # needed for Slurm modes only
  task-runs/              # generated records
  slurms/                 # generated scripts
  logs/                   # cluster logs
```

`CSVs/parameters.csv`:

```csv
value,rep,#note
0.1,0,first case
0.2,1,second case
```

`tasks/run.toml`:

```toml
csv_path = "CSVs/parameters.csv"
pipeline = [["python", "worker.py"]]
mode = "local"
name_columns = ["value", "rep"]
task_dir = "task-runs/run"
manifest_path = "task-runs/run/manifest.csv"
```

`worker.py`:

```python
import argparse

parser = argparse.ArgumentParser()
parser.add_argument("--value", type=float, required=True)
parser.add_argument("--rep", type=int, required=True)
args = parser.parse_args()
print(args.value, args.rep)
```

This runs `worker.py --value 0.1 --rep 0`, followed by the second row.
Multiple pipeline commands are supported; each receives the same row arguments.

## Execution modes

| Mode | Behavior |
| --- | --- |
| `local` | Executes each row's commands sequentially on the current machine. |
| `write-slurm` | Writes job scripts and a manifest, without executing or submitting. |
| `slurm` | Writes scripts and submits one job per CSV row with sbatch. |

Only `csv_path` and `pipeline` are required. **Omitting mode defaults to local,
which executes real work.** To inspect a generated script instead:

```bash
ntrun -p tasks/run.toml --mode write-slurm --limit 1
```

This still writes files and requires a valid Slurm template; it is not a
read-only validation command. Slurm mode does not wait for cluster completion.

## Slurm template

The external template, usually `@BaseFiles/base.sh`, controls resources,
partition, account, modules and environment activation:

```bash
#!/bin/bash
#SBATCH --job-name=@NAME@
#SBATCH --output=@OUT_NAME@_%j.out
#SBATCH --error=@OUT_NAME@_%j.err
#SBATCH --cpus-per-task=1
#SBATCH --time=00:05:00

# Add cluster-specific directives and activate your environment above.
BASE_COMMAND
```

Use `BASE_COMMAND` on its own line. `@NAME@` and `@OUT_NAME@` are replaced
by the task name and log prefix. Set `slurm_template`, `slurm_dir` and
`log_dir` in TOML as needed. Generated scripts use LF line endings, enter
the project root, join commands with `&&` and exit with the pipeline status.
Put cleanup in an EXIT trap, not after BASE_COMMAND.
The tool does not choose cluster resources or activate Conda for you.

## CSV argument rules

- UTF-8 CSV, optionally with a BOM; nonempty unique headers are required.
- One row becomes one task. Blank cells, `#`-prefixed columns and
  `exclude_columns` are omitted. The prefix denotes a column, not a comment row.
- `axonRatio`, `axon_ratio` and `axon-ratio` become `--axon-ratio`.
  Headers already starting with `--` are preserved.
- Boolean cells remain values: `false` becomes `--render false`, not a switch.
- Values use POSIX shlex splitting. An unquoted comma list becomes multiple
  arguments; first quote it as a CSV cell, e.g. `"1,2,3"`.
  To preserve spaces as one argument, include shell quotes in the cell content:
  the CSV text `"""hello world"""` produces one `hello world` argument.
- Every pipeline command receives all selected parameters and must accept them.
  Pipeline commands are token lists, not shell strings; pipes and redirects
  require an explicit trusted shell or wrapper script.
- Pipeline tokens ending in .py/.sh are resolved as project paths, even when
  used as arguments. Other relative data paths pass unchanged to the worker.
  Forward slashes are recommended for Windows paths.
- Keep varying case parameters in CSV. Fixed analysis thresholds, fit windows
  and figure settings normally belong in the target scripts or static configs.

## Repeated experiments

ntrun does not interpret a repetition count. A CSV row with `rep=0` and
`repeatCount=32` produces one job with `--rep 0 --repeat-count 32`.
**Your worker must implement the loop**, independent seeds and output paths.
`--limit` counts CSV rows, not repeats. Local rows are serial, not parallel.

## Configuration

| TOML field | Default / meaning |
| --- | --- |
| `csv_path` | Required. CSV path relative to the project root, or absolute. |
| `pipeline` | Required. Nonempty list of commands, each a list of strings. |
| `mode` | `local`; also `write-slurm`, `slurm`. |
| `stop_on_error` | `true`; stops later rows after a dispatch/local error. |
| `row_limit` | Unset; optional positive row count. |
| `slurm_template` | `@BaseFiles/base.sh`. |
| `slurm_dir` | `slurms`; generated scripts. |
| `task_dir` | Legacy default `tasks`; explicitly use `task-runs/<batch>` for new work. |
| `log_dir` | `logs`; Slurm output prefix directory. |
| `manifest_path` | `<task_dir>/TaskRunnerManifest.csv`; use a separate path per batch. |
| `submit_command` | `sbatch`; POSIX-tokenized command, without shell expansion. |
| `use_srun` | `false`; prefixes generated commands with srun when true. |
| `name_columns` | `[]`; CSV columns used in readable task names. |
| `exclude_columns` | `[]`; exact header names not forwarded. |
| `column_flag_style` | `kebab`; or `raw` to preserve header spelling after --. |
| `matplotlib_cache_dir` | `.cache/matplotlib`; per-job cache root. |
| `matplotlib_style_dir` | `~/.config/matplotlib/stylelib`; plotting style source. |

For example, keep static definitions and generated records separate:

```toml
task_dir = "task-runs/batch-a"
manifest_path = "task-runs/batch-a/manifest.csv"
slurm_dir = "slurms/batch-a"
log_dir = "logs/batch-a"
```

Command-line overrides take precedence over TOML:

```bash
ntrun -p tasks/run.toml --mode write-slurm --limit 3
ntrun -p tasks/run.toml --csv CSVs/another.csv
ntrun --project-dir /path/to/project -p tasks/run.toml
ntrun --help
python -m ntrun --version
```

The project root defaults to the current working directory. Relative profile,
CSV and template paths resolve there. The pipeline's `python` is found on PATH.

## Records, errors and limits

The manifest stores `rowIndex, taskName, mode, status, returnCode,
elapsedSeconds, workingDir, slurmScript, command`. Status values are
`success`, `failed`, `scriptWritten`, `submitted`, `submitFailed`, `error`.
The legacy filename TaskRunnerManifest.csv does not depend on another script.

The first failed command stops the rest of that row. Setting
`stop_on_error=false` continues later rows, but the dispatcher still exits
nonzero if any row fails. Exit codes: 0 for success, 1 for handled errors,
130 for Ctrl+C. Slurm success means submission acceptance, not job completion
or scientific validity. Use squeue/sacct to inspect jobs.

Local child output streams to the terminal; it is not automatically saved
under log_dir. Slurm stdout/stderr follow the template. Noninteractive plotting
and per-task caches are configured without importing a plotting library.
Existing local environment values are preserved; generated Slurm scripts
explicitly export their configured values.

There is no queue polling, automatic retry, checkpoint resume, duplicate
suppression, submission throttle or job-array conversion. Job IDs appear in
submission output but are not stored in a dedicated manifest column.
Reruns can overwrite manifests and scripts: use separate record, script, log
and worker-output paths for concurrent batches. Ctrl+C does not cancel
submitted jobs or guarantee termination of local worker descendants.
Only run trusted configurations and programs; ntrun is not a security sandbox
or tamper-proofing system. Directory names are recommendations, not constraints;
existing TOML paths remain compatible.

## Development and examples

From an extracted source distribution, using Python 3.10+:

```bash
python -m pip install -e .
python -m unittest discover -s tests -v
ntrun --project-dir examples -p tasks/local.toml
ntrun --project-dir examples -p tasks/write-slurm.toml --limit 1
```

The local example writes JSON under examples/results; the other only writes
scripts. Tests mock scheduler submission and use temporary directories.
Real cluster scheduling is not exercised by these tests.

For offline installation:

```bash
python -m pip install --no-index /path/to/ntrun-0.1.0-py3-none-any.whl
```

Python 3.10 also needs tomli installed or provided as a local wheel.
Python 3.11+ requires no mandatory third-party runtime dependencies.
See PUBLISH.md for building, checking and manually uploading a release.
