Metadata-Version: 2.4
Name: bglaunch
Version: 0.1.0
Summary: Launch Python training jobs in the background on one machine, with a core budget
Author: Sawan Singh Mahara
License-Expression: MIT
Project-URL: Homepage, https://github.com/SmellingSalt/bglaunch
Project-URL: Issues, https://github.com/SmellingSalt/bglaunch/issues
Keywords: multiprocessing,jobs,gpu,cuda,scheduler,background
Classifier: Programming Language :: Python :: 3
Classifier: Operating System :: POSIX :: Linux
Classifier: Operating System :: MacOS
Classifier: Topic :: System :: Distributed Computing
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Dynamic: license-file

# bglaunch

Start Python training jobs in the background on one machine, with a hard limit
on how many cores they may use.

`launchJob` takes a Python file and a configuration, starts it in a separate
process, and returns one line of text saying what happened. It does not import
your code, so the job can be any script that takes one config argument.

Each job is a fresh interpreter started by the shell (fork followed by exec), so
nothing CUDA-related is inherited from the caller. That is the pattern NVIDIA
allows: a forked child must exec before it touches the driver. The alternative,
forking a process that already holds a CUDA context, is what produces the
`Cannot re-initialize CUDA in forked subprocess` failures.

## Install

```bash
pip install git+https://github.com/SmellingSalt/bglaunch
```

Or from a clone of this repository:

```bash
pip install .
```

## Use it

```python
from bglaunch import launchJob

print(launchJob(pythonFile="train.py", config="config.yaml"))
```

```
launched pythonFile "train.py" with config config.yaml with 1 thread(s), PID 27930 log logs/config.log
```

`threads` is optional and defaults to 1. It sets `OMP_NUM_THREADS` and
`MKL_NUM_THREADS` for the job and counts the same number against the core
budget. It does not pin cores; a script that ignores those variables can still
use more than it asked for.

The bash script is the same API one level down, if you would rather not use
Python at all:

```bash
bash bglaunch/launchParallel.sh train.py config.yaml -threads 1
```

The package is two files plus examples:

```
bglaunch/launchJob.py        launchJob(...), the API
bglaunch/launchParallel.sh   the engine: core budget, logs, pid registry
bglaunch/examples/           delayExample.py, delay.py, configs/
```

## What a launch guarantees

- The job runs in the background; the call returns in milliseconds.
- `logs/<config name>.log` holds its stdout and stderr.
- `logs/pids.txt` gains a `pid  threads  config` row.
- At most `CPUS - 1` cores are handed out across all jobs, so one core always
  stays free. A launch that would take the last one is refused instead of
  oversubscribing the machine.
- Finished jobs are dropped from the registry on the next launch. A pid reused
  by an unrelated process does not hold cores, because the registry entry only
  counts while the live command line still ends in the same config.

Exit codes, if you call the script yourself: `0` started, `2` bad arguments,
`3` no free core left. `launchJob` turns all of them into text.

## Example

`examples/delay.py` sleeps for the number of seconds in its config and records
when it started and finished. `examples/configs/` has five delays (2, 10, 15, 26,
32 seconds); the demo below runs two of them.

```bash
python3 -m bglaunch.examples.delayExample
```

```
launched pythonFile "./delay.py" with config configs/delay_02.yaml with 1 thread(s), PID 29631 log logs/delay_02.log
launched pythonFile "./delay.py" with config configs/delay_10.yaml with 1 thread(s), PID 29641 log logs/delay_10.log
```

The example switches to its own folder first, so the logs land in
`bglaunch/examples/logs/` when you run it from a clone. Each job also writes
`results/delayTest/<config name>/timing.txt` next to it, so one folder per job
shows up in the shell.

## Watching a batch

```bash
cat logs/pids.txt        # what is running: pid, threads, config
tail -f logs/*.log       # live output of every job
```

To check per-job cpu use, for instance when judging the thread count, ask ps
for those pids. Linux:

```bash
ps -o pid,psr,pcpu,etime -p $(awk -F'\t' 'NR>1{print $1}' logs/pids.txt | paste -sd, -)
```

`psr` is the cpu a job ran on. macOS has no `psr` column, so drop it there.

## Limits

- The launcher reports whether a job started, not whether it succeeded. Outcomes
  are in the logs.
- The log name is the config filename without its extension, so relaunching the
  same config overwrites its log.
- Jobs survive the process that launched them; stopping them means killing the
  pids in `logs/pids.txt`.
- Paths resolve against the caller's working directory, and so do the `logs/`
  and `results/` folders the jobs write.
- Tested on Linux and macOS. Windows is not supported, because the launcher is a
  bash script.
