Roadmap · Features · P2 · M

Stop / re-run a training job

A build-it-yourself guide. Right now a run only ends when on_train returns on its own — there's no way to cancel a run you can already see is going badly. This adds a Stop control (cooperative cancellation) and makes re-launching an edited graph a first-class action.

Done when: a running job can be stopped from the editor — it settles to phase="stopped", drops out of the live jobs, keeps its partial curve — and the Train button cleanly re-launches the edited graph as a fresh run.

0 / 0 steps

← roadmap ← run comparison hyperparameter sweep →

The hard truth about stopping

Jobs run in a daemon thread (_Job.thread in server.py), and Python threads cannot be forcibly killed — there is no thread.kill(). So "stop" has to be cooperative: set a flag the training loop checks between epochs and returns early. Since on_train is user code, the stop signal has to be reachable through the one object it already holds — the reporter.

You can't stop a loop that never checks. A user whose on_train ignores the signal (e.g. one long model.fit() with no epoch loop) can't be interrupted mid-call — that's inherent to in-process training, not a bug. Document the check as the contract; the built-in demo/example trainers should model it.

Steps

Give each job a cancel flag

modulearn/server.py — _Job + RunReporter

Add a threading.Event to the job and hand it to the reporter so on_train can poll it. Expose it as a friendly method.

@dataclass
class RunReporter:
    run_dir: Path
    _state: dict = field(default_factory=dict)
    _cancel: "threading.Event | None" = None

    def should_stop(self) -> bool:      # on_train checks this between epochs
        return self._cancel is not None and self._cancel.is_set()

class _Job:
    def __init__(self, app_state, compiled):
        ...
        self.cancel = threading.Event()

    def _run(self):
        run_dir = self._app.run_dir(self.compiled.run_id)
        reporter = RunReporter(run_dir, _cancel=self.cancel)
        ...

Record the stopped phase in _Job._run

modulearn/server.py — the finalize logic in _Job._run
Today, if on_train returns while phase=="running" it's marked done. When cancellation was requested, settle to stopped instead so a deliberately-halted run isn't mislabeled a success.
```
    self._app.on_train(self.compiled, reporter)
    if reporter._state.get("phase") == "running":
        reporter.state(phase="stopped" if self.cancel.is_set() else "done")
```
The state contract already anticipates this. The docstring lists paused as a known phase; add stopped alongside it, and teach the frontend's pollRun to treat it as a terminal, non-error state (it already special-cases done and error).

Add the stop endpoint

modulearn/server.py — a new route in create_app

Look the live job up in state.jobs (keyed by run_id) and set its event. Idempotent: stopping an already-finished run is a no-op, not an error.

@app.post("/api/run/{run_id}/stop")
def api_stop(run_id: str):
    job = state.jobs.get(run_id)
    if job is None:
        return {"run_id": run_id, "stopped": False, "reason": "not live"}
    job.cancel.set()
    return {"run_id": run_id, "stopped": True}

Make the demo/example trainers cancellation-aware

modulearn/demo.py, modulearn/cli.py (_APP_TEMPLATE), examples/*.py
The signal only works if loops check it. Add one line to the epoch loop so the bundled trainers — and the scaffold users copy — model the contract.
```
    for e in range(epochs):
        if reporter.should_stop():
            break                       # cooperative stop: leaves the partial curve intact
        ...
        reporter.metric(epoch=e, train=..., val=...)
```
Wire the Stop button + re-run in the UI

modulearn/static/graph.js — toolbar near train / pollRun
Show a Stop action while a run is live (there's already a #bar .dot.live signal and tn._running). Stop POSTs the new endpoint; the existing poller will pick up phase="stopped" and settle the panel. Re-run is the existing train() — each launch already compiles the current canvas into a fresh run_id, so an edited graph just relaunches.
```
async function stopRun(runId) {
  await fetch("/api/run/" + runId + "/stop", { method: "POST" });
  // no manual UI change needed: pollRun sees phase="stopped" next tick and
  // calls stopPolling() + refreshRuns() the same way it does for done/error.
}
```
Track the active run id. train() gets j.run_id back from /api/graph/start — stash it (e.g. currentRunId) so the Stop button knows which run to cancel.

Verify stop and re-run

modulearn demo
# Train a long run (bump epochs), then hit Stop mid-run
# expect: panel settles to "stopped", the dot stops pulsing, partial curve stays
curl -s localhost:8000/api/runs | python -m json.tool   # that run: phase "stopped", live false
# edit a hyperparameter and Train again -> a new run_id launches cleanly

Test the "does nothing" path too. Call /api/run/<id>/stop on a finished run — it should return stopped:false, reason:"not live", never a 500. Stopping is a request, not a guarantee.

Add a server test

tests/test_run_errors.py or a new tests/test_stop.py

Drive a job whose on_train loops until should_stop(), set the event, and assert the run settles to phase="stopped" and is dropped from state.jobs — mirroring the synchronous _run() style already in the run-error tests.

Commit + mark shipped

git add modulearn/server.py modulearn/static/graph.js modulearn/demo.py tests/
git commit -m "Stop a running job (cooperative cancel) + re-run an edited graph"
git push
gh run list --branch main --limit 1   # expect: success

Then check the {area:"features", … title:"Stop / re-run a training job"} card's box in docs/roadmap.html (shipped state is per-browser).

Worth knowing

Cooperative > forceful, here. A cancel flag the loop checks lets the run stop cleanly — flush the last metric, leave the partial curve, run the finally that de-registers the job. A hard kill would risk half-written state.json and orphaned entries in state.jobs. The same should_stop() hook is what a sweep uses to abort a whole batch.