Hyperparameter sweep

Steps

1. Define the sweep request
   modulearn/server.py — a new request model

   Extend the graph payload with which hyperparameter to vary and the values to try. Keep it explicit values (not just min/max/steps) so the frontend owns how the range is generated.

   class _SweepReq(_GraphReq):
       param: str            # the hyperparameter id to vary, e.g. "lr"
       values: list[float]   # e.g. [1e-4, 1e-3, 1e-2]

2. Compile once, then override per value
   modulearn/server.py — a new /api/graph/sweep route

   Compile the base graph first so a broken graph fails once with the usual 400, before launching anything. Then for each value, compile a fresh run (new run_id) and override the swept key in its hyperparameters. Validate the param is actually a wired hyperparameter — don't silently sweep a knob that isn't on the graph.

   @app.post("/api/graph/sweep")
   def api_sweep(req: _SweepReq):
       graph = {"nodes": req.nodes, "links": req.links}
       try:
           base = compiler.compile_graph(graph, registry)     # fail fast, once
       except compiler.GraphError as e:
           raise HTTPException(400, {"errors": e.errors})
       if req.param not in base.hyperparameters:
           raise HTTPException(400, {"errors": [f"'{req.param}' is not a wired hyperparameter"]})
   
       run_ids = []
       for v in req.values:
           c = compiler.compile_graph(graph, registry)         # fresh run_id each time
           c.hyperparameters[req.param] = v                    # the one override
           d = state.run_dir(c.run_id)
           (d / "graph.json").write_text(json.dumps(graph, indent=2))
           (d / "config.json").write_text(json.dumps(c.as_dict(), indent=2))
           run_ids.append(c.run_id)
       _launch_sweep(state, run_ids)                            # queued, see next step
       return {"sweep": {"param": req.param, "values": req.values, "run_ids": run_ids}}

   Bounds still apply. The compiler range-checks hyperparameters against the declared min/max. Overriding after compile skips that — so re-validate each value against the registry's declared range, or (cleaner) thread the override into compile so an out-of-range sweep value is rejected the same way a hand-wired one is.

3. Add a queued launcher
   modulearn/server.py — a small coordinator alongside _Job

   Instead of job.start() × N, run a coordinator that keeps at most k jobs live and starts the next as each finishes. The existing finally that pops the run from state.jobs is your "a slot freed up" signal.

   def _launch_sweep(state, run_ids, k=2):
       pending = list(run_ids)
       def pump():
           while pending and len([j for j in state.jobs.values()]) < k:
               rid = pending.pop(0)
               c = _compiled_from_dir(state, rid)     # rebuild from config.json
               job = _Job(state, c); state.jobs[rid] = job; job.start()
       # a tiny daemon that re-pumps as slots free; or hook pump() into _Job's finally
       ...

   Reuse, don't fork, the job runner. Each swept value is an ordinary _Job — same _run, same state/metrics files, same error handling. The only new thing is when you press start.

4. Group runs under a sweep id
   modulearn/server.py — write a marker into each run's config

   So the frontend can find "the runs from this sweep," stamp each run with a shared sweep_id and its swept value (add them to config.json, or a small sweep.json at the runs root). /api/runs can then surface the grouping.

5. Build the sweep UI
   modulearn/static/graph.js — toolbar + a small sweep dialog

   Let the user choose a wired hyperparameter (read them off the current serialize() / the compiled config), enter values or a min/max/steps the frontend expands, and POST to /api/graph/sweep. On success, feed the returned run_ids straight into compareRuns() from the overlay feature so the fan renders as it fills in.

   async function sweep(param, values) {
     const r = await fetch("/api/graph/sweep", {
       method: "POST", headers: { "content-type": "application/json" },
       body: JSON.stringify({ ...serialize(), param, values }),
     });
     const j = await r.json();
     if (r.ok) compareRuns(j.sweep.run_ids);     // overlay the whole fan
     else setPanel("err", "✗ sweep rejected: " + ((j.detail&&j.detail.errors)||[j.detail]).join(", "));
   }

6. Verify a small sweep end-to-end

   modulearn demo
   # sweep lr over [1e-4, 1e-3, 1e-2]
   curl -s -X POST localhost:8000/api/graph/sweep -H 'content-type: application/json' \
     -d '{"nodes":[...],"links":[...],"param":"lr","values":[0.0001,0.001,0.01]}'
   # expect: {"sweep":{"param":"lr","run_ids":[...3 ids...]}}
   curl -s localhost:8000/api/runs | python -m json.tool   # 3 runs appear, filling in over time
   # in the editor: the overlay shows 3 val curves, one per lr

   Prove the guards. Sweep a param that isn't wired → 400 with a clear message. Sweep a value outside the declared min/max → rejected, not launched. A sweep that launches an impossible run is worse than no sweep.

7. Test the coordinator
   tests/test_sweep.py

   With a toy fast on_train, assert a sweep of N values produces N run dirs each with the overridden value in config.json, never more than k live at once, and all settle to done. TestClient (from the server-endpoint tests) makes the HTTP side easy.

8. Commit + mark shipped

   git add modulearn/server.py modulearn/static/graph.js tests/test_sweep.py
   git commit -m "Hyperparameter sweep: fan a graph over one knob, overlay the fan"
   git push
   gh run list --branch main --limit 1   # expect: success

   Then check the {area:"features", … title:"Hyperparameter sweep"} card's box in docs/roadmap.html (shipped state is per-browser).

Worth knowing