Metadata-Version: 2.5
Name: stablemate-groom
Version: 2.0.0
Summary: Local push-driven dashboard for workhorse agent-workflow operator gates — tends the horses while they run.
Project-URL: Homepage, https://github.com/GabrielCpp/stablemate/tree/main/groom
Project-URL: Repository, https://github.com/GabrielCpp/stablemate
Project-URL: Issues, https://github.com/GabrielCpp/stablemate/issues
Author: Gabriel Côté
License-Expression: MIT
License-File: LICENSE
Keywords: agent,claude,dashboard,litestar,operator,preact,workflow
Classifier: Development Status :: 3 - Alpha
Classifier: Intended Audience :: Developers
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Software Development :: Build Tools
Classifier: Topic :: Utilities
Requires-Python: >=3.12
Requires-Dist: jinja2>=3.1
Requires-Dist: litestar[standard]>=2.10
Requires-Dist: markdown-it-py>=3
Requires-Dist: opentelemetry-proto>=1.25
Requires-Dist: platformdirs>=4.0
Requires-Dist: pydantic>=2.0
Requires-Dist: textual>=8.2.8
Requires-Dist: watchfiles>=0.21
Requires-Dist: websockets>=15
Requires-Dist: workhorse-agent
Provides-Extra: test
Requires-Dist: playwright>=1.44; extra == 'test'
Requires-Dist: pytest>=8.0; extra == 'test'
Description-Content-Type: text/markdown

# groom

A local, single-process web dashboard for `workhorse` agent-workflow operator
gates. Run `groom serve` on your host while `author`/`coder` (or any other
`workhorse`-based workflow) runs in the background. `groom` shows every running
workflow, pages you the moment one blocks on an operator gate, and lets you answer
the gate from the browser. The shared `await_operator` node blocks in place rather
than exiting, so the run keeps running. The answer travels over the run's own
workhorse control socket and the run wakes at once. The gate file stays the durable
record, and the fallback channel when nothing is listening.

![groom's run list with a blocked coder run selected: the gate question rendered on the right, an answer typed, ready to send](../docs/services/groom/gui/screenshots/operator-answers-blocked-gate-answer-typed.png)

Beyond the gates, the same dashboard is the fleet's telemetry reader — every run's
spans, per-node timings and error status, filterable without leaving the browser:

![The telemetry view: one run's spans with started/node/duration/status columns, two of them flagged ERROR](../docs/services/groom/gui/screenshots/groom-dashboard-telemetry.png)

## How it works

- Each run pushes its telemetry to `groom` over OTLP. A run on groom's own host also
  advertises its run directory, workspace and pid. `groom` reads both from local disk
  to serve the run's Files and Diff panels and to answer its gates. A run with no
  `groom` listening behaves the same.
- `groom` holds live fleet and browser state in memory and pushes
  JSON state to open browser tabs over a websocket; the browser renders it with
  Preact + htm (the vendored `htm/preact` standalone build — no build step, no
  `node_modules`). Every shape the socket pushes is also fetchable over HTTP, so a
  tab whose socket has gone quiet resyncs instead of going stale.
  Those pushes are edge-triggered — something changed, so tell the
  tabs — which covers everything except the half of a run row that is derived from
  the clock: the liveness dot, `silent 4m`, `in node 12m`. The event that should
  turn a row dead is the run *ceasing* to emit, and an absence cannot be pushed, so
  the run list is additionally re-rendered and broadcast every `GROOM_LIVE_TICK_S`
  (5s), skipped entirely when no tab is connected. That is what makes the dashboard
  safe to leave open and read without refreshing.
  The `/feed` socket is the one socket meant for a program rather than a tab: it
  carries one run's events and a periodic digest, and `groom wait` and
  `groom feed` read it.
  A selected run loads its span summary, recent logs, and metrics from SQLite
  once when its watch subscription opens. A loading indicator covers that initial
  read. The history lists timestamped log records, completed spans, and metric
  observations, including runs with no script logs. The type filter shows up to
  60 recent records per type; an explicit run selection includes older spans beyond
  the fleet’s activity window. Committed OTLP batches update the history in
  memory and push it immediately to the watching tabs; the clock tick only
  refreshes elapsed labels, with no database reads. Tabs share the watched history,
  which is released when the last tab leaves. Reopening the run loads a fresh
  snapshot, and reconnecting restores the subscription automatically.
- Fleet rows and the detail pane project the same received telemetry. Operator and
  machine wait gauges carry their gate path and question; node, activity, elapsed
  time and terminal state come from the run's exports. The freshness label shows
  how old that evidence is. Silence leaves the last reported state visible; groom
  does not infer completion by reading checkpoints, `run.json`, or the host PID.
  A resumed process exports a new session identity, replacing the prior session's
  live values as its telemetry arrives.
  The gate pane shows only the latest question set, in full, including nested
  headings. Questions render as Markdown (`marked`, sanitized with `DOMPurify`
  before insertion since the content is LLM-authored). The automatic attendant
  receives that same latest question set. Earlier exchanges stay in the gate file;
  the pane has no additional context disclosure. Each workflow row can expand a
  `git diff` of its working tree (rendered
  with `diff2html`). All front-end assets are vendored locally; nothing is loaded
  from a CDN at runtime.
- Gates travel over the run's control socket, with the file as the record. An
  answer goes socket-first — `groom` asks the run what it is waiting on and
  delivers the answer on the run's own path spelling; the run persists it into
  the gate file before acknowledging — and falls back to writing the gate file
  from outside only when nothing answers on the control socket.
  Wait telemetry updates native gate discovery, including after groom reconnects.
  Socket questions are queried when submitting an answer so the write uses the
  waiting process's exact path. Machine waits show their pending result without
  an operator answer form.

## Install

groom will publish to PyPI as `stablemate-groom`, because the name `groom` belongs to an
unrelated project. Until that release, install it from a checkout of the
[stablemate](https://github.com/GabrielCpp/stablemate) workspace:

```bash
pipx install ./groom        # isolated CLI on your PATH
```

Requires Python ≥ 3.12. groom is an optional add-on — no base workflow requires it — and
nothing has to be configured on the workflow side: `workhorse` probes for a collector at
the default port when a run starts.

## Usage

```bash
groom serve                       # binds 127.0.0.1:8787 — loopback only, the default
groom serve --host 0.0.0.0       # all interfaces: for runs on another machine (see note)
```

From a checkout of this workspace, prefix each command with `uv run` (`uv run groom serve`)
to use the working tree instead of the installed copy.

> **Binding.** groom binds loopback by default because it has **no
> authentication**, and it answers operator gates. A run on another machine needs
> `--host 0.0.0.0`. That choice prints a one-line exposure warning, and
> `--allow-non-loopback` acknowledges it. Only do that on a trusted network.

## Telemetry collector (OTLP) + AFK alerting

`groom` is also the default local **OpenTelemetry collector** for `workhorse`
runs. The same uvicorn process
and port expose standard OTLP/HTTP receivers — `POST /v1/traces`,
`POST /v1/metrics` and `POST /v1/logs` — so an ordinary run

```bash
workhorse-coder run                # probes the endpoint at start; exports if groom answers
```

(No env var is needed while `groom serve` is up on the default port — workhorse
enables telemetry when it finds a collector listening. Point elsewhere with
`OTEL_EXPORTER_OTLP_ENDPOINT`, or opt out with `WORKHORSE_OTEL=0`.)

It streams node/agent-turn spans, gas/heartbeat metrics, and the log records of the
engine and every node it runs in-process into `groom`. Because a
pushed span carries its own identity, **every run appears with no discovery
step**. Spans and metrics persist in an embedded SQLite file
(`groom.db` in the platform data dir; override with `GROOM_DB`), searchable
from the dashboard's *Telemetry* pane, via `GET /traces?run=…&node=…&status=…&
slower_than=…`, or with raw `sqlite3` queries. Rows older than
`GROOM_RETENTION_DAYS` (30) are pruned at startup and on a periodic tick — but
only for a run whose telemetry is already **archived** to disk (below), so the
window bounds the *database*, not the evidence.

A batch the store cannot take is answered `503` with a `Retry-After`, not `500`: the
exporter re-sends it, where an unhandled error would have said "stored" by omission.
That is a deliberate trade of duplicates for losses — `spans` and `turns` are keyed and
absorb a re-send, `metrics` and `logs` are plain appends and a retried batch can land
twice. Duplicate rows are cosmetic in tables read by recency; a dropped batch is gone.
The connection behind it recycles itself on a SQLite error and retries once, so a
wedged handle heals on the next request instead of at the next restart. Whether any of
that has happened is in `GET /api/state` under `store` — `reopens`, `failures`,
`last_error`, and `ok`, which is false only while a failure is *newer* than the last
statement that worked.

The liveness counters — `workhorse.run.heartbeat`, `workhorse.turn.heartbeat`,
`workhorse.cap_wait.heartbeat` — are **never stored at all**: `insert_metrics`
drops them at the door. They tick every ~10s for every open node, so
on long runs they outgrow everything else combined: in one real store the run
heartbeat alone was 1.77M of 2.21M metric rows, and a later one hit 2.8 GB with
heartbeats ~80% of 18M rows. Nothing reads their history — the alert rules fold
them into memory at ingest, and `groom status` asks the running server for that
in-memory picture over HTTP (`/api/live`) — so persistence bought nothing and
cost most of the file. Rows written before this change drain on age alone: the
never-archived series are swept by the ordinary prune without waiting for an
archive behind them, because there is none. The seven liveness *gauges*
(`turn.active`, `turn.idle_s`, `turn.elapsed_s`, `wait.active`, `wait.elapsed_s`,
`node.active`, `node.elapsed_s`) are stored and read live but not archived, on the
same reasoning: climbing idle on an active turn diagnoses a wedged agent *now*,
while an explicit wait is expected parked work, and neither question is asked of a
run that ended a month ago. `models.UNARCHIVED_DELETABLE_METRICS` names them, in
one place, because what may be deleted without a copy is a rule that has to be
readable rather than inferred.

### Archival: SQLite is the live tier, disk is the permanent one

A run's spans and logs are not lost when they age out. Once a run has been quiet
longer than `GROOM_RETENTION_DAYS`, an archival pass writes everything it holds —
every span, every log, and the four budget metric series — as one
`telemetry.jsonl` beside that run's transcripts, and only then does prune delete
the rows. Write, move, delete, in that order: prune is fail-closed and refuses to
touch a run the archive has not reported back.

```
<data dir>/transcripts/<run-id>/…      live runs, the harvester's only target
<data dir>/archives/<run-id>/          frozen: transcripts + telemetry.jsonl
```

Two roots on one filesystem, so promotion is a single `os.rename`. **An archived
run is frozen** — never re-archived, never resumed, never written again. A second
run claiming a name already under `archives/` is archived under a derived name
(`R-a1b2c3d4`) rather than overwriting it.

The file is one JSON object per line, ordered by `ts`, discriminated by `kind`
(`span` / `log` / `metric`), with line 1 a manifest naming the run and its row
counts. Records carry `run_id`, `node`, `generation` and `seq`, which is exactly
the key the per-node transcripts are classified by — so the telemetry joins the
transcripts with no correlation step. The heartbeats and the liveness gauges never
reach it; they are for the in-memory picture, not for history.

The archive has **no retention of its own** and **no index table**: the set of
archived runs is what is on disk under `archives/`, read by listing it. A sweep
interrupted halfway resumes from the manifest rather than rewriting the file.

`groom cost --run`, `groom loops --run` and `groom profile` answer for an archived run
through `telemetry.db`, a derived index in the run's archive folder. Groom builds it from
`telemetry.jsonl` on the first query and reuses it until the jsonl is newer. The jsonl stays
the source, so the index is safe to delete and costs one rebuild.

```bash
groom archive status         # root, frozen runs, pending, held, last sweep
groom archive ls             # what is frozen (--long for the manifests)
groom archive show RUN       # stream one run's telemetry.jsonl
groom archive now --dry-run  # force a pass; --dry-run writes nothing
```

The pass runs in its own task at `groom serve` launch, seeded already-expired so
a restart archives immediately, then every `GROOM_ARCHIVE_EVERY_S` (6h) for at
most `GROOM_ARCHIVE_RUNS_PER_PASS` (25) runs — oldest first, the rest reported as
pending. Nothing in it may raise into groom's tick.

**The pane shows the runs connected right now.** A month of retention means the
unfiltered strip is mostly runs that ended days ago, and a dashboard is for
watching, so a run card (and its spans — a span table listing a hidden run's nodes
is telemetry from nowhere) appears only while the run is `live` by the same
heartbeat predicate the fleet rows use. History is one tick of *show ended* away
(`GET /traces?show_ended=1`), and searching an explicit `run=` always finds it:
naming a run is asking for that run, finished or not.

**Native runs are first-class dashboard rows, not just telemetry.** A run on
groom's own host advertises its `run_dir`, workspace path, pid, and per-node
`activity` label on the OTLP resource; groom materializes a fleet row from that
(keyed by `run_id`), shows what it is doing ("coder · reviewing ACME-A2JX"), and —
because it shares the host — serves the row's Files/Diff panels and answers its
operator gates straight from the local filesystem (`groom.localfs`). The native test
is self-validating. groom draws the row only for a run whose `run_dir` it can read
locally, so a run whose paths don't resolve on this host never double-lists.

Alert rules run on every ingest plus a periodic tick, and page you through
browser notifications **and** an away-from-keyboard push — configure
`GROOM_NTFY_TOPIC` (posts to `https://ntfy.sh/<topic>`; override the server
with `GROOM_NTFY_URL`) and/or `GROOM_WEBHOOK_URL` (JSON `{"title","message"}`):

| Rule | Fires when | Knob (default) |
|---|---|---|
| STALL | a live run emits **nothing** — no span, no heartbeat | `GROOM_STALL_MIN` (90) |
| STUCK | a live agent turn is silent too long, or deterministic work has sat in one node too long; explicit waits are excluded | `GROOM_STUCK_MIN` (75) |
| CHURN | the same node span completes again and again under an unchanged `labels()` signature — i.e. on the same unit of work. A visit a live reload cut short (`workhorse.cut`) is an interruption, not a repeat, and does not count | `GROOM_CHURN_REPEATS` (5) |
| WATCHDOG | a `watchdog_kill` span event arrives | — |
| GAVE-UP | a give-up node's span arrives | `GROOM_GIVEUP_NODES` (qa_give_up,fix_give_up) |
| ENDED | the run's root span arrives — the run is over, whatever the verdict, and nothing is executing for it now | — |
| BLOCKED | an operator gate opens — the run is parked until someone answers it. A cap wait does not count | — |
| WAITING | that gate is still unanswered later | `GROOM_WAIT_MIN` (30) |

Groom deliberately does not alert on total run age: resumptions reuse a run identity and multi-day
runs are normal. Use workhorse's `WORKHORSE_MAX_RUNTIME_S` when a run needs a hard wall-clock limit.

The crux: workhorse **heartbeats** for as long as its process lives, so silence
and slowness are different observations. STALL means the run stopped emitting
(dead/killed/frozen); STUCK means it is alive and parked. Alerts dedupe per
`(run, rule)`.

ENDED covers the case neither absence rule can, because a terminal is precisely
what retires a run from both of them: the run *finished*. That is the loudest
event on a queue — nothing is executing until someone launches the next run — and
until it existed it was the only one that paged nobody. It fires on every ending,
naming the terminal and the error class, because from outside "it crashed" and
"it succeeded" are the same silence. A resume reuses the run id and clears the
fired set, so the next session's ending pages on its own.

A process killed before it exports its terminal remains in its last reported state
with an aging freshness label. STALL reports the missing telemetry; completion
requires the run's terminal export.

BLOCKED and WAITING cover the mirror image: a run parked on an operator gate is
behaving correctly, so no rule described it, and STUCK skips an open wait by
design. Yet it is the only alert whose subject can end it — the human reading the
page *is* the missing input. Answering the gate closes the wait and retires both.
Run `groom serve` under nohup/systemd as the always-on collector
(the consuming repo's `make groom-serve` target does exactly that).

## Where is my run right now?

Spans export **on completion**, so a run's current node — the only one that
matters when it will not finish — has no row in `spans` and never will while it
hangs. The live picture therefore comes from metrics, which ship on a timer
regardless of span state:

```bash
uv run groom status              # every live run: open node, node age, agent idleness
uv run groom status --run <id>   # one run
uv run groom status --json       # same data, machine-readable
```

Read it like this:

| Reading | Means |
|---|---|
| `alive`, node age small | working normally |
| `alive`, explicit wait | operator/cap/retry wait — parked deliberately |
| `alive`, active turn age large, agent `idle` small | a long but **streaming** turn — healthy |
| `alive`, active turn, agent `idle` large | **wedged** agent/tool/API inside the node |
| `alive`, no turn or wait, node age large | **wedged deterministic work** inside the node |
| no heartbeat (`DEAD?`) | the process is gone — SIGKILL, OOM, crashed host |

## Follow a run: events and a periodic digest

```bash
uv run --all-packages groom feed --run RUN --every 30m
uv run --all-packages groom feed --run RUN --json --since 1792339000.5
uv run --all-packages groom wait --run RUN --json
uv run --all-packages groom wait --run RUN --until blocked,ended
```

`/feed` is a WebSocket on `groom serve` for one run. It carries two kinds of
frame and works with no dashboard tab open. A program or an agent session reads
it. A tab reads `/ws`.

| Query parameter | Meaning |
| --- | --- |
| `run` | Required. The exact telemetry run ID. An unknown ID is allowed: the feed waits for the run to appear. |
| `every` | Digest interval in seconds. Default `1800`. Values below `60` are raised to `60`. |
| `since` | Optional replay cursor, a timestamp. Use the `ts` of the last frame received before a reconnect. |

Every frame carries `type`, `seq` and `ts`. `seq` counts from 1 on each
connection. `ts` is the wall clock when the server sent the frame.

An `event` frame arrives the moment an alert fires. Its `event` field holds
`run_id`, `event`, `node`, `question`, `gate_path`, `terminal` and `message`.
`event` is one of `blocked,ended,stuck,stalled,gave-up,churn,watchdog,died,waiting`.
Machine, capacity and retry waits do not count as operator gates.

A `digest` frame arrives every `every` seconds, even when nothing happened.
Silence is not success: a run that stopped emitting and a run that has nothing
to report look the same without it. Its fields:

- `since` and `until`: the window the digest covers.
- `quiet`: true when the window holds no spans, no events and no log line at
  WARNING or above. A quiet digest needs no action.
- `summary`: one line to read first, such as
  `quiet 14:30-15:00 · in reground-file 47m 00s · turn streaming, idle 3.0s · +$1.20`.
- `standing`: where the run is now. It holds the node and its age, the turn state
  and idle time, the wait kind and age, gas, liveness, the terminal state, and
  the open gate's path and question.
- `window`: what happened between `since` and `until`. It holds per-node visits,
  turns, cost and output tokens, laps per work item, operator gates with their
  outcome, the events that fired, up to ten warnings, and the totals.

On connect the feed sends the conditions already active for the run as `event`
frames, so a run that is blocked before anyone listens is reported at once. It
then sends a digest for the last `every` seconds. With `since`, it sends instead
every event recorded after the cursor, then a digest from the cursor to now. The
server keeps the last 200 events per run in memory. A restart of `groom serve`
empties that history.

`groom feed` prints one line per frame and never exits on its own. `--json`
prints each frame as one JSON line. `--every` accepts `30m`, `10m`, `3600s` or a
bare number of seconds. `--since` takes a timestamp.

`groom wait` is a second client of the same feed. It stays silent during normal
progress and ignores `digest` frames. It returns one event and exits when the
named run is blocked, ended, stuck, stalled, or gave up. `--until` selects a
comma-separated set from `blocked,ended,stuck,stalled,gave-up,churn,watchdog,died,waiting`.
There is no timeout. Ctrl+C cancels the monitor without stopping the workflow.
A run not yet known to the server is watched until it reports a matching
condition. This uses the server's current cache, not an archive search: an
ending evicted from that cache is no longer available. Starting a new waiter
while the same condition remains active returns that condition again.

`--run` takes the exact telemetry run ID. `GROOM_URL` selects the collector, as
for `groom status`. Both CLI and server must support the feed protocol. An older
server is reported as a monitoring error. JSON output of `wait` contains
`run_id`, `event`, `node`, `question`, `gate_path`, `terminal`, and `message`;
unavailable context is an empty string.

| `wait` exit | Meaning |
| --- | --- |
| `0` | A matching event arrived, even if the workflow failed. |
| `1` | The connection or protocol failed. |
| `2` | Invalid arguments. |
| `130` | Cancelled. |

Neither command reconnects on its own. A disconnection is reported instead of
silently crossing a gap in monitoring. `feed --since` is how a caller closes
that gap.

## What was it saying? (`groom logs`)

`status` says *where* a run is; logs say what it was doing on the way there.

```bash
uv run groom logs --run <id>                 # everything, oldest-first
uv run groom logs --run <id> --node select_item
uv run groom logs --level WARNING            # a FLOOR: WARNING + ERROR + FATAL
uv run groom logs --contains "over budget"
```

Deterministic (non-agent) nodes appear here because workhorse **runs them in-process**
and hands each one a per-node `logger`, preserving the node's decisions alongside its
spans and artifacts.

Records carry the same `run_id`/`run_dir` resource as the spans, so a log line
joins to its node span and its on-disk artifacts with no correlation step. They
are correlated by an explicit `node` attribute rather than `trace_id`, which is
zeroes: workhorse never makes its node spans current, so there is no ambient
context for the SDK to attach.

No alert rule fires on logs, deliberately — liveness is already answered by the
heartbeat metrics, and paging on log *content* would mean guessing which strings
are worth waking someone for, per workflow. Logs are for reading once a metric has
told you where to look. They expire on the same window as everything else and
are archived with it — one signal, one clock, one file.

### Test runs are not telemetry

A test suite is not a run anyone comes back to, and on a machine where
`groom serve` is up it is by far the loudest producer: one `make test` of the
workflows suite wrote a six-figure number of spans, burying every real run in
the fleet view. So test telemetry is kept out at both ends — workhorse declines
to auto-enable export from a test process (`WORKHORSE_OTEL=1` still forces it
on), and the receivers drop any record whose `run_dir` is *certainly* a test
dir — one containing `pytest-of-` or `.workhorse-test/`. Neither undoes what an
older producer already wrote:

```bash
uv run groom purge-tests --dry-run   # what would go
uv run groom purge-tests             # delete it, then VACUUM to shrink the file
```

`purge-tests` casts the wider net: it also evicts runs whose dir is a
`tempfile.mkdtemp` scratch dir (`/tmp/tmpXXXXXXXX/…`), which is how the largest
single junk run in a real store got there. That one is a guess — a genuine run
launched from a mkdtemp directory matches too — so it is confined to a command
you run deliberately and can preview with `--dry-run`, rather than to the ingest
path, where the same guess would discard evidence unasked. Runs are identified
by their run dir in both cases, never by name.

## Reading the store afterwards

The same file the dashboard reads answers the after-the-fact questions — what a run cost
and where it looped, what it would have cost on another model set, what a node actually
said, and where the wall clock went:

```bash
uv run groom cost --run <id>          # money and rework per node, with /work turns-per-item
uv run groom prices --run <id>        # re-price the same turns against another model set
uv run groom loops --run <id>         # which review→rework loops converge, and which do not
uv run groom transcript --run <id>    # the reasoning and tool trace prompt.md omits
uv run groom profile --run <id>       # what occupied the wall clock
uv run groom db-path                  # …and the SQLite file, for anything the CLI does not ask
```

Cost coverage is not uniform — it is harness × *provider*, and the two ways of not pricing
a turn (nothing reported, versus a reported `0`) differ in how visible they are. Every
command above says how much of itself is real rather than presenting a partial sum as a
complete one.

Those commands, the schema and its one footgun, and worked SQL are in
[docs/STORE.md](docs/STORE.md).

See `docs/services/groom/` at the repo root for the full design — groom's own OKF book,
queryable with `ostler graph --surface groom`.

## Map

- `groom/`: the package behind the `groom` command: the server, the collector, the store and the dashboard assets.
