Compiled Replay vs. Computer-Use Agent
One clinical task, two ways to automate it, one success check.
The wedge in one line: for a task you have already demonstrated, a compiled
replay is model-free, ~$0 per run, and
faster on these measured trials. Each arm's success is
reported separately. Generated
straight from the repo's real benchmark results.json files; no
number here is hand-typed.
What this is. The economics of the 500th run of a known task: once a workflow has been demonstrated once, replaying it should not cost a model call.
What this isn't. A capability claim. Compiled replay is the wrong tool for a task nobody has automated yet — exploring an unfamiliar screen is exactly the agent's job. The agent explores; the compiled bundle exploits a demonstration.
Both arms drive the same vision-only interface against the same
app; the only difference is whether a model is in the per-action loop.
Reproduce with python -m benchmark.comparison_artifact.generate.
OpenEMR public demo — a real third-party EMR
OpenEMR public demo: log in, search the demo patient, open the chart, scroll the dashboard to the Messages card, open Patient Messages, add a parameterized note, save · 18 compiled steps
| modeled runs | compiled model API | agent model API |
|---|---|---|
| 1 | $0 | $0.5522 |
| 100 | $0 | $55.22 |
| 1,000 | $0 | $552 |
| 10,000 | $0 | $5,522 |
Agent values multiply the measured per-run model charge ($0.5522) at list price; an introductory $2/$10 rate applies through 2026-08-31. The compiled column projects the sample's zero model calls only. It excludes authoring, maintenance, compute, storage, review, and exception-handling cost and assumes the measured task and rates remain unchanged. The one-time demonstration was about a minute of human demonstration.
Both arms drive the same vision-only interface
(screenshots in; pixel clicks / typed text / scrolls out) against the
same target; the agent is claude-sonnet-5 with the
computer-use tool, prompted with user intent, not steps. Success is one
arm-independent OCR check applied to both arms — neither arm's
self-report is trusted. Not CI-reproducible: a single shared public instance that every internet visitor mutates and that resets daily. Treat as a field result.
figures: benchmark/openemr/results.json
One compiled run (#20) self-flagged expected-screen drift at step_017 and aborted. The corrected saved-row OCR check also rejects it: the retained final frame shows the note in the unsaved entry form, not in a saved message row. It counts as a failure.
MockMed — the bundled demo clinic (CI-reproducible anchor)
MockMed triage: login, open first referral, create a Triage encounter, enter the note, save
| modeled runs | compiled model API | agent model API |
|---|---|---|
| 1 | $0 | $0.2716 |
| 100 | $0 | $27.16 |
| 1,000 | $0 | $272 |
| 10,000 | $0 | $2,716 |
Agent values multiply the measured per-run model charge ($0.2716) at list price; an introductory $2/$10 rate applies through 2026-08-31. The compiled column projects the sample's zero model calls only. It excludes authoring, maintenance, compute, storage, review, and exception-handling cost and assumes the measured task and rates remain unchanged. The one-time demonstration was about a minute of human demonstration.
Both arms drive the same vision-only interface
(screenshots in; pixel clicks / typed text / scrolls out) against the
same target; the agent is claude-sonnet-5 with the
computer-use tool, prompted with user intent, not steps. Success is one
arm-independent OCR check applied to both arms — neither arm's
self-report is trusted. Anyone can rerun this deterministically — same orchestrator, same agent harness, same style of OCR success check, on a local app.
figures: benchmark/results.json
Under a hostile theme swap (?drift=theme, which invalidates every recorded template crop), compiled still succeeded in 9.7 s with 8 self-heals; the agent succeeded in 87.4 s at $0.6319. This drift row is n=1 per arm — an existence result, not a rate.
Read before quoting these numbers
This comparison would be dishonest without the caveats it does not
wave away. Pulled straight from the two BENCHMARK.md methodology
sections:
- Small N, wide error bars.
The agent arm is N=10 on OpenEMR and N=20 on MockMed, because agent runs cost real money, real minutes, and real load on a shared public service. A 100% success rate over ten runs is not a five-nines claim — its confidence interval is wide. The compiled arm (N=20 / 100) is cheap to repeat, so its bars are tighter, but the honest comparison is still small-sample on the agent side.
- The lead result is a field result, not CI-reproducible.
OpenEMR is a single shared public demo that anyone on the internet can mutate and that resets daily; every successful run also appends a message that grows the dashboard for the next run. Numbers depend on that instance's state and load on the day. The MockMed row is the reproducible anchor — treat OpenEMR as a real-world sighting, not a repeatable measurement.
- Cost is list price with hard caps — billed cost is lower.
Costs are computed from API token counts at list pricing. list price; an introductory $2/$10 rate applies through 2026-08-31, so the amount actually billed today is about a third lower than the figures shown. Every agent run is capped at $1.50 and the whole agent arm at $8.00 (list price); the caps stop the arm and the truncation is disclosed in the source data rather than hidden.
- Success is corrected saved-row screen evidence, not a record read.
The legacy whole-frame OCR check accepted one compiled run whose note was still in the unsaved entry form. The corrected contract requires the note in a saved Patient Messages row, and replay of all 30 retained final frames changes only that run. OCR can still miss dense text. This field result does not read the OpenEMR system of record out of band.
- This measures bounded outcomes, cost, and latency — not capability.
The OpenEMR outcomes are 19/20 and 10/10 under one corrected screen contract. The result shows the measured cost and latency difference. It is NOT a claim that compiled replay is more capable, more robust to novel situations, or a substitute for an agent on a task it has never seen. MockMed is also a deliberately simple app; harder surfaces would slow and likely degrade both arms, plausibly at different rates.
Bottom line: on these measured trials, the compiled path removes the model from the loop — $0 and 1.8× faster on the real EMR, and a tighter, reproducible version of the same gap on MockMed. That is a cost/latency result on known tasks, disclosed with its limits — not a general capability claim.