Research

Compile Once, Govern Every Repair

Deterministic Replay for Repeated GUI Work

Reasoning through a known workflow on every run is wasteful and unsafe. It adds latency and cost, and—because a rendered “Saved” banner is not a persisted write—it can report success after a partial, duplicate, stale, or rejected business effect. OpenAdapt compiles one recorded demonstration into a deterministic program. The default healthy replay makes no generative-model API calls. Under drift, retained evidence re-resolves targets or supports a governed repair proposal. The runtime verifies effects against the application's own system of record and refuses the result when it cannot verify it.

The headline result

We injected ten transaction-fault classes behind a real HTTP boundary and replayed each nine times per arm—90 runs per arm, end to end through the actual replayer into an on-disk SQLite system of record, graded by a direct read-only database connection that bypasses the service entirely. Judged by the screen—the same signal a computer-use agent or an RPA script trusts—replay silently accepted 75.0% of the wrong effects that actually occurred. Adding one out-of-band oracle that reads the system of record cut that to 12.5%. That middle rung is the number a real deployment ships.

All nine residual misses are one named class: a collateral write to a surface the oracle's read path does not cover. An out-of-band oracle catches exactly what its read path can read, and widening that path closes the gap. Full method, per-fault outcomes, and the closed-world caveat on the 0 are in the end-to-end effect study.

How to read this pack

The 90-run injected-fault study above is the current method result. It names the task, the sample size, the oracle, and the silent-incorrect-success rate. The OpenEMR and MockMed rows below are a 2026-07-08 field comparison. They used saved-row OCR, not a stored-record read, and they do not report silent-incorrect-success or over-halt. The 29-application corpus is one recording and one replay per app: failure discovery, not a reliability rate. One published customer outcome exists. It's a related-party founding deployment, disclosed on the case study.

Reproduce the screen-vs-record mismatch locally with openadapt-flow tutorial --break-it. The bundled backend paints a success banner and rejects the write. The independent read of the system of record refuses, and the run halts. The engine README is the source for that command.

Next measurements, owned, not yet published: a WorkArena extra-field census whose inventory is still running, so we won't quote a rate from it; a release-bound evals campaign against the Flow version currently on PyPI; an OpenEMR re-measure that reads the stored record and counts silent-incorrect-success and over-halt; an independent customer outcome beyond the related-party founding deployment.

RVU audit case study, with the related-party disclosure on the page.

What the paper reports

• Historical field comparison, not current-release reliability. On the historical OpenEMR field task, saved-row OCR passed 19/20 compiled runs. The agent arm had a higher pass rate in its smaller sample. The compiled median was 39.2 s. Those compiled runs recorded zero model API calls. The agent median was 70.4 s with $0.55 in estimated model API charges per run. The field oracle did not read the stored record. Under the 2026-07-08 OCR check, the retained MockMed rows marked 100/100 compiled and 20/20 agent attempts as passes. The recorded medians were 4.9 s and 37.5 s, respectively. Final frames were not retained for current verifier rescoring. Use that comparison for latency and estimated model API charges, not current correctness.
• One-shot public-web breadth. Both comparisons above are one rehearsed task. On a 29-application public-web corpus, each application recorded once and replayed once against an unchanged UI, all 29 compiled, 17 replays reached a verified success, 10 halted safely, and 2 reported success while the external oracle disagreed. One observation per application is failure discovery, not a generalization rate. Method and caveats are in the reliability study. A qualified workflow is one sealed bundle version with an admission.
• Repair under drift. When a theme re-render invalidated every recorded template crop, compiled replay re-resolved every target in 9.7 s with 8 target repairs and recorded zero model API calls, while the same agent under the same drift took 87.4 s and $0.63.
• Identity before action. Against adversarial look-alike records (O/0, l/1), the identity ladder recorded zero false accepts in every tested configuration, halting rather than clicking a confusable target. Availability is what pays for that. Structured browser identity over-halted on none of the 14 correct homonym cases, but the pure-pixel configurations halted on all of them: zero false accepts at a 100% over-halt rate. Pixel-only identity is not ready for a dense clinical list.
• One governed contract, many substrates. The same semantics drive browser, native Windows UI Automation, native macOS, and real-network RDP tasks, each with an independent effect oracle and explicit refusal controls.

Reproducibility

Every headline number in the paper is bound by a machine-check to a released raw benchmark file: the build fails if a constant in the text drifts from its artifact. The implementation, the raw run data, and the failure taxonomy are published together in the openadapt-flow repository. This is a technical report; see the paper for the exact scope of each measured result and its limitations.

← Back to openadapt.ai