Research

Compile Once, Govern Every Repair

Deterministic Replay for Repeated GUI Work

Reasoning through a known workflow on every run is wasteful and unsafe. It adds latency and cost, and—because a rendered “Saved” banner is not a persisted write—it can report success after a partial, duplicate, stale, or rejected business effect. OpenAdapt compiles one recorded demonstration into a deterministic program: healthy replay makes no model calls, a resolution ladder repairs targets when the interface drifts, and the runtime verifies effects against the application's own system of record rather than the screen—refusing when it cannot verify.

Read the paper (PDF)Code & artifacts on GitHub

The headline result

We injected ten transaction-fault classes behind a real HTTP boundary and replayed each nine times per arm—90 runs per arm, end to end through the actual replayer into an on-disk SQLite system of record, graded by a direct read-only database connection that bypasses the service entirely. Judged by the screen—the same signal a computer-use agent or an RPA script trusts—replay silently accepted 75.0% of the wrong effects that actually occurred. Adding one out-of-band oracle that reads the system of record cut that to 12.5%. That middle rung is the number a real deployment ships.

54 / 90

wrong effects silently accepted by screen-only verification — 75.0% of the runs where a wrong effect actually persisted

9 / 90

silently accepted once one out-of-band system-of-record oracle is configured — 12.5%, the realistic deployment number

0 / 90

silently accepted under a complete read path over every mutable surface — the best case under full in-database instrumentation, not the expected field result

0 model calls

on the healthy replay path; models are optional repair tiers, not the controller

All nine residual misses are one named class: a collateral write to a surface the oracle's read path does not cover. An out-of-band oracle catches exactly what its read path can read, and widening that path closes the gap. Full method, per-fault outcomes, and the closed-world caveat on the 0 are in the end-to-end effect study.

What the paper reports

Measured on Flow 0.1.0, 2026-07-08

openadapt-flow 0.1.0 — the version declared at the pinned commit. It predates v0.2.0, the first release tag containing that commit, so these figures describe a pre-release engine build and have not been re-measured on a later release. Open the pinned commit cbec44c.

Compiled replay vs. a computer-use agent. On an already-demonstrated OpenEMR task, compiled replay completed 20/20 runs at a 39.2 s median with no model calls, versus the agent's 70.4 s and $0.55/run. On the bundled CI fixture, 100/100 at 4.9 s versus 37.5 s.
Repair under drift. When a theme re-render invalidated every recorded template crop, compiled replay self-healed in 9.7 s with 8 target repairs and zero model calls, while the same agent under the same drift took 87.4 s and $0.63.
Identity before action. Against adversarial look-alike records (O/0, l/1), the identity ladder recorded zero false accepts in every tested configuration—halting rather than clicking a confusable target.
One governed contract, many substrates. The same semantics drive browser, native Windows UI Automation, native macOS, and real-network RDP tasks, each with an independent effect oracle and explicit refusal controls.

Reproducibility

Every headline number in the paper is bound by a machine-check to a released raw benchmark file: the build fails if a constant in the text drifts from its artifact. The implementation, the raw run data, and the failure taxonomy are published together in the openadapt-flow repository. This is a technical report; see the paper for the exact scope of each measured result and its limitations.

← Back to openadapt.ai