OpenAdapt Evals
Measure the change you made.
OpenAdapt Evals helps developers compare GUI agents and compiled workflows with benchmark verifiers and retained results. Inspect failures as well as completed tasks.
Work with OpenAdapt Evals
What you can do
Use a benchmark verifier
Score the outcome with the environment’s checks. Keep the task and verifier visible when you compare agents or replay methods.
Inspect the failure counts
Track silent incorrect success and unnecessary halts alongside task success. An aggregate score can hide either failure.
Keep results tied to a release
Published evidence identifies the exact Flow wheel and measured environment. Read those pins and the report limits before applying a result to your deployment.
Budget a live evaluation
Use the evaluation CLI’s dry run before provisioning infrastructure. Live benchmarks can incur VM and model costs; the command reference explains the controls.
First steps
Check the harness without a VM.
Install the toolkit and run the mock adapter first. This checks the evaluation setup. Its deterministic success result says nothing about agent reliability on real tasks.
pip install openadapt-evals
openadapt-evals mock --tasks 5