OpenAdaptEvals: measure agents and programs, failures included
OpenAdapt Evals is a research toolkit for comparing GUI agents and compiled programs on benchmark tasks. It counts runs that passed but left the record wrong, and needless stops, as well as task success. Our own research on where checkers miss is on the research page.
Updated 2026-10-18
What it does
- Scores runs with a benchmark's own checkers, with the task and checker visible.
- Tracks wrong successes and needless stops next to task success.
- Ties every published result to an exact engine release.
- A dry run shows the cost before you start paid machines or models.
Status
Research toolkit on GitHub. Not part of the paid offer.
See it on your process
Bring one task your team types every day. We'll tell you whether it fits, and what a test on your own system would cover.