Evals: measure agents and programs, failures included
OpenAdapt Evals is a research toolkit for comparing GUI agents and compiled programs on benchmark tasks. It counts runs that passed but left the record wrong, and needless stops, as well as task success. Our own research on where checkers miss is on the research page.
Updated 2026-11-19
What it does
- Scores runs with a benchmark's own checkers, with the task and checker visible.
- Tracks runs that passed but left the record wrong, and needless stops, next to task success.
- Ties each published result to an exact engine release.
- A dry run shows the cost before you start paid machines or models.
Status
Evals is a research toolkit on GitHub. It isn't part of the paid offer.
See it on your process
Bring a task your team types every day. We'll tell you whether it fits and what a test on your own system would cover.