# Write leaderboard

> How often does a computer-use agent say a clinical data-entry write is done when the record says otherwise? Claude Sonnet 5.5 did in 25.2% of 111 runs and GPT-6.1-sol in 29.7%, with overlapping intervals. Fara-7B, run locally, did in 18.2% of 110 runs, and 20 of its 22 claims were wrong.

- Canonical: https://openadapt.ai/proof/write-leaderboard
- Updated: 2026-11-22

## Results

*Ordered by the first column, as pre-registered.*

| Agent | Said done, record wrong | Collateral writes said done | pass^3 |
| --- | --- | --- | --- |
| Claude Sonnet 5.5 (Anthropic computer use) | 25.2% of 111 runs (95% CI 13.5–37.8) | 20 of 20 | 56.8% |
| GPT-6.1-sol (OpenAI computer use) | 29.7% of 111 runs (95% CI 16.2–46.0) | 30 of 30 | 62.2% |
| Fara-7B (Microsoft, run locally) | 18.2% of 110 runs (95% CI 10.0–27.0) | 2 of 4 | 0.0% |

- The Claude and GPT-6.1-sol intervals overlap, so the data do not rank those two.
- Neither of those two said done wrongly on any clean case.
- Fara-7B ran on a Mac at about 46 seconds per model call. In 84 of its 110 runs it reached the 40-minute limit and made no claim. Its lower rate comes from claiming less, not from being right more often.
- Every Claude and GPT-6.1-sol run with a collateral write (another record changed, or a second record) was said done.
- Some injected faults can't be seen from the app's screens at all.

## What was measured

- **Said done, record wrong:** the agent's final RESULT line says DONE and the end-state check's verdict is not correct, over all runs.
- **Collateral writes said done:** of the runs whose end state holds a change beyond the one record asked for, how many the agent said done.
- **pass^3:** the share of cases whose 3 runs were all truthful successes.

## How it was run

There are 37 public cases in the web app that ships with Medplum 5.2.8: 13 patient registrations, 12 allergy records and 12 appointment bookings. Of these, 11 are clean, 25 have a fault on the write path or a write by another user, and 1 needs a person.

Each agent ran each case 3 times in a 1280x800 browser. After each run, an end-state check reads the record on the server. It never uses the agent's claim. The protocol was written, and its hash committed, before any scored run.

## The same runs with OpenAdapt's check

OpenAdapt's check read the server's records for each run after the agent finished. It blocked nothing and the agents never saw it, so it is not part of the ranking above. It flags a run for a person when it cannot confirm the write.

*Wrong “done” claims that passed unflagged: the agent said a case was done, the record was wrong, and nothing flagged it.*

| Agent | Wrong “done” claims that passed unflagged, agent alone | Wrong “done” claims that passed unflagged, with OpenAdapt | Right runs OpenAdapt also flagged |
| --- | --- | --- | --- |
| Claude Sonnet 5.5 | 28 of 111 runs | 3 of 111 runs | 26 of 72 |
| GPT-6.1-sol | 33 of 111 runs | 3 of 111 runs | 11 of 68 |
| Fara-7B | 20 of 110 runs | 0 of 110 runs | 0 of 2 |

Each wrong claim OpenAdapt did not flag was a stray record written after its check ran. Most of the correct runs it flagged are a new record saved in steps, as a create followed by updates, or a value the agent corrected after the fault changed it. OpenAdapt holds these for a person to review.

## Conflict of interest

OpenAdapt wrote the cases, the fault proxy, the end-state check and the harness, and makes the verifier.

## Scope

These results are for one application, Medplum's web app, with injected faults on 37 cases, so the intervals are wide. Each agent ran with its provider's defaults.

## Evidence

- [REPORT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/leaderboard/REPORT.md): the results report.
- [PROTOCOL.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/leaderboard/PROTOCOL.md): the pre-registered protocol.
- [DEVIATIONS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/leaderboard/DEVIATIONS.md): every change from the protocol.
- [README.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/leaderboard/README.md): how to run your own agent.
- [results/](https://github.com/OpenAdaptAI/OpenAdapt/tree/main/packages/flow/benchmark/fhir_medplum/leaderboard/results): every run's row, transcript and screenshots.

## Sources

1. Write leaderboard on Medplum 5.2.8, 37 cases, 3 runs per case per agent: [packages/flow/benchmark/fhir_medplum/leaderboard](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/leaderboard/REPORT.md) in OpenAdaptAI/OpenAdapt, MIT-licensed.
