# What your AI agent writes wrong, and what OpenAdapt catches

> Wrong “done” claims that passed unflagged, for computer-use agents in a clinical web app: 28, 33 and 20 with the agent alone, 3, 3 and 0 with OpenAdapt. Harmful writes that passed as done, for API agents on four new apps: 1, 14 and 4 alone, 0 with OpenAdapt, which also stopped 227 of 265 correct writes.

- Canonical: https://openadapt.ai/proof
- Updated: 2026-11-22

## 1. Computer-use agents in a clinical web app

Each agent used the web app that ships with Medplum 5.2.8 like a person would, on 37 data-entry cases, 3 times each: register a patient, record an allergy, book an appointment. A proxy broke some of the saves on the way. After each run, a check of the server's records decided whether the agent's answer was true.

| Agent | Runs | Wrong "done" claims (agent alone) | Passed unflagged (with OpenAdapt) | Right runs also flagged |
| --- | --- | --- | --- | --- |
| Claude Sonnet 5.5 | 111 | 28 | 3 | 26 of 72 |
| GPT-6.1-sol | 111 | 33 | 3 | 11 of 68 |
| Fara-7B | 110 | 20 | 0 | 0 of 2 |

Each wrong claim OpenAdapt missed was a stray record written after its check ran. In this test OpenAdapt checked each run after the agent finished and flagged runs for a person; it did not block.

## 2. API agents on four apps OpenAdapt had never seen

Each agent did the same 132 tasks through the APIs of InvenTree, Snipe-IT, Kimai and OpenMRS: add a record, or change one. Separately, we planted faults in the same apps with no agent. OpenAdapt ran as a blocking check: a write it could not confirm was held for a person instead of passing as done.

| Source | Harmful writes (alone) | Passed as done (with OpenAdapt) | Exact problem named | Correct writes also stopped |
| --- | --- | --- | --- | --- |
| Claude Sonnet 5.5 (claude-sonnet-5-5) | 1 | 0 | 1 of 1 | 111 of 131 |
| Codex (gpt-5.6-sol) | 14 | 0 | 7 of 14 | 101 of 118 |
| Llama 3.1 8B (llama3.1:8b) | 4 | 0 | 3 of 4 | 15 of 16 |
| Planted faults (no agent) | 87 | 0 | 62 of 87 | 14 of 16 |

170 of OpenAdapt's 241 stops on correct writes came from a column the app changes on every call, such as a last-login time. Separately, 1 planted harmful write on Snipe-IT, made after the check ran, came back confirmed.

## 3. Shadow mode beside an agent creating records

Shadow mode sits between the agent and the app. It passes every write through unchanged, reads the app's records after each one, and reports each write that landed differently from what the agent sent. A proxy behind it broke some writes. Each setup is a separate result.

| Setup | Harmful writes (unreported alone) | Unreported (with OpenAdapt) | Correct writes also reported |
| --- | --- | --- | --- |
| Medplum 5.2.8, scripted agent | 18 | 0 | 0 of 13 |
| Medplum 5.2.8, Claude Sonnet 5.5 | 9 | 0 | 4 of 22 |
| Dolibarr 21.0.4, customer creates | 8 | 0 | 0 of 5 |

Shadow mode made 0 writes of its own in Medplum's audit trail.

## Interactive walkthrough (scripted synthetic sample)

The [live walkthrough](https://openadapt.ai/proof#live-demo) steps through a scripted synthetic vendor-invoice sample (`INV-4082`, `Northwind Supplies`, `ap@northwind.example`) where a planted fault blanks `remittance_email` during Save while the form returns HTTP 200. It pairs the walkthrough with a `record/run/v1` DSSE receipt signed with a synthetic demo Ed25519 key (`seal-ed25519-2fb5335a79897786`, `role: demo`, payload SHA-256 `9ee4564e6a1930b84c0e9f894ab0d47eaf92ee59673f96a85456f623d468c065`) that readers can verify or tamper-test in the browser.

| Step | Agent alone | With OpenAdapt |
| --- | --- | --- |
| 1. Fill record — Update vendor invoice INV-4082 | Filling fields in the web form | Confirming target record INV-4082 before typing |
| 2. Click Save — HTTP 200 returns while a planted fault blanks one field | Clicked Save. HTTP 200 OK received. | Clicked Save. Running independent database read-back… |
| 3. Outcome — Agent reports done; OpenAdapt catches the mismatch and halts | Reported: "Saved successfully!" — moved to next invoice. | STOPPED (effect_refuted): remittance_email is empty in database. Stopped before downstream step and signed receipt. |

Verified receipt outcome: `STOPPED (effect_refuted)` (synthetic demo key; flipping 1 byte in the envelope payload fails Ed25519 verification closed).

## Sources

1. Write leaderboard on Medplum 5.2.8, 37 cases, 3 runs per case per agent, with OpenAdapt's check replayed on the same runs: [packages/flow/benchmark/fhir_medplum/leaderboard/REPORT.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/main/packages/flow/benchmark/fhir_medplum/leaderboard/REPORT.md) in OpenAdaptAI/OpenAdapt, commit 7110207cd (sha256 95266395a0b811b8ac96cbdd18f1daf8e01d1d77a5ab934709494bed5b2df04e).
2. Shadow mode on live Medplum 5.2.8 and Dolibarr 21.0.4, 2026-10-31: [packages/flow/benchmark/fhir_medplum/public/SHADOW_RESULTS.md](https://github.com/OpenAdaptAI/OpenAdapt/blob/e3117c14b/packages/flow/benchmark/fhir_medplum/public/SHADOW_RESULTS.md) in OpenAdaptAI/OpenAdapt, commit e3117c14b (sha256 d513a137196dac049b72f940160da68b4f7ec6778b1f247562472bf4661f62fe).
3. First-contact retest of the write verifier, 2026-10-31: 396 agent tasks (Claude Sonnet 5.5, Codex gpt-5.6-sol and Llama 3.1 8B, 132 each) and 116 planted faults on InvenTree, Snipe-IT, Kimai and OpenMRS, synthetic records. Results file evidence/first-contact-retest-2026-10/results/score_all.json (sha256 11e6736ee9eb3e3808610dd18cbf729ea0af1414218946d5d40e2d419555206b) and report evidence/first-contact-retest-2026-10/REPORT.md (sha256 3cf23312d95addf9dc63a70af0c9b21669d23ed5233657b82c5c0a55a430fe9c), commit d807c5e8. Pre-registered in evidence/first-contact-retest-2026-10/PREREG.md (sha256 518053f5e6df81e454d899a79270201cb347a03e3f1943c28f60e2d7490519c4) before any run. Verifier frozen at OpenAdapt [76cd4fd4f](https://github.com/OpenAdaptAI/OpenAdapt/commit/76cd4fd4f).
4. Planted fault sets, frozen before the runs and kept private (sha256 of each faults.yaml): InvenTree 29 faults 0e618791f58f1bc9e312ee8d4a8dc63bcc8d783b48c6e23bcaef39bf8b680414; Snipe-IT 29 faults 451d7ab02708303c3a6958c2572fa48bfd663e5f095af7ce3effce715fb4b37a; Kimai 31 faults ab66a37869a0304270eb135cc515e3138fa4cfba81afd15dbf72c46f5174dd1b; OpenMRS 27 faults f5a2fa831ab9e9c37522ebb2fff02dbbab45e3b7499a5d9575e8a6d4bced58fe.
5. Medplum write benchmark, MIT-licensed, 37 cases: [packages/flow/benchmark/fhir_medplum/public](https://github.com/OpenAdaptAI/OpenAdapt/tree/main/packages/flow/benchmark/fhir_medplum/public) in OpenAdaptAI/OpenAdapt.
