Proof

What your AI agent writes wrong, and what OpenAdapt catches

We ran AI agents on data-entry tasks in open-source apps and broke some of their writes on the way. Each card counts one thing twice, for the same runs: with the agent alone, and with OpenAdapt checking each write against the app's own records.

  1. Computer-use agents in a clinical web appClaude Sonnet 5.5 · GPT-6.1-sol · Fara-7BWrong “done” claims that passed unflaggedAgent alone28 · 33 · 20With OpenAdapt3 · 3 · 0
  2. API agents on four apps OpenAdapt had never seenClaude Sonnet 5.5 · Codex · Llama 3.1 8BHarmful writes that passed as doneAgent alone1 · 14 · 4With OpenAdapt0 · 0 · 0
  3. Shadow mode beside an agent creating recordsMedplum, scripted agent · Medplum, Claude Sonnet 5.5 · Dolibarr, customer createsHarmful writes that nobody reportedAgent alone18 · 9 · 8With OpenAdapt0 · 0 · 0

Synthetic records on open-source apps, Oct and Nov 2026. Each number is for one agent or one setup, never added across tests. Sources · Try the live catch demo

Interactive walkthrough

Watch an AI agent report done on a bad save, then catch it live

Step through a scripted synthetic vendor-invoice sample (INV-4082 / northwind.example) where a planted fault blanks one field during Save. Compare what an agent alone reports against independent read-back, then verify the receipt signed with a synthetic demo Ed25519 key directly in your browser.

Agent reports done; OpenAdapt catches the mismatch and halts. Synthetic invoice data only.

Agent alone

Reported done (bad save)
  • Invoice IDINV-4082
  • VendorNorthwind Supplies
  • Amount$4,250.00
  • Remittance email(empty in database)
Reported: "Saved successfully!" — moved to next invoice.

With OpenAdapt

STOPPED · receipt signed
  • Invoice IDINV-4082 (matched)
  • VendorNorthwind Supplies (matched)
  • Amount$4,250.00 (matched)
  • Remittance emailexpected ap@northwind.example, read ""
STOPPED (effect_refuted): remittance_email is empty in database. Stopped before downstream step and signed receipt.

Signed record/run/v1 DSSE receipt (WebCrypto Ed25519)

Runs crypto.subtle.verify('Ed25519', ...) locally in your browser against the signed DSSE envelope (zero network calls, demo key seal-ed25519-2fb5335a79897786).

1

Computer-use agents in a clinical web app

Each agent used the web app that ships with Medplum 5.2.8 like a person would, on 37 data-entry cases, 3 times each: register a patient, record an allergy, book an appointment. A proxy broke some of the saves on the way. After each run, a check of the server's records decided whether the agent's answer was true.

  • Claude Sonnet 5.5 Anthropic computer use

    Claude Sonnet 5.5 did the 37 cases in Medplum's web app, 111 runs in all, while a proxy broke some of its saves. It said done 28 times when the record was wrong.

    Wrong “done” claims that passed unflagged

    Agent alone
    28
    With OpenAdapt
    3

    OpenAdapt flagged 25 of the 28 for a person.

    Right runs OpenAdapt also flagged: 26 of 72

  • GPT-6.1-sol OpenAI computer use

    GPT-6.1-sol did the 37 cases in Medplum's web app, 111 runs in all, while a proxy broke some of its saves. It said done 33 times when the record was wrong.

    Wrong “done” claims that passed unflagged

    Agent alone
    33
    With OpenAdapt
    3

    OpenAdapt flagged 30 of the 33 for a person.

    Right runs OpenAdapt also flagged: 11 of 68

  • Fara-7B Microsoft, run locally

    Fara-7B did the 37 cases in Medplum's web app, 110 runs in all, while a proxy broke some of its saves. It said done 20 times when the record was wrong.

    Wrong “done” claims that passed unflagged

    Agent alone
    20
    With OpenAdapt
    0

    OpenAdapt flagged 20 of the 20 for a person.

    Right runs OpenAdapt also flagged: 0 of 2

  • Flagged by OpenAdapt
  • Passed unflagged with OpenAdapt

Each wrong claim OpenAdapt missed was a stray record written after its check ran. In this test OpenAdapt checked each run after the agent finished and flagged runs for a person; it did not block.

2

API agents on four apps OpenAdapt had never seen

Each agent did the same 132 tasks through the APIs of InvenTree, Snipe-IT, Kimai and OpenMRS: add a record, or change one. Separately, we planted faults in the same apps with no agent. OpenAdapt ran as a blocking check: a write it could not confirm was held for a person instead of passing as done.

  • Claude Sonnet 5.5 claude-sonnet-5-5

    Claude Sonnet 5.5 did 132 add-or-change tasks through the APIs of InvenTree, Snipe-IT, Kimai and OpenMRS. Its own mistakes were the harmful writes.

    Harmful writes that passed as done

    Agent alone
    1
    With OpenAdapt
    0

    OpenAdapt named the exact problem for 1 of the 1.

    Correct writes OpenAdapt also stopped: 111 of 131

  • Codex gpt-5.6-sol

    Codex did 132 add-or-change tasks through the APIs of InvenTree, Snipe-IT, Kimai and OpenMRS. Its own mistakes were the harmful writes.

    Harmful writes that passed as done

    Agent alone
    14
    With OpenAdapt
    0

    OpenAdapt named the exact problem for 7 of the 14 and stopped the other 7 for a general reason.

    Correct writes OpenAdapt also stopped: 101 of 118

  • Llama 3.1 8B llama3.1:8b

    Llama 3.1 8B did 132 add-or-change tasks through the APIs of InvenTree, Snipe-IT, Kimai and OpenMRS. Its own mistakes were the harmful writes.

    Harmful writes that passed as done

    Agent alone
    4
    With OpenAdapt
    0

    OpenAdapt named the exact problem for 3 of the 4 and stopped the other 1 for a general reason.

    Correct writes OpenAdapt also stopped: 15 of 16

  • Planted faults no agent

    We planted faults directly in the same four apps, with no agent involved. 87 were harmful writes of the kind the check covers.

    Harmful writes that passed as done

    No check
    87
    With OpenAdapt
    0

    OpenAdapt named the exact problem for 62 of the 87 and stopped the other 25 for a general reason.

    Correct writes OpenAdapt also stopped: 14 of 16

  • Stopped, exact problem named
  • Stopped for a general reason
  • Passed as done with OpenAdapt

170 of OpenAdapt's 241 stops on correct writes came from a column the app changes on every call, such as a last-login time. Separately, 1 planted harmful write on Snipe-IT, made after the check ran, came back confirmed.

3

Shadow mode beside an agent creating records

Shadow mode sits between the agent and the app. It passes every write through unchanged, reads the app's records after each one, and reports each write that landed differently from what the agent sent. A proxy behind it broke some writes. Each setup is a separate result.

  • Medplum 5.2.8 scripted agent

    A scripted agent created patients, allergies and appointments in Medplum 5.2.8 through its FHIR API, while a proxy broke some of the writes.

    Harmful writes that nobody reported

    Agent alone
    18
    With OpenAdapt
    0

    OpenAdapt reported 18 of the 18.

    Correct writes OpenAdapt also reported: 0 of 13

  • Medplum 5.2.8 Claude Sonnet 5.5

    A Claude Sonnet 5.5 agent created patients, allergies and appointments in Medplum 5.2.8 through its FHIR API, while a proxy broke some of the writes.

    Harmful writes that nobody reported

    Agent alone
    9
    With OpenAdapt
    0

    OpenAdapt reported 9 of the 9.

    Correct writes OpenAdapt also reported: 4 of 22. Each was saved as the wrong record first, then fixed by the agent.

  • Dolibarr 21.0.4 customer creates

    An agent created customers in Dolibarr 21.0.4 through its REST API, while a proxy broke some of the writes.

    Harmful writes that nobody reported

    Agent alone
    8
    With OpenAdapt
    0

    OpenAdapt reported 8 of the 8.

    Correct writes OpenAdapt also reported: 0 of 5

  • Reported by OpenAdapt
  • Unreported with OpenAdapt

Shadow mode made 0 writes of its own in Medplum's audit trail.

4

What it costs

Setup

10 to 17 min

We set up OpenAdapt with you on your system. On the four new apps it took 10 to 17 minutes and 9 to 15 decisions by a person per app. Shadow mode on Medplum took 76.5 s of machine time and 3 decisions.

Time added per write

15.2 to 292 ms

Median time shadow mode added to each write, by setup: 15.2 ms (Medplum 5.2.8, scripted agent), 25.3 ms (Medplum 5.2.8, Claude Sonnet 5.5), 292 ms (Dolibarr 21.0.4 and customer creates).

Price

$0 locally

Running OpenAdapt locally is free, a hosted run is $0.05 when verified, Builder is $79 a month, Team is $499 a month, and regulated and enterprise qualification sprints start at $15k.

Pricing · Security

Run it on your own agent

The public Medplum benchmark gives your agent 37 clinical data-entry tasks against a local FHIR server, breaks some of its writes on the way, and scores each one with and without OpenAdapt. It runs on your machine with synthetic patients. Run the steps in order, in one terminal.

You need Docker (or Podman or Colima), git, and Python 3.10, 3.11 or 3.12.

  1. Get the benchmark: the 37 cases and the local Medplum server setup.

    git clone --depth 1 https://github.com/OpenAdaptAI/OpenAdapt && cd OpenAdapt
  2. Install the OpenAdapt CLI in a fresh environment.

    python3.12 -m venv .venv && . .venv/bin/activate && pip install openadapt
  3. Check your setup with the reference agent. It uses no model and costs nothing.

    openadapt bench medplum --agent scripted --down
  4. Score your own agent. It reads each task as JSON on stdin and prints its claim as the last line.

    openadapt bench medplum --agent "cmd:python my_agent.py" --down
  5. Read the scorecard: harms caught, harms missed and correct writes stopped, with and without OpenAdapt.

    cat bench-medplum-results/SCORECARD.txt

Run it beside your agent for a week.

In a shadow pilot, we set up OpenAdapt with you on one workflow where your agent creates records through your system's API. It checks each create and passes every write through unchanged. At the end of the week, the report shows each write that landed differently from what your agent sent.

Sources

  1. Write leaderboard on Medplum 5.2.8, 37 cases, 3 runs per case per agent, with OpenAdapt's check replayed on the same runs: packages/flow/benchmark/fhir_medplum/leaderboard/REPORT.md in OpenAdaptAI/OpenAdapt, commit 7110207cd (sha256 95266395a0b811b8ac96cbdd18f1daf8e01d1d77a5ab934709494bed5b2df04e).
  2. Shadow mode on live Medplum 5.2.8 and Dolibarr 21.0.4, 2026-10-31: packages/flow/benchmark/fhir_medplum/public/SHADOW_RESULTS.md in OpenAdaptAI/OpenAdapt, commit e3117c14b (sha256 d513a137196dac049b72f940160da68b4f7ec6778b1f247562472bf4661f62fe).
  3. First-contact retest of the write verifier, 2026-10-31: 396 agent tasks (Claude Sonnet 5.5, Codex gpt-5.6-sol and Llama 3.1 8B, 132 each) and 116 planted faults on InvenTree, Snipe-IT, Kimai and OpenMRS, synthetic records. Results file evidence/first-contact-retest-2026-10/results/score_all.json (sha256 11e6736ee9eb3e3808610dd18cbf729ea0af1414218946d5d40e2d419555206b) and report evidence/first-contact-retest-2026-10/REPORT.md (sha256 3cf23312d95addf9dc63a70af0c9b21669d23ed5233657b82c5c0a55a430fe9c), commit d807c5e8. Pre-registered in evidence/first-contact-retest-2026-10/PREREG.md (sha256 518053f5e6df81e454d899a79270201cb347a03e3f1943c28f60e2d7490519c4) before any run. Verifier frozen at OpenAdapt 76cd4fd4f.
  4. Planted fault sets, frozen before the runs and kept private (sha256 of each faults.yaml): InvenTree 29 faults 0e618791f58f1bc9e312ee8d4a8dc63bcc8d783b48c6e23bcaef39bf8b680414; Snipe-IT 29 faults 451d7ab02708303c3a6958c2572fa48bfd663e5f095af7ce3effce715fb4b37a; Kimai 31 faults ab66a37869a0304270eb135cc515e3138fa4cfba81afd15dbf72c46f5174dd1b; OpenMRS 27 faults f5a2fa831ab9e9c37522ebb2fff02dbbab45e3b7499a5d9575e8a6d4bced58fe.
  5. Medplum write benchmark, MIT-licensed, 37 cases: packages/flow/benchmark/fhir_medplum/public in OpenAdaptAI/OpenAdapt.