AgentTwin scoreboard

How well npx agenttwin works on code we did not write, measured and published as it stands. Each number comes from a run anyone can repeat with the commands at the end. We publish the misses too.

Known money-losing bugs

8 bugs, each a small app in two versions: the bug as people write it, and its fix. Measured 2026-11-17.

Measured Goal
Bugs caught 100% 80%
False alarms on the fixed version 0% under 10%

Read this with care: we wrote these bugs, and we fixed AgentTwin against them. The first run on 2026-11-17 caught 50% with 12.5% false alarms. The fixes are general (no bug-specific code), but a library we wrote is a floor, not proof. The plan is 50 bugs, including ones taken from public incident reports.

Bug Caught Fixed version
A charge retried after a timeout, without an idempotency key Yes Clean
A partial refund retried after a timeout, without an idempotency key Yes Clean
A customer created again on retry Yes Clean
A subscription created again on retry Yes Clean
A charge retried in a loop on server errors, without an idempotency key Yes Clean
A fulfilment request posted to Slack again on retry or a repeated request Yes Clean
A payment webhook that fulfils the order on every delivery Yes Clean
A partial refund made twice in one request (an old refund call left in after a refactor) Yes Clean

Stranger repos

A seeded random draw of 15 active public TypeScript repos, run untouched with npx agenttwin --yes. Measured 2026-11-17, 0.3.30 before Stage 1 (PR #85).

Outcome Repos
Clean run and faults 0
Library: correctly nothing to test 4 (docx, fedify, kysely, nuxt-mail)
Ran, nothing checkable (said so) 1 (chanhdai.com)
Could not install on this machine 4 (novex-kit, CoWork-OS, graphile/crystal, nodejs.org)
Stopped with an error 6 (opennextjs-netlify, growi, prisma/orm, nao, fumadocs, vltpkg)

0 of 15 reached a clean run. The goal is 70% of backend repos.

The 6 error repos re-run on the Stage 1 branch: every run now ends in a plain outcome, none with a stack trace.

Repo Outcome now
getnao/nao Couldn't run: a Bun app (now explained)
prisma/orm Partly tested
opennextjs-netlify Partly tested
fumadocs Couldn't run: workspace package build (scoped build added after)
vltpkg Couldn't run: wrong route picked (ranking fixed after)
growi Couldn't run: helper picked (route candidates fixed after)

The Stage 1 bar is measured on a fresh draw of 20 backend repos (backend-sample.txt), not yet run.

Repeat it

git clone https://github.com/AgentTwin/AgentTwin && cd AgentTwin/packages/cli && npm ci
AGENTTWIN_BUG_LIBRARY=1 npx vitest run test/bug-library.test.ts

The repo lists are in artifacts/scoreboard. In each repo, run npx agenttwin --yes.