AgentTwin scoreboard
How well npx agenttwin works on code we did not write, measured and published as it stands. Each number comes from a
run anyone can repeat with the commands at the end. We publish the misses too.
Known money-losing bugs
8 bugs, each a small app in two versions: the bug as people write it, and its fix. Measured 2026-11-17.
| Measured | Goal | |
|---|---|---|
| Bugs caught | 100% | 80% |
| False alarms on the fixed version | 0% | under 10% |
Read this with care: we wrote these bugs, and we fixed AgentTwin against them. The first run on 2026-11-17 caught 50% with 12.5% false alarms. The fixes are general (no bug-specific code), but a library we wrote is a floor, not proof. The plan is 50 bugs, including ones taken from public incident reports.
| Bug | Caught | Fixed version |
|---|---|---|
| A charge retried after a timeout, without an idempotency key | Yes | Clean |
| A partial refund retried after a timeout, without an idempotency key | Yes | Clean |
| A customer created again on retry | Yes | Clean |
| A subscription created again on retry | Yes | Clean |
| A charge retried in a loop on server errors, without an idempotency key | Yes | Clean |
| A fulfilment request posted to Slack again on retry or a repeated request | Yes | Clean |
| A payment webhook that fulfils the order on every delivery | Yes | Clean |
| A partial refund made twice in one request (an old refund call left in after a refactor) | Yes | Clean |
Stranger repos
A seeded random draw of 15 active public TypeScript repos, run untouched with npx agenttwin --yes. Measured 2026-11-17, 0.3.30 before Stage 1 (PR #85).
| Outcome | Repos |
|---|---|
| Clean run and faults | 0 |
| Library: correctly nothing to test | 4 (docx, fedify, kysely, nuxt-mail) |
| Ran, nothing checkable (said so) | 1 (chanhdai.com) |
| Could not install on this machine | 4 (novex-kit, CoWork-OS, graphile/crystal, nodejs.org) |
| Stopped with an error | 6 (opennextjs-netlify, growi, prisma/orm, nao, fumadocs, vltpkg) |
0 of 15 reached a clean run. The goal is 70% of backend repos.
The 6 error repos re-run on the Stage 1 branch: every run now ends in a plain outcome, none with a stack trace.
| Repo | Outcome now |
|---|---|
| getnao/nao | Couldn't run: a Bun app (now explained) |
| prisma/orm | Partly tested |
| opennextjs-netlify | Partly tested |
| fumadocs | Couldn't run: workspace package build (scoped build added after) |
| vltpkg | Couldn't run: wrong route picked (ranking fixed after) |
| growi | Couldn't run: helper picked (route candidates fixed after) |
The Stage 1 bar is measured on a fresh draw of 20 backend repos (backend-sample.txt), not yet run.
Repeat it
git clone https://github.com/AgentTwin/AgentTwin && cd AgentTwin/packages/cli && npm ci
AGENTTWIN_BUG_LIBRARY=1 npx vitest run test/bug-library.test.ts
The repo lists are in artifacts/scoreboard. In each repo, run npx agenttwin --yes.