---
title: "AgentTwin scoreboard"
description: "Measured: 100% of 8 known money-losing bugs caught, 0% false alarms, and how npx agenttwin does on random public repos."
canonical: "https://agenttwin.si/scoreboard/"
---

# AgentTwin scoreboard

How well `npx agenttwin` works on code we did not write, measured and published as it stands. Each number comes from a
run anyone can repeat with the commands at the end. We publish the misses too.

## Known money-losing bugs

8 bugs, each a small app in two versions: the bug as people write it, and its fix.
Measured 2026-11-17.

| | Measured | Goal |
| --- | --- | --- |
| Bugs caught | **100%** | 80% |
| False alarms on the fixed version | **0%** | under 10% |

Read this with care: we wrote these bugs, and we fixed AgentTwin against them. The first run on 2026-11-17 caught 50% with 12.5% false alarms. The fixes are general (no bug-specific code), but a library we wrote is a floor, not proof. The plan is 50 bugs, including ones taken from public incident reports.

| Bug | Caught | Fixed version |
| --- | --- | --- |
| A charge retried after a timeout, without an idempotency key | Yes | Clean |
| A partial refund retried after a timeout, without an idempotency key | Yes | Clean |
| A customer created again on retry | Yes | Clean |
| A subscription created again on retry | Yes | Clean |
| A charge retried in a loop on server errors, without an idempotency key | Yes | Clean |
| A fulfilment request posted to Slack again on retry or a repeated request | Yes | Clean |
| A payment webhook that fulfils the order on every delivery | Yes | Clean |
| A partial refund made twice in one request (an old refund call left in after a refactor) | Yes | Clean |

## Stranger repos

A seeded random draw of 15 active public TypeScript repos, run untouched with `npx agenttwin --yes`. Measured 2026-11-17, 0.3.30 before Stage 1 (PR #85).

| Outcome | Repos |
| --- | --- |
| Clean run and faults | 0 |
| Library: correctly nothing to test | 4 (docx, fedify, kysely, nuxt-mail) |
| Ran, nothing checkable (said so) | 1 (chanhdai.com) |
| Could not install on this machine | 4 (novex-kit, CoWork-OS, graphile/crystal, nodejs.org) |
| Stopped with an error | 6 (opennextjs-netlify, growi, prisma/orm, nao, fumadocs, vltpkg) |

**0 of 15** reached a clean run. The goal is 70% of backend repos.

The 6 error repos re-run on the Stage 1 branch: every run now ends in a plain outcome, none with a stack trace.

| Repo | Outcome now |
| --- | --- |
| getnao/nao | Couldn't run: a Bun app (now explained) |
| prisma/orm | Partly tested |
| opennextjs-netlify | Partly tested |
| fumadocs | Couldn't run: workspace package build (scoped build added after) |
| vltpkg | Couldn't run: wrong route picked (ranking fixed after) |
| growi | Couldn't run: helper picked (route candidates fixed after) |

The Stage 1 bar is measured on a fresh draw of 20 backend repos (backend-sample.txt), not yet run.

## Repeat it

```
git clone https://github.com/AgentTwin/AgentTwin && cd AgentTwin/packages/cli && npm ci
AGENTTWIN_BUG_LIBRARY=1 npx vitest run test/bug-library.test.ts
```

The repo lists are in [artifacts/scoreboard](https://github.com/AgentTwin/AgentTwin/tree/main/artifacts/scoreboard). In each repo, run `npx agenttwin --yes`.
