Leaderboards
Behavioural Failure Detection
Testing agents generate test conversations from high-level task descriptions, without access to the original issue report, failure symptoms, or reproduction steps.
43 issues · 23 target agents · Up to 50 test executions per issue
| Rank | Testing Agent | |||||
|---|---|---|---|---|---|---|
| 1 | CodexAstra 6 | 9.30%4 / 43 | 11.63%5 / 43 | 16.28%7 / 43 | 20.93%9 / 43 | 30.23%13 / 43 |
| 2 | Claude CodeOpus 5.5 | 11.63%5 / 43 | 13.95%6 / 43 | 16.28%7 / 43 | 16.28%7 / 43 | 16.28%7 / 43 |
Detection@k is the percentage of issues whose reported failure is triggered within k test executions, as verified by an independent failure oracle. Higher is better.
Per-issue results
Hit (n) indicates the first test execution that triggered the reported failure. No hit means no confirmed trigger within the test budget.
| ID | Target Agent | Issue | Codex | Claude Code |
|---|---|---|---|---|
| 1 | agentty | #116 | No hit | Hit (1) |
| 2 | claurst | #442 | No hit | No hit |
| 3 | claurst | #448 | Hit (11) | No hit |
| 4 | code-assistant | #276 | No hit | No hit |
| 5 | code-assistant | #277 | Hit (24) | Hit (1) |
| 6 | codewhale | #5546 | No hit | No hit |
| 7 | deepagents-code | #5785 | No hit | No hit |
| 8 | deepagents-code | #5787 | Hit (1) | No hit |
| 9 | deer-flow | #7526 | No hit | No hit |
| 10 | easy-agent | #124 | Hit (23) | No hit |
| 11 | easy-agent | #125 | Hit (21) | No hit |
| 12 | fast-agent | #1039 | No hit | No hit |
| 13 | fx | #1345 | No hit | Hit (1) |
| 14 | gptme | #4490 | Hit (1) | No hit |
| 15 | langchain-code | #17 | No hit | No hit |
| 16 | llxprt-code | #3599 | No hit | No hit |
| 17 | llxprt-code | #3943 | No hit | No hit |
| 18 | mastracode | #14823 | Hit (33) | No hit |
| 19 | mini-agent | #130 | No hit | No hit |
| 20 | minimax-code | #155 | No hit | No hit |
| 21 | mistral-vibe | #1179 | Hit (6) | Hit (1) |
| 22 | mistral-vibe | #186 | Hit (9) | No hit |
| 23 | mistral-vibe | #259 | No hit | No hit |
| 24 | mistral-vibe | #561 | Hit (15) | No hit |
| 25 | nanocoder | #1057 | Hit (2) | No hit |
| 26 | nanocoder | #1058 | No hit | No hit |
| 27 | nanocoder | #1237 | No hit | No hit |
| 28 | nanocoder | #1343 | No hit | No hit |
| 29 | nanocoder | #1391 | No hit | Hit (7) |
| 30 | opencrabs | #2210 | Hit (1) | No hit |
| 31 | opencrabs | #2211 | No hit | No hit |
| 32 | opencrabs | #2212 | No hit | No hit |
| 33 | opencrabs | #2227 | No hit | No hit |
| 34 | openharness | #140 | Hit (1) | Hit (1) |
| 35 | peri | #287 | No hit | No hit |
| 36 | reasonix | #10067 | No hit | No hit |
| 37 | reasonix | #14403 | No hit | No hit |
| 38 | reasonix | #8299 | No hit | No hit |
| 39 | reasonix | #8406 | No hit | No hit |
| 40 | reasonix | #8811 | No hit | Hit (5) |
| 41 | reasonix | #9487 | No hit | No hit |
| 42 | vtcode | #792 | No hit | No hit |
| 43 | zeroclaw | #10912 | No hit | No hit |
Source for Codex and Claude Code: MLSYSR27_AgentBehaviorBench.pdf, Table 13, p. 18. Their scores are calculated from the paper’s per-issue results.