crbench//

By diff size

Does a bot stay good as pull requests get bigger? Every bot catches fewer bugs on a large PR, so this page compares each bot with the other bots on the same PRs, bucket by bucket.

How to read this

  1. PR size. Each case is bucketed by its diff's changed lines (added + removed): small is ≤100 lines, medium is 101–500 lines, large is >500 lines.
  2. The big number is the bot's coverage: the share of the known bugs it caught, where a critical bug counts 8× and a low one 1×. Beside it is the raw count.
  3. The line under it is what the other bots caught on the exact same PRs, and the gap between the two. +22 means 22 points more than the others.
  4. Trend compares that gap on small PRs with the gap on large PRs. A move of 10 points or more either way gets a label; anything smaller means the bot keeps pace with the field. The trend is relative: a bot can pull ahead on large diffs and still trail the others.

Example: Drape (Deep Review). On small PRs it catches 99% and the other bots catch 85% on the same PRs: a gap of +14 pts. On large PRs it catches 80% against 57%: +23 pts. The gap moves +9 pts, so it drops off about as much as the other bots do as PRs grow: Keeps pace with the field

Bot small
≤100 lines
medium
101–500 lines
large
>500 lines
Trend (gap, small → large)
Drape (Deep Review) 99% 63/64
others 85% · +14 pts
90% 66/74
others 71% · +19 pts
80% 81/98
others 57% · +23 pts
Keeps pace with the field
+14 pts → +23 pts
Drape (Review) 92% 59/64
others 86% · +6 pts
81% 57/74
others 72% · +9 pts
74% 73/98
others 58% · +16 pts
Pulls ahead on large diffs
+6 pts → +16 pts
Mjolnir (Opus 5.5) 88% 55/64
others 87% · +2 pts
78% 55/74
others 72% · +6 pts
64% 57/98
others 60% · +4 pts
Keeps pace with the field
+2 pts → +4 pts
cursor 86% 53/64
others 87% · -1 pts
75% 53/74
others 73% · +3 pts
65% 57/98
others 59% · +6 pts
Keeps pace with the field
-1 pts → +6 pts
Claude Code (Opus 5.5) 93% 59/64
others 86% · +7 pts
78% 60/74
others 72% · +5 pts
70% 69/98
others 59% · +11 pts
Keeps pace with the field
+7 pts → +11 pts
Mjolnir (Sonnet 5.5) 82% 50/64
others 87% · -5 pts
71% 52/74
others 73% · -2 pts
54% 48/98
others 61% · -6 pts
Keeps pace with the field
-5 pts → -6 pts
greptile 85% 53/64
others 87% · -2 pts
65% 47/74
others 74% · -9 pts
43% 45/98
others 62% · -19 pts
Falls behind on large diffs
-2 pts → -19 pts
Cubic 75% 47/64
others 88% · -14 pts
57% 41/74
others 75% · -18 pts
57% 58/98
others 60% · -4 pts
Pulls ahead on large diffs
-14 pts → -4 pts
qodo 81% 49/64
others 88% · -7 pts
62% 46/74
others 74% · -13 pts
32% 33/98
others 63% · -31 pts
Falls behind on large diffs
-7 pts → -31 pts
All bots, pooled87%73%60%

Buckets with fewer than 8 bugs are shown but never set the trend. With a few dozen bugs per bucket, treat a gap that moves 10 to 15 points as a signal worth watching, not a verdict.