Methodology
Ground truth is real, verified bugs — not the comments a human happened to leave on a PR. A tool is credited for every real finding and penalized only for confirmed false positives.
How a review is scored
Each submission is a bot's review of a case (a code change with known defects). An LLM judge, calibrated against human verdicts, does three things: marks which known defects were found, rules each finding valid / invalid / minor, and grades the severity the bot assigned. Tier-1 defects also ship a hidden harness that proves the bug by running code and verifies a suggested fix by re-running it. From that we get one quality score per case: the balance of how much of the bug's severity weight the bot caught and how clean it kept the review (few false positives). A clean catch scores near 1, catching the bug but spamming false positives scores lower, and a miss scores near 0.
Precisely: the score is the F1 of the bot's precision on the case (valid findings ÷ valid plus invalid; uncertain and minor findings count for neither) and its severity-weighted coverage, then multiplied by 0.9 for every finding judged invalid. A case with no planted defect is scored on precision alone, with the same 0.9 per false alarm. Without that factor, false alarms could hide behind valid findings: five wrong claims next to eight right ones still leave precision near 65%, so a noisy bot that caught the bug could beat a careful one that also caught it. On the October 2026 board that let a bot with 10% false alarms tie one with 3%. A miss still scores 0, and a catch with any number of false alarms still scores above 0, so staying silent never beats catching the bug. False alarms on a case whose bug was missed cost nothing extra, since that case already scores 0.
How we rank: head-to-head
Bots review different, overlapping sets of cases, so raw averages aren't comparable — qodo's score came from the cases it reached, Cubic's from the full set including the hardest. A bot shouldn't look worse for reviewing more. So we rank like a sports league instead. Every case that two bots both reviewed is a match, won by whichever had the better review that case:
| shared case | cubic | greptile | qodo | winner |
|---|---|---|---|---|
| discourse-graphite-6 | 100% | 100% | 0% | cubic |
| discourse-graphite-10 | 75% | 12% | 88% | qodo |
| cal-com-22532 | 100% | 33% | 100% | cubic |
| cal-com-8330 | 50% | 100% | 50% | greptile |
| cal-com-10600 | 0% | 50% | 25% | greptile |
A Bradley-Terry model finds the single rating per bot that best predicts every match outcome. Because a rating only ever depends on the cases a bot shares with an opponent, reviewing more — or harder — cases can never lower it. The Rating on the leaderboard is the result, expressed as a win probability: if this bot and another both reviewed a fresh case, how often would it catch more? 50% is an even match with the field; higher beats it.
Here are the raw head-to-heads the ratings distill — each cell is the row bot's win rate against the column bot on the cases they both reviewed:
| vs → | Drape (Deep Review) | Drape (Review) | Mjolnir (Opus 5.5) | cursor | Claude Code (Opus 5.5) | Mjolnir (Sonnet 5.5) | greptile | Cubic | qodo |
|---|---|---|---|---|---|---|---|---|---|
| Drape (Deep Review) | – | 57% | 60% | 61% | 66% | 67% | 67% | 71% | 67% |
| Drape (Review) | 43% | – | 54% | 55% | 59% | 63% | 61% | 65% | 63% |
| Mjolnir (Opus 5.5) | 40% | 46% | – | 52% | 54% | 58% | 59% | 62% | 61% |
| cursor | 39% | 45% | 48% | – | 53% | 57% | 57% | 60% | 59% |
| Claude Code (Opus 5.5) | 34% | 41% | 46% | 47% | – | 56% | 54% | 57% | 57% |
| Mjolnir (Sonnet 5.5) | 33% | 37% | 42% | 43% | 44% | – | 48% | 54% | 52% |
| greptile | 33% | 39% | 41% | 43% | 46% | 52% | – | 54% | 51% |
| Cubic | 29% | 35% | 38% | 40% | 43% | 46% | 46% | – | 50% |
| qodo | 33% | 37% | 39% | 41% | 43% | 48% | 49% | 50% | – |
Confidence
Some bots have only reviewed a handful of shared cases, so their rating is genuinely uncertain. We resample the cases many times and watch how far each rating moves; the tighter it holds, the higher the Confidence — High when the rating shifts under ~13 points across resamples, Medium under ~20, Low beyond. That's driven by how many head-to-head matchups back the rating, not by what share of the benchmark a bot has completed. A Low bot doesn't have enough cases to trust its number yet — not a verdict that it's bad. (Confidence is about how well-pinned the rating is, not how good it is: a consistently last-place bot can be pinned with High confidence.)
The tiers
Tier 1 — cases we author by hand. Each is a small, self-contained change with a hidden harness that proves the bug by running code, and verifies a suggested fix by re-running it. The hard ones are cross-file: a change that breaks an unchanged consumer elsewhere, invisible to a reviewer that only reads the diff.
Tier 2 — real merged pull requests from open-source projects, with the defect confirmed against the project's own history. Every trace page links the upstream repository and the original PR. Current sources (generated from the live corpus):
| Project | Cases | Languages | Bots reviewing upstream |
|---|---|---|---|
| Cal.com | 11 | typescript | CodeRabbit, Cubic |
| Sentry | 10 | python, typescript | Cursor Bot |
| TaskFlow (Convex) | 10 | typescript | — |
| Grafana | 9 | go, typescript | Copilot |
| Keycloak | 9 | java | — |
| Discourse | 7 | ruby | — |
| PostHog | 6 | python, typescript | Greptile, Mendable |
| CockroachDB | 5 | go | — |
| Kubernetes | 4 | go | — |
| Next.js | 4 | typescript | — |
| Formbricks | 2 | typescript | — |
| NetBeans | 2 | java | — |
| SigNoz | 2 | go | Cursor Bot |
| ComfyUI | 1 | python | — |
| Deno | 1 | rust | — |
| Django | 1 | python | — |
| LVGL | 1 | c | — |
| mpv | 1 | c | — |
| n8n | 1 | typescript | Cubic, Cursor Bot |
| Novu | 1 | typescript | CodeRabbit, Greptile |
| OpenCV | 1 | cpp | — |
| Redis Client | 1 | ruby | — |
| Tauri | 1 | rust | — |
Framework-specific cases
Some Tier-2 cases test more than general reasoning — they test whether a reviewer understands a specific framework's APIs and idioms. The TaskFlow set is built on Convex: catching its bugs requires knowing how Convex queries, mutations, indexing, and auth actually behave, not just reading the diff. The goal is to measure understanding of real libraries and frameworks, not only abstract logic. Django and Rails framework sets are planned.
Diff size
Each case is bucketed by its diff's changed lines: small is up to 100, medium 101 to 500, large over 500. The cutoffs are fixed, so a bucket keeps its meaning as cases are added. Every bot catches less on a large diff, so the By size page and each bot's page report the bot's edge over the other bots on the same PRs, not only its raw coverage.
What the columns mean
| Column | Meaning |
|---|---|
| Rating | The Bradley-Terry win probability above — how often the bot catches more than the average opponent on a shared case. This is what the board is sorted by. |
| Confidence | How tightly the rating is pinned across resampling. Low means too few cases to trust the number yet, not that the bot is bad. |
| Coverage | Severity-weighted recall of known defects: a critical counts 8×, high 4×, medium 2×, low 1×. Missing a critical hurts far more than missing a nitpick. |
| Validity | Precision: valid ÷ (valid + invalid) findings. A finding counts as valid even if it's a real bug we never pre-recorded. Trivial/unconfirmable findings are excluded, never penalized. |
| False-alarms | Confirmed false positives ÷ all findings. Lower is better. |
| Cross-File | Coverage restricted to cross-file cases — bugs where a change breaks an unchanged consumer elsewhere, invisible to a diff-only reviewer. |
| Security | Defects caught ÷ security defects reviewed, where a defect is tagged security when it sits in a security control — authn, authz, sanitization, crypto, secrets or an abuse limit — whether it fails open (an attacker gets in) or fails closed (the control breaks safely but is still wrong). Shown as a count, not a rate, because the security set is a fraction of the board and the denominator is the point: two bots with different denominators were asked different questions. The tooltip carries the severity-weighted rate. A bot that reviewed fewer than 8 of them shows a dash. Note the overlap: several security cases are also cross-file, so the two columns are not independent. |
| Cases | How many cases the bot reviewed, and what share of the case universe. |
| Review time | The median time the bot took per review, over the reviews the board scores. A hosted bot is timed from the pull request opening (or the comment that asked for a re-review) to its last comment, which includes any time the review waited in the vendor's queue. A bot we run ourselves is timed from the start of its review to its end, two to four reviews at a time on one machine, with no queue. The two are close but not strictly comparable. A dash means no times were recorded for the bot. |
Versioning
A bot's standing is built from every case it has reviewed, unioned across runs — so the benchmark can grow over time and a re-review on a newer date wins for that case. Each bot's page shows its trajectory over time.