crbench//

← all cases

validate() now skips on a falsy return, and an unchanged handler's validate still returns None, dropping every refund

Tier 1 easy python · cross-file · authored, execution-grounded case

Ground-truth defects

d1 · correctness · high · beacon/handlers/refund.py:12
The diff changes the contract of Handler.validate (beacon/handlers/base.py): it used to raise RecordRejected to reject a record with its return value ignored, and now returns True to keep a record and False to skip it, with run() doing `if not keep: result.skipped += 1; continue`. It moves the five handlers that skip expected traffic (pageview, click, login, email, experiment) to `return True`. RefundHandler (beacon/handlers/refund.py), which the diff never touches or names, also overrides validate: it calls super().validate(record), raises on a bad record and otherwise falls off the end, returning None. run() reads None as "skip", so every valid refund is counted as skipped and never becomes a row: net revenue on the warehouse side silently stops subtracting refunds, with nothing rejected and no error. The handlers are found only through the registry Handler.__init_subclass__ fills, so no call edge leads to refund.py, and the other untouched overriders (signup, purchase, subscription) end with `return super().validate(record)` and are fine, so every overrider has to be read.

The change under review

diff
diff --git a/beacon/handlers/base.py b/beacon/handlers/base.py
index 799f43f..cf8d9df 100644
--- a/beacon/handlers/base.py
+++ b/beacon/handlers/base.py
@@ -28,13 +28,17 @@ class HandlerResult:
     kind: str
     rows: list[Row] = field(default_factory=list)
     rejected: list[tuple[str | None, str]] = field(default_factory=list)
+    skipped: int = 0
 
 
 class Handler:
     """The handler for one event type. Subclass it, set `kind`, and override the hooks
     `run` calls on each record:
 
-    - validate(record) raises RecordRejected to reject the record; its return value is ignored.
+    - validate(record) returns True to keep the record and False to skip it. Skipped records
+      are counted but are not errors: use it for traffic that is expected but not wanted
+      (bots, uptime probes, internal test accounts). Raise RecordRejected for a malformed
+      record; it goes to the rejects table with the message.
     - transform(record) returns the warehouse row for a valid record. Start from
       super().transform(record), which has the columns every event type shares.
 
@@ -48,10 +52,11 @@ class Handler:
         super().__init_subclass__(**kwargs)
         Handler.registry[cls.kind] = cls
 
-    def validate(self, record: Record) -> None:
+    def validate(self, record: Record) -> bool:
         missing = [name for name in REQUIRED if not record.get(name)]
         if missing:
             raise RecordRejected(f"missing {', '.join(missing)}")
+        return True
 
     def transform(self, record: Record) -> Row:
         return {
@@ -66,9 +71,12 @@ class Handler:
         result = HandlerResult(self.kind)
         for record in records:
             try:
-                self.validate(record)
+                keep = self.validate(record)
             except RecordRejected as exc:
                 result.rejected.append((record.get("id"), str(exc)))
                 continue
+            if not keep:
+                result.skipped += 1
+                continue
             result.rows.append(self.transform(record))
         return result
diff --git a/beacon/handlers/click.py b/beacon/handlers/click.py
index faee1a4..6725ee7 100644
--- a/beacon/handlers/click.py
+++ b/beacon/handlers/click.py
@@ -8,12 +8,13 @@ from beacon.handlers.base import Handler, Record, RecordRejected, Row
 class ClickHandler(Handler):
     kind = "click"
 
-    def validate(self, record: Record) -> None:
+    def validate(self, record: Record) -> bool:
         super().validate(record)
         if not record.get("target"):
             raise RecordRejected("click without a target")
         if record.get("synthetic"):
-            raise RecordRejected("synthetic click from the UI test suite")
+            return False  # synthetic click from the UI test suite
+        return True
 
     def transform(self, record: Record) -> Row:
         row = super().transform(record)
diff --git a/beacon/handlers/email.py b/beacon/handlers/email.py
index e3cccf0..cdeca1d 100644
--- a/beacon/handlers/email.py
+++ b/beacon/handlers/email.py
@@ -11,14 +11,15 @@ INTERNAL_DOMAIN = "@beacon.example"
 class EmailHandler(Handler):
     kind = "email"
 
-    def validate(self, record: Record) -> None:
+    def validate(self, record: Record) -> bool:
         super().validate(record)
         if not record.get("message_id"):
             raise RecordRejected("email event without a message_id")
         if record.get("status") not in STATUSES:
             raise RecordRejected(f"unknown email status {record.get('status')!r}")
         if record.get("to", "").endswith(INTERNAL_DOMAIN):
-            raise RecordRejected("internal test address")
+            return False  # internal test address
+        return True
 
     def transform(self, record: Record) -> Row:
         row = super().transform(record)
diff --git a/beacon/handlers/experiment.py b/beacon/handlers/experiment.py
index c3b2ff8..58a3eff 100644
--- a/beacon/handlers/experiment.py
+++ b/beacon/handlers/experiment.py
@@ -10,14 +10,15 @@ EXCLUDED = "__excluded__"  # the assignment service holds these users out
 class ExperimentHandler(Handler):
     kind = "experiment"
 
-    def validate(self, record: Record) -> None:
+    def validate(self, record: Record) -> bool:
         super().validate(record)
         if not record.get("experiment"):
             raise RecordRejected("exposure without an experiment")
         if not record.get("variant"):
             raise RecordRejected("exposure without a variant")
         if record["variant"] == EXCLUDED:
-            raise RecordRejected("held out of the experiment")
+            return False  # held out of the experiment
+        return True
 
     def transform(self, record: Record) -> Row:
         row = super().transform(record)
diff --git a/beacon/handlers/login.py b/beacon/handlers/login.py
index 9242c47..063f284 100644
--- a/beacon/handlers/login.py
+++ b/beacon/handlers/login.py
@@ -10,12 +10,13 @@ METHODS = ("password", "sso", "magic_link")
 class LoginHandler(Handler):
     kind = "login"
 
-    def validate(self, record: Record) -> None:
+    def validate(self, record: Record) -> bool:
         super().validate(record)
         if record.get("method") not in METHODS:
             raise RecordRejected(f"unknown login method {record.get('method')!r}")
         if record.get("user_agent", "").startswith("uptime-probe/"):
-            raise RecordRejected("uptime probe signing in")
+            return False  # uptime probe signing in
+        return True
 
     def transform(self, record: Record) -> Row:
         row = super().transform(record)
diff --git a/beacon/handlers/pageview.py b/beacon/handlers/pageview.py
index 45fd2ea..f446736 100644
--- a/beacon/handlers/pageview.py
+++ b/beacon/handlers/pageview.py
@@ -10,12 +10,13 @@ from beacon.useragent import browser_family, is_bot
 class PageviewHandler(Handler):
     kind = "pageview"
 
-    def validate(self, record: Record) -> None:
+    def validate(self, record: Record) -> bool:
         super().validate(record)
         if not record.get("url"):
             raise RecordRejected("pageview without a url")
         if is_bot(record.get("user_agent", "")):
-            raise RecordRejected("bot traffic")
+            return False  # bot traffic
+        return True
 
     def transform(self, record: Record) -> Row:
         row = super().transform(record)
diff --git a/beacon/report.py b/beacon/report.py
index 1970409..7e2399b 100644
--- a/beacon/report.py
+++ b/beacon/report.py
@@ -11,16 +11,21 @@ from beacon.handlers.base import HandlerResult
 @dataclass
 class IngestReport:
     kept: Counter[str] = field(default_factory=Counter)
+    skipped: Counter[str] = field(default_factory=Counter)
     rejected: list[tuple[str, str | None, str]] = field(default_factory=list)
     unknown: Counter[str] = field(default_factory=Counter)
 
     def add(self, result: HandlerResult) -> None:
         self.kept[result.kind] += len(result.rows)
+        self.skipped[result.kind] += result.skipped
         self.rejected.extend((result.kind, event_id, why) for event_id, why in result.rejected)
 
     def summary(self) -> str:
         kinds = ", ".join(f"{k} {n}" for k, n in sorted(self.kept.items()))
-        text = f"kept {self.kept.total()} ({kinds}), rejected {len(self.rejected)}"
+        text = (
+            f"kept {self.kept.total()} ({kinds}), skipped {self.skipped.total()}, "
+            f"rejected {len(self.rejected)}"
+        )
         if self.unknown:
             text += f", unknown types {dict(sorted(self.unknown.items()))}"
         return text
diff --git a/tests/test_handlers.py b/tests/test_handlers.py
index 12f19e7..6b91457 100644
--- a/tests/test_handlers.py
+++ b/tests/test_handlers.py
@@ -10,10 +10,11 @@ def run(kind: str, *records: dict) -> HandlerResult:
     return HANDLERS[kind]().run(records)
 
 
-def test_bot_pageviews_are_rejected():
+def test_bot_pageviews_are_skipped():
     result = run("pageview", {**PAGEVIEW, "user_agent": "Googlebot/2.1"})
     assert result.rows == []
-    assert result.rejected == [("e1", "bot traffic")]
+    assert result.skipped == 1
+    assert result.rejected == []
 
 
 def test_purchase_rows_keep_amounts_in_cents():

Leaderboard for this case

easy · 9 of 9 bots fully caught it (100% solve rate) · 100% avg coverage.

#BotResultCaughtCoverageFindingsFalse alarms
1 Claude Code (Opus 5.5) solved 1/1 100.0% 2 0
2 Cubic solved 1/1 100.0% 1 0
3 cursor solved 1/1 100.0% 1 0
4 Drape (Deep Review) solved 1/1 100.0% 1 0
5 Drape (Review) solved 1/1 100.0% 1 0
6 greptile solved 1/1 100.0% 2 0
7 Mjolnir (Opus 5.5) solved 1/1 100.0% 2 0
8 Mjolnir (Sonnet 5.5) solved 1/1 100.0% 2 0
9 qodo solved 1/1 100.0% 1 0

Bot results

Claude Code (Opus 5.5) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✓ exact
valid · matches d1 · impact high
valid · impact high
RefundHandler.validate calls super().validate(record) without returning anything, so it returns None. The new run() code reads that None as a skip (`if not keep`), so every valid refund is silently dropped. Purchase, signup and subscription happen to work only because they `return super().validate(record)`. This is an incomplete migration caused by this change. [grounded: tree beacon/handlers/refund.py] [consensus 3/3: valid=3]
raw comments (2)
beacon/handlers/refund.py:12
RefundHandler.validate wasn't updated for the new contract. It calls super().validate(record), throws away the result, and then falls off the end, so it returns None. Base.run now does `if not keep: skipped += 1; continue`, which means every valid refund is silently skipped and never written to the warehouse. Purchases still land, so revenue sums that net purchases against refunds will be overstated. No error or reject is raised; the drop only shows up as a 'skipped' count. Add `return True` (or `return super().validate(record)`).
beacon/handlers/base.py:79
Any record whose validate returns a falsy value is treated as skipped, and that includes an implicit None. Under the old contract the return value was ignored and returning None was correct, so any existing or future override without an explicit `return True` (refund today) now drops data silently. Consider checking `keep is False` explicitly, or raising/asserting when validate returns something that isn't a bool. Several overrides (purchase, signup, subscription, refund) also still declare `-> None`, which hides the mismatch from type checkers.
Cubic · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✓ exact
valid · matches d1 · impact high
raw comments (2)
beacon/handlers/base.py:78
<!-- cubic:v=121580c4-4840-481a-867b-932493675adf --> <!-- cubic:review-run=9920a703-04e3-5b23-b89f-31b30e4d915a --> <!-- metadata:{"confidence":10} --> P1: `RefundHandler.validate` returns `None` for valid refunds, so this truthiness check skips every one instead of transforming it; treat only `False` as a skip. <details> <summary>Prompt for AI agents</summary> ```text Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At beacon/handlers/base.py, line 78: <comment>`RefundHandler.validate` returns `None` for valid refunds, so this truthiness check skips every one instead of transforming it; treat only `False` as a skip.</comment> <file context> @@ -66,9 +71,12 @@ def run(self, records: Iterable[Record]) -> HandlerResult: except RecordRejected as exc: result.rejected.append((record.get("id"), str(exc))) continue + if not keep: + result.skipped += 1 + continue </file context> ``` </details> ```suggestion if keep is False: ```
PR-level
<!-- cubic:review-summary:start --> **1 issue found** across 8 files <!-- cubic:review-summary:end --> <details> <summary>Prompt for AI agents (unresolved issues)</summary> ```text Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately. <file name="beacon/handlers/base.py"> <violation number="1" location="beacon/handlers/base.py:78"> P1: `RefundHandler.validate` returns `None` for valid refunds, so this truthiness check skips every one instead of transforming it; treat only `False` as a skip.</violation> </file> ``` </details> <sub>Reply with feedback, questions, or to request a fix.<br /><br />[View guided diff](https://www.cubic.dev/pr/drape-io/crbench-cubic-2026-09/pull/110?utm_source=github&utm_medium=review_footer) | [Turn on auto-fix](https://www.cubic.dev/action/auto-fix/pr/drape-io/crbench-cubic-2026-09/110?returnTo=https%3A%2F%2Fgithub.com%2Fdrape-io%2Fcrbench-cubic-2026-09%2Fpull%2F110) | [Re-trigger cubic](https://www.cubic.dev/action/re-review/pr/drape-io/crbench-cubic-2026-09/110/ai_pr_review_github_delivery_da8304877ce42a397692a0cd6b7594a5a7c9e0a81259826d65758b8bed1f77fe?returnTo=https%3A%2F%2Fgithub.com%2Fdrape-io%2Fcrbench-cubic-2026-09%2Fpull%2F110)</sub> <!-- cubic:review-post:ai_pr_review_github_delivery_da8304877ce42a397692a0cd6b7594a5a7c9e0a81259826d65758b8bed1f77fe:d81d5aca001bcd6ab25518bb826819e4f9e1f3a2:9920a703-04e3-5b23-b89f-31b30e4d915a -->
cursor · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✓ exact
valid · matches d1 · impact high
raw comments (2)
beacon/handlers/base.py:80
### Valid refunds silently skipped **High Severity** <!-- DESCRIPTION START --> `run` now treats a falsy `validate` return as a skip, and `RefundHandler.validate` still returns `None`. Every valid refund is dropped from ingest and never reaches the warehouse or the rejects table. <!-- DESCRIPTION END --> <!-- BUGBOT_BUG_ID: a4d1c6d7-1cd2-465c-9e6e-552245865d6d --> <!-- LOCATIONS START beacon/handlers/base.py#L73-L80 LOCATIONS END --> <div><a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9DVVJTT1IiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90Ojg3YTEyZGRmLTU4OTItNGM0ZC1iMzNjLTRmNDc4YjNlYzA2NCIsImVuY3J5cHRpb25LZXkiOiJVbFM1OFNtdF9UZi1BeUxOSFoxcWEwQ2JjT2RvZFVVVlZtUFZxQTZ1MXVJIiwiYnJhbmNoIjoicHItMjJiYTkzZWYyYyIsInJlcG9Pd25lciI6ImRyYXBlLWlvIiwicmVwb05hbWUiOiJjcmJlbmNoLWN1cnNvci0yMDI2LTA5IiwicHJvdmlkZXIiOiJnaXRodWIifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/agents?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9XRUIiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90Ojg3YTEyZGRmLTU4OTItNGM0ZC1iMzNjLTRmNDc4YjNlYzA2NCIsImVuY3J5cHRpb25LZXkiOiJVbFM1OFNtdF9UZi1BeUxOSFoxcWEwQ2JjT2RvZFVVVlZtUFZxQTZ1MXVJIiwiYnJhbmNoIjoicHItMjJiYTkzZWYyYyIsInJlcG9Pd25lciI6ImRyYXBlLWlvIiwicmVwb05hbWUiOiJjcmJlbmNoLWN1cnNvci0yMDI2LTA5IiwicHJOdW1iZXIiOjExMCwiY29tbWl0U2hhIjoiM2M1NzMzYjQ2MzRhNWNiOGMzYmExODY1YzRlYzhkODQ4NjAyOGQ1OSIsInByb3ZpZGVyIjoiZ2l0aHViIn19" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-web-light.png"><img alt="Fix in Web" width="99" height="28" src="https://cursor.com/assets/images/fix-in-web-dark.png"></picture></a></div> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 3c5733b4634a5cb8c3ba1865c4ec8d8486028d59. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
PR-level
<!-- BUGBOT_REVIEW --> Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue. <!-- BUGBOT_FIX_ALL --> <a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9BTExfSU5fQ1VSU09SIiwiZGF0YSI6eyJyZWRpc0tleSI6ImJ1Z2JvdC1tdWx0aTozMjU0NTFjMi0wZjZiLTRjNDMtYmI0NS05MDgzNjI4NzgzYWIiLCJlbmNyeXB0aW9uS2V5IjoiNDJjbVowdTJRTkpuZ212bm93RVFFSUhxQ3J2QldMV2NUYVByR3J0b1F0NCIsImJyYW5jaCI6InByLTIyYmE5M2VmMmMiLCJyZXBvT3duZXIiOiJkcmFwZS1pbyIsInJlcG9OYW1lIjoiY3JiZW5jaC1jdXJzb3ItMjAyNi0wOSIsInByb3ZpZGVyIjoiZ2l0aHViIn19" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix All in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a> <!-- /BUGBOT_FIX_ALL --> <!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_BEGIN --> <sup>❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the [Cursor dashboard](https://www.cursor.com/dashboard/bugbot).</sup> <!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_END --> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 3c5733b4634a5cb8c3ba1865c4ec8d8486028d59. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
Drape (Deep Review) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✓ exact
valid · matches d1 · impact high
raw comments (1)
beacon/handlers/refund.py:12
**Every valid refund is now silently skipped** `RefundHandler.validate` was not updated: it calls `super().validate(record)` but returns nothing. `run` now treats that `None` as "skip", so every good refund is counted as skipped and never written. Revenue no longer nets out refunds. Return True at the end of `RefundHandler.validate`, or `return super().validate(record)`. ``` beacon/handlers/base.py:74 keep = self.validate(record) beacon/handlers/base.py:75 except RecordRejected as exc: beacon/handlers/base.py:76 result.rejected.append((record.get("id"), str(exc))) beacon/handlers/base.py:77 continue beacon/handlers/refund.py:12 def validate(self, record: Record) -> None: beacon/handlers/refund.py:13 super().validate(record) beacon/handlers/refund.py:14 if not record.get("order_id"): beacon/handlers/refund.py:15 raise RecordRejected("refund without an order_id") beacon/handlers/refund.py:16 if not isinstance(record.get("amount_cents"), int) or record["amount_cents"] <= 0: beacon/handlers/refund.py:17 raise RecordRejected( ```
Drape (Review) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✓ exact
valid · matches d1 · impact high
raw comments (1)
beacon/handlers/refund.py:12
**Every valid refund is silently skipped** `RefundHandler.validate` was not updated and still returns None. `run` now treats a falsy result as 'skip', so every valid refund is counted as skipped and never produces a row. Return True at the end of `RefundHandler.validate` and change its annotation to bool. ``` beacon/handlers/base.py:74 keep = self.validate(record) beacon/handlers/base.py:75 except RecordRejected as exc: beacon/handlers/base.py:76 result.rejected.append((record.get("id"), str(exc))) beacon/handlers/base.py:77 continue ```
greptile · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✓ exact
valid · matches d1 · impact high
minor · impact low
Accurate: the diff adds only the bot-pageview skip test, and nothing tests the other skip branches or IngestReport.skipped/summary. The finding asks for more tests but does not claim that any code behaves incorrectly, so it is a coverage suggestion rather than a defect. [grounded: diff] [consensus 5/5: minor=5]
raw comments (3)
beacon/handlers/base.py:80
<a href="#"><img alt="P1" src="https://greptile-static-assets.s3.amazonaws.com/badges/p1.svg?v=9" align="top"></a> **Valid refunds disappear** `if not keep` silently drops every valid refund. `RefundHandler.validate()` still returns `None` after its checks, so these records now count as skipped instead of reaching `RefundHandler.transform()`. That removes the negative amounts used to offset purchases and overstates revenue. Update `RefundHandler.validate()` to return `True` on success before enforcing this new contract, and add a refund regression test.
tests/test_handlers.py:17
<a href="#"><img alt="P2" src="https://greptile-static-assets.s3.amazonaws.com/badges/p2.svg?v=9" align="top"></a> **Most skip paths lack tests** The new test covers only one bot pageview. The four other skip branches and `IngestReport.skipped` have no tests. Add cases for synthetic clicks, internal email addresses, uptime probes, and held-out exposures, plus a mixed ingest batch that checks kept rows, rejected records, per-type skipped totals, and `summary()`. Otherwise a broken skip branch or missing report count can pass unnoticed. Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
PR-level
<!-- greptile_summary --> <h2><a href="https://app.greptile.com/api/retrigger?id=76825512"><picture><source media="(prefers-color-scheme: dark)" srcset="https://greptile-static-assets.s3.amazonaws.com/badges/RetriggerDark.svg?v=2"><source media="(prefers-color-scheme: light)" srcset="https://greptile-static-assets.s3.amazonaws.com/badges/Retrigger.svg?v=2"><img alt="Retrigger" src="https://greptile-static-assets.s3.amazonaws.com/badges/Retrigger.svg?v=2" align="right"></picture></a>Confidence Score: 4/5</h2> **[Medium risk]** Changes how traffic filtering counts in analytics reports\. Fix the silent loss of valid refunds before merging. <h2>Findings</h2> 1. <img alt="P1" src="https://greptile-static-assets.s3.amazonaws.com/badges/p1.svg?v=9" align="top">&nbsp;**Valid refunds disappear** <a href="https://github.com/drape-io/crbench-greptile-2026-09/pull/110#discussion_r4214137412">▶</a> 2. <img alt="P2" src="https://greptile-static-assets.s3.amazonaws.com/badges/p2.svg?v=9" align="top">&nbsp;**Most skip paths lack tests** <a href="https://github.com/drape-io/crbench-greptile-2026-09/pull/110#discussion_r4214137417">▶</a> <h3>Summary</h3> Adds a skip outcome for expected bot and test traffic, separate from malformed-record rejection. - Updates five handlers to return `False` for unwanted traffic. - Counts skipped records by type and adds their total to the report summary. - Valid refunds now disappear because their handler still returns `None`. - Tests cover only the bot skip path, not the other skip cases or report totals. <details open><summary>Diagram</summary> ```mermaid %%{init: {'theme': 'neutral'}}%% flowchart TD A[Record] --> B[validate] B -->|RecordRejected| C[Add to rejected] B -->|Falsey return| D[Count as skipped] B -->|True| E[Transform and write row] F[Valid refund returns None] --> D ``` </details> <!-- greptile_confidence_score:4 --> <sub>Reviews (1) · Last reviewed commit: ["Skip bot and test traffic without counti..."](https://github.com/drape-io/crbench-greptile-2026-09/commit/44698fecad6cdf7e9b1285695281edc2ddb7cfe4) · [Reviewed by Greptile](https://www.greptile.com/?utm_source=greptile_expert&utm_medium=github&utm_campaign=code_reviews&utm_content=drape-io/crbench-greptile-2026-09)</sub>
Mjolnir (Opus 5.5) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✓ exact
valid · matches d1 · impact high
valid · impact high
In beacon/handlers/refund.py, RefundHandler.validate still has the `-> None` signature and never returns a value, so it returns None for every valid refund. The changed run() treats any falsy result as a skip, so every refund is counted as skipped and never written. Purchase, signup and subscription are unaffected because they return super().validate(record), which is now True. [grounded: diff] [consensus 3/3: valid=3]
raw comments (2)
beacon/handlers/base.py:74
[P1] Refund handler wasn't updated, so every valid refund is now silently skipped. `run` now treats any falsy return from `validate` as "skip", so a `validate` that returns `None` drops the record without an error. `RefundHandler.validate` in `beacon/handlers/refund.py` wasn't changed in this diff. It still calls `super().validate(record)` without returning the result and then falls off the end, which returns `None`. As a result, every well-formed refund is counted under `skipped` and never written to the warehouse, and no reject is recorded. Refund rows carry negative amounts so that revenue over purchases and refunds nets out, so this would overstate revenue and nothing would show it apart from a higher skip count. Purchase, signup and subscription happen to work because they `return super().validate(record)`. The fix is to make `RefundHandler.validate` return a bool. Treating only an explicit `False` as a skip (`if keep is False`) would also stop un-updated or third-party handlers from silently dropping data.
PR-level
[P1] RefundHandler still returns None from validate, so every refund is now skipped beacon/handlers/base.py:74-79 `run` now treats any falsy return from `validate` as "skip". `RefundHandler.validate` (`beacon/handlers/refund.py`) was not updated: it calls `super().validate(record)` and ignores the result, then falls off the end and returns `None`. Every valid refund is now counted in `skipped` and never reaches `transform` or the sink. Purchases still land, so revenue sums that rely on refunds netting out (the refund module docstring says this is their purpose) will be overstated. Nothing gets rejected, so the only trace in the nightly report is a higher skipped count. Any third-party or future subclass that keeps the old `-› None` shape breaks the same silent way. Either fix refund.py, or have `run` only skip on an explicit `False` (`if keep is False`) so a forgotten return keeps the record instead of dropping it.
Mjolnir (Sonnet 5.5) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✓ exact
valid · impact high
RefundHandler.validate (unchanged) calls super().validate(record) without returning it, so it returns None. Under the new run() logic, None is falsy, so every valid refund is counted as skipped and never transformed or written. The change made this happen and did not update this sibling handler. [grounded: diff] [consensus 3/3: valid=3]
valid · matches d1 · impact high
raw comments (2)
beacon/handlers/refund.py:12
[P1] Refund handler silently skips every valid refund `RefundHandler.validate` was not updated for the new contract. It still returns `None` after its checks, and `Handler.run` now treats a falsy return from `validate` as "skip this record". Every refund that passes validation is therefore counted in `skipped` and never reaches `transform` or the sink. Refund rows are meant to net against purchase revenue, so this silently inflates revenue figures. The ingest summary would show only "skipped N" and no error. The fix is to return `True` at the end of `RefundHandler.validate`, with the annotation changed to `bool`. Purchase, signup and subscription return `super().validate(record)` and so work by accident. Their `-› None` annotations are also stale.
beacon/handlers/refund.py:12
[P1] RefundHandler.validate still returns None, so every refund is silently dropped The base contract changed so that `validate` returns True to keep a record and False to skip it. `run` now does `keep = self.validate(record)` and `if not keep: result.skipped += 1`. `RefundHandler.validate` was not updated. It still has a `-› None` signature and calls `super().validate(record)` without returning. It falls off the end and returns None, which is falsy. Every valid refund event is therefore counted as skipped and never reaches `transform`. Refunds are never written, and the report shows them as skipped rather than rejected, so nothing flags the loss. Revenue stops netting out refunds. `Click`, `Email`, `Experiment`, `Login` and `Pageview` were converted. `Purchase`, `Signup` and `Subscription` end with `return super().validate(record)`, so they still work. The fix is to end `RefundHandler.validate` with `return True`, or `return super().validate(record)`, and to change its annotation to `-› bool`. The same applies to any out-of-tree subclass that follows the old "return value is ignored" contract. It would be safer for `run` to check `keep is False`, or to require an explicit bool.
qodo · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✓ exact
valid · matches d1 · impact high
raw comments (3)
beacon/handlers/base.py:80
<img src="https://img.shields.io/badge/High-634FD1?style=flat-square" height="20px" alt="Action required"> 1\. Valid refunds disappear from the warehouse <code>🐞 Bug</code> <code>≡ Correctness</code> <pre> <b><i>RefundHandler.validate()</i></b> returns <b><i>None</i></b> after successful validation, and <b><i>Handler.run()</i></b> now treats that falsy value as a skip. Every valid refund is therefore counted as skipped rather than transformed into a negative-amount row or written to the warehouse. </pre> <details> <summary><strong>Agent Prompt</strong></summary> ``` ## Issue description Valid refunds are silently skipped because `RefundHandler.validate()` has no success return value and `Handler.run()` now skips falsy results. ## Fix Focus Areas - beacon/handlers/refund.py[11-20] - beacon/handlers/base.py[74-81] - tests/test_handlers.py[13-19] ## Recommended Fix Change `RefundHandler.validate()` to return `bool` and explicitly return `True` after its checks. Add a test asserting that a valid refund produces a negative-amount row and is not counted as skipped. ``` <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> </details> [Dismiss ↗](https://app.qodo.ai/findings/848957f7-72e6-4021-ac6d-1b231cc53e2d?action=dismiss&origin=git) | [View ↗](https://app.qodo.ai/findings/848957f7-72e6-4021-ac6d-1b231cc53e2d)
PR-level
<h3>PR Summary by Qodo</h3> Skip expected bot and test traffic without recording rejects <code>🐞 Bug fix</code> <code>🕐 10-20 Minutes</code> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <details> <summary>AI Description</summary> <dl> <dd> <br/> ><pre> >• Skip expected bot, synthetic, probe, internal-email, and held-out traffic instead of rejecting it. >• Count skipped events by type in ingest reports without adding them to reject totals. >• Keep malformed records on the rejection path. ></pre> </dd> </dl> </details> <details> <summary>Diagram</summary> <dl> <dd> <br/> ```mermaid graph TD Input["NDJSON events"] --> Handler["Event handler"] --> Validate["Validate record"] --> Decision{"Validation result"} Decision -->|keep| Warehouse["Warehouse rows"] --> Report["Ingest report"] Decision -->|skip| Skipped["Skipped count"] --> Report Decision -->|reject| Rejected["Reject details"] --> Report ``` </dd> </dl> </details> <details> <summary>High-Level Assessment</summary> <dl> <dd> <br/> >The following are alternative approaches to this PR: <details> <summary><b>1. Separate eligibility filtering from validation</b></summary> <dl> <dd> - ➕ Keeps malformed-record checks distinct from traffic-exclusion rules. - ➖ Adds another handler hook and processing step for a small set of exclusions. </dd> </dl> </details> >**Recommendation:** Use the PR&#x27;s boolean validation contract for this limited set of exclusions. It preserves validation before skipping, so malformed records still reach the rejection path, without adding a separate filtering stage. </dd> </dl> </details> <details> <summary> Files changed (8) <code> +35 / -16 </code> </summary> <dl> <dd> <br/> <details> <summary>Bug fix (7) <code> +32 / -14 </code></summary> <dl> <dd> <details> <summary>base.py<code>Add a skip outcome to the shared handler contract</code> <code>+11/-3</code></summary> <br/> >Add a skip outcome to the shared handler contract > ><pre> >• Validation now returns a boolean: false counts a record as skipped, while RecordRejected still records a rejection. HandlerResult gains a skipped count and the hook documentation explains the distinction. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/110/files#diff-618d4f2243db6a325870ec78964be0ac4d3b9c8a44b8f5a57aa3c861778c740a'>beacon/handlers/base.py</a> </details> <details> <summary>click.py<code>Skip synthetic UI-test clicks</code> <code>+3/-2</code></summary> <br/> >Skip synthetic UI-test clicks > ><pre> >• Synthetic clicks return false after required click fields are checked. Clicks without a target remain rejected. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/110/files#diff-5f9b6bc08080c0dab3d4532ea96f4007c76cf2eb2d58d0b0c49e3fc872a89071'>beacon/handlers/click.py</a> </details> <details> <summary>email.py<code>Skip email events for internal test addresses</code> <code>+3/-2</code></summary> <br/> >Skip email events for internal test addresses > ><pre> >• Events addressed to the internal domain are skipped after message ID and status validation. Invalid email events remain rejected. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/110/files#diff-2b67df54e5faf4536ba50e190025739d9cf6f94802a06602d8fdf32541392fec'>beacon/handlers/email.py</a> </details> <details> <summary>experiment.py<code>Skip held-out experiment exposures</code> <code>+3/-2</code></summary> <br/> >Skip held-out experiment exposures > ><pre> >• Exposures assigned the excluded variant are skipped. Missing experiment or variant fields remain rejection reasons. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/110/files#diff-441ace3389b0b5bb2a41e97fd17ed762125f08a066b9d9a58f2ea61c79fccaa7'>beacon/handlers/experiment.py</a> </details> <details> <summary>login.py<code>Skip uptime-probe sign-ins</code> <code>+3/-2</code></summary> <br/> >Skip uptime-probe sign-ins > ><pre> >• Sign-ins identified by the uptime-probe user agent are skipped after login-method validation. Unknown methods remain rejected. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/110/files#diff-0532e36c2b73d170c55c5c3c98a540501ac9c4311f0da739c36ba3f5f194fa3c'>beacon/handlers/login.py</a> </details> <details> <summary>pageview.py<code>Skip bot page views</code> <code>+3/-2</code></summary> <br/> >Skip bot page views > ><pre> >• Bot page views are skipped rather than rejected after URL validation. Page views without a URL remain rejected. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/110/files#diff-4c9bb8c1988e28173aefc6453f24495e1ad576db7acac8e11f03904c4cf5efe7'>beacon/handlers/pageview.py</a> </details> <details> <summary>report.py<code>Report skipped events separately from rejects</code> <code>+6/-1</code></summary> <br/> >Report skipped events separately from rejects > ><pre> >• IngestReport accumulates skipped counts by event type and includes their total in its summary. Skipped events do not enter the rejected list. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/110/files#diff-be5c37c8b956a39e966f66264d74a9c0d65c0596e89501f00bfba67349be9e9f'>beacon/report.py</a> </details> </dd> </dl> </details> <details> <summary>Tests (1) <code> +3 / -2 </code></summary> <dl> <dd> <details> <summary>test_handlers.py<code>Assert bot page views are skipped</code> <code>+3/-2</code></summary> <br/> >Assert bot page views are skipped > ><pre> >• The page-view test now checks that a bot event produces no row or rejection and increments the skipped count. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/110/files#diff-a80edd2ae6b30f0f5a2a8f7bdd4089a43fc5dca4d21588fa4879392950b609f1'>tests/test_handlers.py</a> </details> </dd> </dl> </details> </dd> </dl> </details>
PR-level
<h3>Code Review by Qodo</h3> <code>🐞 Bugs (1)</code> <code>📘 Rule violations (0)</code> <code>📜 Skill insights (0)</code> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <br/> <img src="https://img.shields.io/badge/High-634FD1?style=flat-square" height="20px" alt="Action required"> <details> <summary> 1. Valid refunds disappear from the warehouse <code>🐞 Bug</code> <code>≡ Correctness</code></summary> <br/> > <details open> ><summary>Description</summary> ><br/> > ><pre> ><b><i>RefundHandler.validate()</i></b> returns <b><i>None</i></b> after successful validation, and <b><i>Handler.run()</i></b> now >treats that falsy value as a skip. Every valid refund is therefore counted as skipped rather than >transformed into a negative-amount row or written to the warehouse. ></pre> ></details> > <details> ><summary>Code</summary> ><br/> > ><code>[beacon/handlers/base.py[R78-80]](https://github.com/drape-io/crbench-qodo-2026-09/pull/110/files#diff-618d4f2243db6a325870ec78964be0ac4d3b9c8a44b8f5a57aa3c861778c740aR78-R80)</code> > >```diff >+ if not keep: >+ result.skipped += 1 >+ continue >``` ></details> > <details> ><summary>Evidence</summary> ><br/> > ><pre> >The refund validator has no return statement on its successful path, so Python returns <b><i>None</i></b>. The >changed run loop skips that result before calling <b><i>transform()</i></b>, while ingestion writes only >transformed rows. ></pre> > > <code>[beacon/handlers/refund.py\[11-20\]](https://github.com/drape-io/crbench-qodo-2026-09/blob/c086aac45f20cf89dcc3b1f9c89e9b35e7c9d442/beacon/handlers/refund.py/#L11-L20)</code> > <code>[beacon/handlers/base.py\[74-81\]](https://github.com/drape-io/crbench-qodo-2026-09/blob/c086aac45f20cf89dcc3b1f9c89e9b35e7c9d442/beacon/handlers/base.py/#L74-L81)</code> > <code>[beacon/ingest.py\[24-32\]](https://github.com/drape-io/crbench-qodo-2026-09/blob/c086aac45f20cf89dcc3b1f9c89e9b35e7c9d442/beacon/ingest.py/#L24-L32)</code> ></details> > <details> ><summary>Agent prompt</summary> ><br/> > >``` >The issue below was found during a code review. Follow the provided context and guidance below and implement a solution > >## Issue description >Valid refunds are silently skipped because `RefundHandler.validate()` has no success return value and `Handler.run()` now skips falsy results. >## Fix Focus Areas >- beacon/handlers/refund.py[11-20] >- beacon/handlers/base.py[74-81] >- tests/test_handlers.py[13-19] >## Recommended Fix >Change `RefundHandler.validate()` to return `bool` and explicitly return `True` after its checks. Add a test asserting that a valid refund produces a negative-amount row and is not counted as skipped. >``` > <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> ></details> [Dismiss ↗](https://app.qodo.ai/findings/848957f7-72e6-4021-ac6d-1b231cc53e2d?action=dismiss&origin=git) | [View ↗](https://app.qodo.ai/findings/848957f7-72e6-4021-ac6d-1b231cc53e2d) <hr/> </details> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <!-- qodo-context:start --> <details><summary><strong>Context sources</strong></summary> <div>Review mode: <code>Auto: ⚖️ Balanced</code>: Behavioral changes span shared ingestion logic, five handlers, and reporting.</div> <!-- qodo-context:end --> </details> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <!-- qodo-daily-tip:start --> <details> <summary><strong>Tip of the day</strong></summary> <br/> <pre>💡 Did you know, you can keep summaries lean with Findings visible per group, which tucks the rest behind a View link</pre> <a href="https://docs.qodo.ai/tips-and-tricks">More tips ↗</a> | <a href="https://app.qodo.ai/configurations?tab=display-preferences">Customize Qodo ↗</a> | <a href="https://docs.qodo.ai">Qodo docs ↗</a> </details> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <!-- qodo-daily-tip:end --> <!-- https://github.com/drape-io/crbench-qodo-2026-09/commit/c086aac45f20cf89dcc3b1f9c89e9b35e7c9d442 --> <a href="https://www.qodo.ai"><img src="https://www.qodo.ai/wp-content/uploads/2025/03/qodo-logo.svg" width="80" alt="Qodo Logo"></a>