crbench//

← all cases

load_flags() returns None for empty config, breaking the membership check in gate.py

Tier 1 easy python · cross-file · authored, execution-grounded case

Ground-truth defects

d1 · correctness · high · config_loader.py:4
load_flags() returns None when the config string is empty instead of an empty set. The unchanged consumer gate.is_enabled() does `flag in load_flags(raw)`, so an empty config raises `TypeError: argument of type 'NoneType' is not iterable`. The diff looks fine in isolation — the break is only visible through the unchanged consumer gate.py.

The change under review

diff
diff --git a/config_loader.py b/config_loader.py
--- a/config_loader.py
+++ b/config_loader.py
@@ -1,4 +1,6 @@
 def load_flags(raw):
     """Return the set of enabled feature flags from a raw config string."""
+    if not raw.strip():
+        return None  # nothing configured
     flags = set()
     for line in raw.splitlines():

Leaderboard for this case

easy · 9 of 9 bots fully caught it (100% solve rate) · 100% avg coverage.

#BotResultCaughtCoverageFindingsFalse alarms
1 Claude Code (Opus 5.5) solved 1/1 100.0% 1 0
2 Cubic solved 1/1 100.0% 1 0
3 cursor solved 1/1 100.0% 1 0
4 Drape (Deep Review) solved 1/1 100.0% 1 0
5 Drape (Review) solved 1/1 100.0% 1 0
6 greptile solved 1/1 100.0% 1 0
7 Mjolnir (Opus 5.5) solved 1/1 100.0% 2 0
8 Mjolnir (Sonnet 5.5) solved 1/1 100.0% 2 0
9 qodo solved 1/1 100.0% 1 0

Bot results

Claude Code (Opus 5.5) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✓ exact
valid · matches d1 · impact high
raw comments (1)
config_loader.py:4
When the config string is empty or only whitespace, load_flags now returns None. The docstring says it returns a set, and before this change it returned an empty set. Callers that do `'x' in load_flags(raw)`, iterate over the result, or call set methods on it will now raise TypeError for an empty config, which is a normal case. Return `set()` instead, so the type stays the same.
Cubic · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✓ exact
valid · matches d1 · impact high
raw comments (2)
config_loader.py:4
<!-- cubic:v=c4053e8d-7e97-47b9-ac49-6c216d0e855b --> <!-- cubic:review-run=2e2249b6-12e6-5ca1-88e8-061842e6089c --> <!-- metadata:{"confidence":10} --> P1: When the raw configuration is empty or whitespace-only, `is_enabled` now raises `TypeError` because it performs membership testing on `None`. Return an empty set here to preserve the documented return type and make disabled flags evaluate to `False`. <details> <summary>Prompt for AI agents</summary> ```text Check if this issue is valid — if so, understand the root cause and fix it. At config_loader.py, line 4: <comment>When the raw configuration is empty or whitespace-only, `is_enabled` now raises `TypeError` because it performs membership testing on `None`. Return an empty set here to preserve the documented return type and make disabled flags evaluate to `False`.</comment> <file context> @@ -1,5 +1,7 @@ def load_flags(raw): """Return the set of enabled feature flags from a raw config string.""" + if not raw.strip(): + return None # nothing configured flags = set() for line in raw.splitlines(): </file context> ``` </details> ```suggestion return set() # nothing configured ```
PR-level
<!-- cubic:review-summary:start --> **1 issue found** across 1 file <!-- cubic:review-summary:end --> <details> <summary>Prompt for AI agents (unresolved issues)</summary> ```text Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately. <file name="config_loader.py"> <violation number="1" location="config_loader.py:4"> P1: When the raw configuration is empty or whitespace-only, `is_enabled` now raises `TypeError` because it performs membership testing on `None`. Return an empty set here to preserve the documented return type and make disabled flags evaluate to `False`.</violation> </file> ``` </details> <sub>Reply with feedback, questions, or to request a fix.<br /><br />[Re-trigger cubic](https://www.cubic.dev/action/re-review/pr/drape-io/crbench-cubic-2026-09/4/ai_pr_review_github_delivery_011e97bb3c4e54a05ffd02c9a9c2cc19f6bb47d53e124d24b88a385e9c860c28?returnTo=https%3A%2F%2Fgithub.com%2Fdrape-io%2Fcrbench-cubic-2026-09%2Fpull%2F4)</sub> <!-- cubic:review-post:ai_pr_review_github_delivery_011e97bb3c4e54a05ffd02c9a9c2cc19f6bb47d53e124d24b88a385e9c860c28:4fed6bd458a732d7431fa989de4af0620600c5e3:2e2249b6-12e6-5ca1-88e8-061842e6089c -->
cursor · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✓ exact
valid · matches d1 · impact high
raw comments (2)
config_loader.py:4
### Empty config returns None instead of set **High Severity** <!-- DESCRIPTION START --> `load_flags` now returns `None` for empty or whitespace-only input instead of an empty set. `is_enabled` still does `flag in load_flags(raw)`, which raises `TypeError` when nothing is configured. The docstring still promises a set, so empty config is treated as a crash rather than “no flags enabled.” <!-- DESCRIPTION END --> <!-- BUGBOT_BUG_ID: 564fd6c0-e0b4-48f4-a3c3-6b730b2fea4d --> <!-- LOCATIONS START config_loader.py#L2-L4 LOCATIONS END --> <div><a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9DVVJTT1IiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OmU2NmNiYmQyLTNhYzEtNDViYS04YzBiLTZkNWIwMzQ1NzhkNCIsImVuY3J5cHRpb25LZXkiOiJTdEUzc08wdWEyNWlPNlBIMXk5ZHVwUC1jMUxmNkplVmNhWUxoRk9qdmQwIiwiYnJhbmNoIjoidDEtcHktZW1wdHktY29uZmlnLW5vbmUiLCJyZXBvT3duZXIiOiJkcmFwZS1pbyIsInJlcG9OYW1lIjoiY3JiZW5jaC1jdXJzb3ItMjAyNi0wOSIsInByb3ZpZGVyIjoiZ2l0aHViIn19" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/agents?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9XRUIiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OmU2NmNiYmQyLTNhYzEtNDViYS04YzBiLTZkNWIwMzQ1NzhkNCIsImVuY3J5cHRpb25LZXkiOiJTdEUzc08wdWEyNWlPNlBIMXk5ZHVwUC1jMUxmNkplVmNhWUxoRk9qdmQwIiwiYnJhbmNoIjoidDEtcHktZW1wdHktY29uZmlnLW5vbmUiLCJyZXBvT3duZXIiOiJkcmFwZS1pbyIsInJlcG9OYW1lIjoiY3JiZW5jaC1jdXJzb3ItMjAyNi0wOSIsInByTnVtYmVyIjo0LCJjb21taXRTaGEiOiJlNTI0ZGFjOTJlMGRkODI3NmIwMWYzMzU1MGU4ZjBlNDI5MDc3NWVjIiwicHJvdmlkZXIiOiJnaXRodWIifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-web-light.png"><img alt="Fix in Web" width="99" height="28" src="https://cursor.com/assets/images/fix-in-web-dark.png"></picture></a></div> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit e524dac92e0dd8276b01f33550e8f0e4290775ec. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
PR-level
<!-- BUGBOT_REVIEW --> Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue. <!-- BUGBOT_FIX_ALL --> <a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9BTExfSU5fQ1VSU09SIiwiZGF0YSI6eyJyZWRpc0tleSI6ImJ1Z2JvdC1tdWx0aTo5YjcyYWM5Yi1jNGM3LTQ1Y2ItODVlZi1mNDdlNjgzMTg2NTQiLCJlbmNyeXB0aW9uS2V5IjoiX2oxZS0zamZBN2ItMGN6VWltLTBGc2NPdDVZZkhUUVE4dW0yb1hzaXdQTSIsImJyYW5jaCI6InQxLXB5LWVtcHR5LWNvbmZpZy1ub25lIiwicmVwb093bmVyIjoiZHJhcGUtaW8iLCJyZXBvTmFtZSI6ImNyYmVuY2gtY3Vyc29yLTIwMjYtMDkiLCJwcm92aWRlciI6ImdpdGh1YiJ9fQ" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix All in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a> <!-- /BUGBOT_FIX_ALL --> <!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_BEGIN --> <sup>❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the [Cursor dashboard](https://www.cursor.com/dashboard/bugbot).</sup> <!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_END --> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit e524dac92e0dd8276b01f33550e8f0e4290775ec. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
Drape (Deep Review) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✓ exact
valid · matches d1 · impact high
raw comments (1)
config_loader.py:4
**Empty config crashes `is_enabled` with TypeError** When the config string is empty or only whitespace, `load_flags` now returns None instead of an empty set. `is_enabled` then runs `flag in None`, which raises TypeError instead of returning False. Return `set()` for an empty config, not None. ``` config_loader.py:3 if not raw.strip(): config_loader.py:4 return None # nothing configured gate.py:6 return flag in load_flags(raw) config_loader.py:1 def load_flags(raw): config_loader.py:2 """Return the set of enabled feature flags from a raw config string.""" ```
Drape (Review) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✓ exact
valid · matches d1 · impact high
raw comments (1)
config_loader.py:4
**Empty config makes is_enabled crash** load_flags now returns None for an empty or whitespace-only config, but its docstring promises a set and the caller does `flag in load_flags(raw)`. That raises TypeError. Return an empty set() instead of None. ``` config_loader.py:3 if not raw.strip(): config_loader.py:4 return None # nothing configured gate.py:6 return flag in load_flags(raw) ```
greptile · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✓ exact
valid · matches d1 · impact high
raw comments (2)
config_loader.py:4
<a href="#"><img alt="P1" src="https://greptile-static-assets.s3.amazonaws.com/badges/p1.svg?v=9" align="top"></a> **Blank Config Breaks Caller** For blank or whitespace-only configuration, this branch returns `None`. The existing `gate.is_enabled()` caller then evaluates `flag in None`, which raises `TypeError` instead of returning `False` as it did when `load_flags()` returned an empty set. ```suggestion if not raw.strip(): return set() # nothing configured ```
PR-level
<h3>Greptile Summary</h3> The PR adds special handling for blank feature-flag configuration strings. - Blank input now returns `None` rather than the function’s documented set type. - This breaks membership testing in the existing `is_enabled()` caller. <h3>Confidence Score: 4/5</h3> This PR is not safe to merge until blank configuration continues to produce a value compatible with `is_enabled()` membership testing. The changed blank-input return path causes the repository’s only caller to raise `TypeError` for empty or whitespace-only configurations instead of returning `False`. **Files Needing Attention:** config_loader.py <details><summary><h3>Important Files Changed</h3></summary> | Filename | Overview | |----------|----------| | config_loader.py | Adds a blank-input branch whose `None` return breaks the existing caller’s set-membership operation. | </details> <!-- greptile_other_comments_section --> <sub>Reviews (1): Last reviewed commit: ["Update config\_loader.py"](https://github.com/drape-io/crbench-greptile-2026-09/commit/9fb1f921a4eaea1f5d83f5398e51e51d426e9024) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=64284093)</sub>
Mjolnir (Opus 5.5) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✓ exact
valid · matches d1 · impact high
valid · impact high
Before this change, an empty config made load_flags return an empty set. Now it returns None, so `flag in load_flags(raw)` in gate.py:6 raises TypeError instead of returning False. [grounded: diff] [consensus 3/3: valid=3]
raw comments (2)
config_loader.py:3
[P1] Empty config now returns None, crashing `is_enabled` When the raw config is empty or contains only whitespace, `load_flags` now returns `None` where it used to return an empty `set()`. `gate.py:6` does `flag in load_flags(raw)`, so any empty or blank config now raises `TypeError: argument of type 'NoneType' is not iterable` instead of returning `False`. This also contradicts the docstring, which promises a set of enabled flags. Returning an empty set (or dropping the early return, since the loop already produces an empty set) keeps callers working.
config_loader.py:3
[P1] Returning None for empty config breaks `in` checks in callers When the raw config is empty or only whitespace, `load_flags` now returns `None` where it used to return an empty set. That breaks its existing caller: `gate.is_enabled` runs `flag in load_flags(raw)` (gate.py:6), so with an empty config it raises `TypeError: argument of type 'NoneType' is not iterable` instead of returning False. The return type also no longer matches the docstring, which promises a set of enabled flags. Returning `set()` here (or leaving out the early return) keeps "nothing configured" meaning "no flags enabled".
Mjolnir (Sonnet 5.5) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✓ exact
valid · matches d1 · impact high
valid · impact high
The new early return gives None for empty or whitespace-only input. gate.is_enabled then runs `flag in load_flags(raw)`, which raises TypeError. Before this change, an empty config returned an empty set and is_enabled returned False. [grounded: diff] [consensus 3/3: valid=3]
raw comments (2)
config_loader.py:3
[P1] load_flags returns None for empty config, breaking is_enabled The new early return gives `None` instead of a set when `raw` is empty or whitespace-only. This contradicts the docstring ("Return the set of enabled feature flags"). `gate.is_enabled` does `flag in load_flags(raw)` with no None check, so an empty or blank config now raises `TypeError: argument of type 'NoneType' is not iterable`. Before this change it returned False. Any caller that passes an empty config string will crash. Returning `set()` for the empty case, or removing the early return, would keep the contract.
config_loader.py:3
[P1] Empty config now returns None, crashing `is_enabled` load_flags` now returns `None` for an empty or whitespace-only config, but the docstring promises a set. The only caller, `gate.is_enabled` (gate.py:6), does `flag in load_flags(raw)`, which raises `TypeError: argument of type 'NoneType' is not iterable` whenever the raw config is empty. Before this change an empty config returned an empty set and `is_enabled` returned False. Return `set()` here, or drop the early return, so the type stays consistent
qodo · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✓ exact
valid · matches d1 · impact high
raw comments (3)
config_loader.py:4
<img src="https://img.shields.io/badge/High-634FD1?style=flat-square" height="20px" alt="Action required"> 1\. Empty configurations crash flag checks <code>🐞 Bug</code> <code>≡ Correctness</code> <pre> <b><i>load_flags</i></b> now returns <b><i>None</i></b> for blank strings instead of the set promised by its docstring and produced by every other successful path. When <b><i>is_enabled</i></b> checks membership in that result, empty or whitespace-only configuration input raises <b><i>TypeError</i></b> rather than reporting the flag as disabled. </pre> <details> <summary><strong>Agent Prompt</strong></summary> ``` ## Issue description `load_flags` returns `None` for blank input, but callers perform set membership on its return value, causing a `TypeError`. ## Fix Focus Areas - config_loader.py[3-4] ## Recommended Fix Return `set()` instead of `None` for empty or whitespace-only input so the function preserves its documented return type and callers treat all flags as disabled. ``` <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> </details>
PR-level
<h3>PR Summary by Qodo</h3> Return None for blank feature flag configuration <code>🐞 Bug fix</code> <code>🕐 Less than 5 minutes</code> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <details> <summary>AI Description</summary> <dl> <dd> <br/> ><pre> >• Treats empty and whitespace-only configuration as unconfigured. >• Distinguishes missing configuration from a parsed configuration with no enabled flags. ></pre> </dd> </dl> </details> <details> <summary>High-Level Assessment</summary> <dl> <dd> <br/> >The following are alternative approaches to this PR: <details> <summary><b>1. Continue returning an empty set</b></summary> <dl> <dd> - ➕ Preserves a consistent set return type - ➕ Avoids requiring callers to handle None - ➖ Cannot distinguish blank configuration from nonblank configuration with no enabled flags </dd> </dl> </details> >**Recommendation:** Returning None is appropriate if callers need to distinguish absent configuration from a valid configuration containing no enabled flags. The optional return contract should be documented and covered by tests because it changes the function&#x27;s previously consistent set return type. </dd> </dl> </details> <details> <summary> Files changed (1) <code> +2 / -0 </code> </summary> <dl> <dd> <br/> <details> <summary>Bug fix (1) <code> +2 / -0 </code></summary> <dl> <dd> <details> <summary>config_loader.py<code>Handle blank feature flag configuration explicitly</code> <code>+2/-0</code></summary> <br/> >Handle blank feature flag configuration explicitly > ><pre> >• Adds an early return of None when the raw configuration is empty or contains only whitespace. Nonblank configurations continue through the existing feature flag parser. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/4/files#diff-45dd6c952f70a0bf5850a91bd436d25c902661bdeb6e3378ad950c369dc8a913'>config_loader.py</a> </details> </dd> </dl> </details> </dd> </dl> </details>
PR-level
<h3>Code Review by Qodo</h3> <code>🐞 Bugs (1)</code> <code>📘 Rule violations (0)</code> <code>📜 Skill insights (0)</code> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <br/> <img src="https://img.shields.io/badge/High-634FD1?style=flat-square" height="20px" alt="Action required"> <details> <summary> 1. Empty configurations crash flag checks <code>🐞 Bug</code> <code>≡ Correctness</code></summary> <br/> > <details open> ><summary>Description</summary> ><br/> > ><pre> ><b><i>load_flags</i></b> now returns <b><i>None</i></b> for blank strings instead of the set promised by its docstring and >produced by every other successful path. When <b><i>is_enabled</i></b> checks membership in that result, empty >or whitespace-only configuration input raises <b><i>TypeError</i></b> rather than reporting the flag as >disabled. ></pre> ></details> > <details> ><summary>Code</summary> ><br/> > ><code>[config_loader.py[R3-4]](https://github.com/drape-io/crbench-qodo-2026-09/pull/4/files#diff-45dd6c952f70a0bf5850a91bd436d25c902661bdeb6e3378ad950c369dc8a913R3-R4)</code> > >```diff >+ if not raw.strip(): >+ return None # nothing configured >``` ></details> > <details> ><summary>Evidence</summary> ><br/> > ><pre> >The loader documents and normally returns a set, while the sole caller directly uses its result as >the right-hand operand of <b><i>in</i></b>; <b><i>None</i></b> does not support membership checks. ></pre> > > <code>[config_loader.py[1-10]](https://github.com/drape-io/crbench-qodo-2026-09/blob/6f502f761ab589e116aa0f3e85c3298637ee42df/config_loader.py/#L1-L10)</code> > <code>[gate.py[4-6]](https://github.com/drape-io/crbench-qodo-2026-09/blob/6f502f761ab589e116aa0f3e85c3298637ee42df/gate.py/#L4-L6)</code> ></details> > <details> ><summary>Agent prompt</summary> ><br/> > >``` >The issue below was found during a code review. Follow the provided context and guidance below and implement a solution > >## Issue description >`load_flags` returns `None` for blank input, but callers perform set membership on its return value, causing a `TypeError`. > >## Fix Focus Areas >- config_loader.py[3-4] > >## Recommended Fix >Return `set()` instead of `None` for empty or whitespace-only input so the function preserves its documented return type and callers treat all flags as disabled. >``` > <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> ></details> <hr/> </details> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <!-- qodo-context:start --> <details><summary><strong>Context sources</strong></summary> <div>Review mode: <code>⚖️ Balanced</code>: This is a behavioral change in configuration parsing, and returning None for empty input may affect callers or violate the function’s expected return contract.</div> <!-- qodo-context:end --> </details> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <!-- qodo-daily-tip:start --> <details> <summary><strong>Tip of the day</strong></summary> <br/> <pre>💡 Did you know, you can reply &#x27;qodo&#x27; on any finding to push back, ask questions, or dig deeper</pre> <a href="https://docs.qodo.ai/tips-and-tricks">More tips ↗</a> | <a href="https://app.qodo.ai/configurations?tab=display-preferences">Customize Qodo ↗</a> | <a href="https://docs.qodo.ai">Qodo docs ↗</a> </details> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <!-- qodo-daily-tip:end --> <!-- https://github.com/drape-io/crbench-qodo-2026-09/commit/6f502f761ab589e116aa0f3e85c3298637ee42df --> <a href="https://www.qodo.ai"><img src="https://www.qodo.ai/wp-content/uploads/2025/03/qodo-logo.svg" width="80" alt="Qodo Logo"></a>