crbench//

← all cases

feat(uptime): Add ability to use queues to manage parallelism

Tier 2 easy python · local · real merged PR from getsentry/sentry #95633

Ground-truth defects

d4 · correctness · high
Per-group queues are created as unbounded `queue.Queue()` (queue_consumer.py:185) and `SimpleQueueProcessingStrategy.submit` always enqueues without ever raising `MessageRejected`, so when `result_processor` lags the inbound Kafka rate the consumer keeps decoding/enqueuing and accumulating uncommitted offsets without bound, growing memory until OOM — directly contradicting the docstring's 'Natural backpressure when queues fill up'.
d5 · correctness · high
SimpleQueueProcessingStrategy.close()/join()/terminate() (queue_consumer.py:335-342) call queue_pool.shutdown() on the factory-shared FixedQueuePool during arroyo's partition-revocation rebalance, so the next create_with_partitions reuses the now-shut-down pool whose work_queue.put() raises queue.ShutDown — swallowed by submit()'s broad except, which then marks offsets complete, silently dropping all messages and committing their offsets after the first rebalance.

The change under review

diff (truncated — full diff)
diff --git a/src/sentry/consumers/__init__.py b/src/sentry/consumers/__init__.py
index 289c02ccd67..431c65b9bef 100644
--- a/src/sentry/consumers/__init__.py
+++ b/src/sentry/consumers/__init__.py
@@ -118,7 +118,7 @@ def uptime_options() -> list[click.Option]:
     options = [
         click.Option(
             ["--mode", "mode"],
-            type=click.Choice(["serial", "parallel", "batched-parallel"]),
+            type=click.Choice(["serial", "parallel", "batched-parallel", "thread-queue-parallel"]),
             default="serial",
             help="The mode to process results in. Parallel uses multithreading.",
         ),
@@ -138,7 +138,7 @@ def uptime_options() -> list[click.Option]:
             ["--max-workers", "max_workers"],
             type=int,
             default=None,
-            help="The maximum number of threads to spawn in parallel mode.",
+            help="The maximum amount of parallelism to use when in a parallel mode.",
         ),
         click.Option(["--processes", "num_processes"], default=1, type=int),
         click.Option(["--input-block-size"], type=int, default=None),
diff --git a/src/sentry/remote_subscriptions/consumers/queue_consumer.py b/src/sentry/remote_subscriptions/consumers/queue_consumer.py
new file mode 100644
index 00000000000..4eed5c7333d
--- /dev/null
+++ b/src/sentry/remote_subscriptions/consumers/queue_consumer.py
@@ -0,0 +1,345 @@
+from __future__ import annotations
+
+import logging
+import queue
+import threading
+import time
+from collections import defaultdict
+from collections.abc import Callable
+from dataclasses import dataclass
+from typing import Any, Generic, TypeVar
+
+import sentry_sdk
+from arroyo.backends.kafka.consumer import KafkaPayload
+from arroyo.processing.strategies import ProcessingStrategy
+from arroyo.types import BrokerValue, FilteredPayload, Message, Partition
+
+from sentry.utils import metrics
+
+logger = logging.getLogger(__name__)
+
+T = TypeVar("T")
+
+
+@dataclass
+class WorkItem(Generic[T]):
+    """Work item that includes the original message for offset tracking."""
+
+    partition: Partition
+    offset: int
+    result: T
+    message: Message[KafkaPayload | FilteredPayload]
+
+
+class OffsetTracker:
+    """
+    Tracks outstanding offsets and determines which offsets are safe to commit.
+
+    - Tracks offsets per partition
+    - Only commits offsets when all prior offsets are processed
+    - Thread-safe for concurrent access with per-partition locks
+    """
+
+    def __init__(self) -> None:
+        self.all_offsets: dict[Partition, set[int]] = defaultdict(set)
+        self.outstanding: dict[Partition, set[int]] = defaultdict(set)
+        self.last_committed: dict[Partition, int] = {}
+        self.partition_locks: dict[Partition, threading.Lock] = {}
+
+    def _get_partition_lock(self, partition: Partition) -> threading.Lock:
+        """Get or create a lock for a partition."""
+        lock = self.partition_locks.get(partition)
+        if lock:
+            return lock
+        return self.partition_locks.setdefault(partition, threading.Lock())
+
+    def add_offset(self, partition: Partition, offset: int) -> None:
+        """Record that we've started processing an offset."""
+        with self._get_partition_lock(partition):
+            self.all_offsets[partition].add(offset)
+            self.outstanding[partition].add(offset)
+
+    def complete_offset(self, partition: Partition, offset: int) -> None:
+        """Mark an offset as completed."""
+        with self._get_partition_lock(partition):
+            self.outstanding[partition].discard(offset)
+
+    def get_committable_offsets(self) -> dict[Partition, int]:
+        """
+        Get the highest offset per partition that can be safely committed.
+
+        For each partition, finds the highest contiguous offset that has been processed.
+        """
+        committable = {}
+        for partition in list(self.all_offsets.keys()):
+            with self._get_partition_lock(partition):
+                all_offsets = self.all_offsets[partition]
+                if not all_offsets:
+                    continue
+
+                outstanding = self.outstanding[partition]
+                last_committed = self.last_committed.get(partition, -1)
+
+                min_offset = min(all_offsets)
+                max_offset = max(all_offsets)
+
+                start = max(last_committed + 1, min_offset)
+
+                highest_committable = last_committed
+                for offset in range(start, max_offset + 1):
+                    if offset in all_offsets and offset not in outstanding:
+                        highest_committable = offset
+                    else:
+                        break
+
+                if highest_committable > last_committed:
+                    committable[partition] = highest_committable
+
+        return committable
+
+    def mark_committed(self, partition: Partition, offset: int) -> None:
+        """Update the last committed offset for a partition."""
+        with self._get_partition_lock(partition):
+            self.last_committed[partition] = offset
+            # Remove all offsets <= committed offset
+            self.all_offsets[partition] = {o for o in self.all_offsets[partition] if o > offset}
+
+
+class OrderedQueueWorker(threading.Thread, Generic[T]):
+    """Worker thread that processes items from a queue in order."""
+
+    def __init__(
+        self,
+        worker_id: int,
+        work_queue: queue.Queue[WorkItem[T]],
+        result_processor: Callable[[str, T], None],
+        identifier: str,
+        offset_tracker: OffsetTracker,
+    ) -> None:
+        super().__init__(daemon=True)
+        self.worker_id = worker_id
+        self.work_queue = work_queue
+        self.result_processor = result_processor
+        self.identifier = identifier
+        self.offset_tracker = offset_tracker
+        self.shutdown = False
+
+    def run(self) -> None:
+        """Process items from the queue in order."""
+        while not self.shutdown:
+            try:
+                work_item = self.work_queue.get()
+            except queue.ShutDown:
+                break
+
+            try:
+                with sentry_sdk.start_transaction(
+                    op="queue_worker.process",
+                    name=f"monitors.{self.identifier}.worker_{self.worker_id}",
+                ):
+                    self.result_processor(self.identifier, work_item.result)
+
+            except queue.ShutDown:
+                break
+            except Exception:
+                logger.exception(
+                    "Unexpected error in queue worker", extra={"worker_id": self.worker_id}
+                )
+            finally:
+                self.offset_tracker.complete_offset(work_item.partition, work_item.offset)
+                metrics.gauge(
+                    "remote_subscriptions.queue_worker.queue_depth",
+                    self.work_queue.qsize(),
+                    tags={
+                        "identifier": self.identifier,
+                    },
+                )
+
+
+class FixedQueuePool(Generic[T]):
+    """
+    Fixed pool of queues that guarantees order within groups.
+
+    Key properties:
+    - Each group is consistently assigned to the same queue
+    - Each queue has exactly one worker thread
+    - Items within a queue are processed in FIFO order
+    - No dynamic reassignment that could break ordering
+    - Tracks offset completion for safe commits
+    """
+
+    def __init__(
+        self,
+        result_processor: Callable[[str, T], None],
+        identifier: str,
+        num_queues: int = 20,
+    ) -> None:
+        self.result_processor = result_processor
+        self.identifier = identifier
+        self.num_queues = num_queues
+        self.offset_tracker = OffsetTracker()
+        self.queues: list[queue.Queue[WorkItem[T]]] = []
+        self.workers: list[OrderedQueueWorker[T]] = []
+
+        for i in range(num_queues):
+            work_queue: queue.Queue[WorkItem[T]] = queue.Queue()
+            self.queues.append(work_queue)
+
+            worker = OrderedQueueWorker[T](
+                worker_id=i,
+                work_queue=work_queue,
+                result_processor=result_processor,
+                identifier=identifier,
+                offset_tracker=self.offset_tracker,
+            )
+            worker.start()
+            self.workers.append(worker)
+
+    def get_queue_for_group(self, group_key: str) -> int:
+        """
+        Get queue index for a group using consistent hashing.
+        """
+        return hash(group_key) % self.num_queues
+
+    def submit(self, group_key: str, work_item: WorkItem[T]) -> None:
+        """
+        Submit a work item to the appropriate queue.
+        """
+        queue_index = self.get_queue_for_group(group_key)
+        work_queue = self.queues[queue_index]
+
+        self.offset_tracker.add_offset(work_item.partition, work_item.offset)
+        work_queue.put(work_item)
+
+    def get_stats(self) -> dict[str, Any]:
+        """Get statistics about queue depths."""
+        queue_depths = [q.qsize() for q in self.queues]
+        return {
+            "queue_depths": queue_depths,
+            "total_items": sum(queue_depths),
+        }
+
+    def wait_until_empty(self, timeout: float = 5.0) -> bool:
+        """Wait until all queues are empty. Returns True if successful, False if timeout."""
+        start_time = time.time()
+        while time.time() - start_time < timeout:
+            if self.get_stats()["total_items"] == 0:
+                return True
+            time.sleep(0.01)
+        return False
+
+    def shutdown(self) -> None:
+        """Gracefully shutdown all workers."""
+        for worker in self.workers:
+            worker.shutdown = True
+
+        for q in self.queues:
+            try:
+                q.shutdown(immediate=False)
+            except Exception:
+                logger.exception("Error shutting down queue")
+
+        for worker in self.workers:
+            worker.join(timeout=5.0)
+
+
+class SimpleQueueProcessingStrategy(ProcessingStrategy[KafkaPayload], Generic[T]):
+    """
+    Processing strategy that uses a fixed pool of queues.
+
+    Guarantees:
+    - Items for the same group are processed in order
+    - No item is lost or processed out of order
+    - Natural backpressure when queues fill up
+    - Only commits offsets after successful processing
+    """
+
+    def __init__(
+        self,
+        queue_pool: FixedQueuePool[T],
+        decoder: Callable[[KafkaPayload | FilteredPayload], T | None],
+        grouping_fn: Callable[[T], str],
+        commit_function: Callable[[dict[Partition, int]], None],
+    ) -> None:
+        self.queue_pool = queue_pool
+        self.decoder = decoder
+        self.grouping_fn = grouping_fn
+        self.commit_function = commit_function
+        self.shutdown_event = threading.Event()
+
+        self.commit_thread = threading.Thread(target=self._commit_loop, daemon=True)
+        self.commit_thread.start()
+
+    def _commit_loop(self) -> None:
+        while not self.shutdown_event.is_set():
+            try:
+                self.shutdown_event.wait(1.0)
+
+                committable = self.queue_pool.offset_tracker.get_committable_offsets()
+
+                if committable:
+                    metrics.incr(
+                        "remote_subscriptions.queue_pool.offsets_committed",
+                        len(committable),
+                        tags={"identifier": self.queue_pool.identifier},
+                    )
+
+                    self.commit_function(committable)
+                    for partition, offset in committable.items():
+                        self.queue_pool.offset_tracker.mark_committed(partition, offset)
+            except Exception:
+                logger.exception("Error in commit loop")
+
+    def submit(self, message: Message[KafkaPayload | FilteredPayload]) -> None:
+        try:
+            result = self.decoder(message.payload)
+
+            assert isinstance(message.value, BrokerValue)
+            partition = message.value.partition
+            offset = message.value.offset
+
+            if result is None:
+                self.queue_pool.offset_tracker.add_offset(partition, offset)
+                self.queue_pool.offset_tracker.complete_offset(partition, offset)
+                return
+
+            group_key = self.grouping_fn(result)
+
+            work_item = WorkItem(
+                partition=partition,
+                offset=offset,
+                result=result,
+                message=message,
+            )
+
+            self.queue_pool.submit(group_key, work_item)
+
+        except Exception:
+            logger.exception("Error submitting message to queue")
+            if isinstance(message.value, BrokerValue):
+                self.queue_pool.offset_tracker.add_offset(
+                    message.value.partition, message.value.offset
+                )
+                self.queue_pool.offset_tracker.complete_offset(
+                    message.value.partition, message.value.offset
+                )
+
+    def poll(self) -> None:
+        stats = self.queue_pool.get_stats()
+        metrics.gauge(
+            "remote_subscriptions.queue_pool.total_queued",
+            stats["total_items"],
+            tags={"identifier": self.queue_pool.identifier},
+        )
+
+    def close(self) -> None:
+        self.shutdown_event.set()
+        self.commit_thread.join(timeout=5.0)
+        self.queue_pool.shutdown()
+
+    def terminate(self) -> None:
+        self.shutdown_event.set()
+        self.queue_pool.shutdown()
+
+    def join(self, timeout: float | None = None) -> None:
+        self.close()
diff --git a/src/sentry/remote_subscriptions/consumers/result_consumer.py b/src/sentry/remote_subscriptions/consumers/result_consumer.py
index acdfb9c8d43..a6ab5424baa 100644
--- a/src/sentry/remote_subscriptions/consumers/result_consumer.py
+++ b/src/sentry/remote_subscriptions/consumers/result_consumer.py
@@ -19,6 +19,10 @@ from arroyo.processing.strategies.run_task import RunTask
 from arroyo.types import BrokerValue, Commit, FilteredPayload, Message, Partition
 
 from sentry.conf.types.kafka_definition import Topic, get_topic_codec
+from sentry.remote_subscriptions.consumers.queue_consumer import (
+    FixedQueuePool,
+    SimpleQueueProcessingStrategy,
+)
 from sentry.remote_subscriptions.models import BaseRemoteSubscription
 from sentry.utils import metrics
 from sentry.utils.arroyo import MultiprocessingPool, run_task_with_multiprocessing
@@ -89,13 +93,19 @@ class ResultsStrategyFactory(ProcessingStrategyFactory[KafkaPayload], Generic[T,
     Does the consumer process all messages in parallel.
     """
 
+    thread_queue_parallel = False
+    """
+    Does the consumer use thread-queue-parallel processing?
+    """
+
     multiprocessing_pool: MultiprocessingPool | None = None
+    queue_pool: FixedQueuePool | None = None
     input_block_size: int | None = None
     output_block_size: int | None = None
 
     def __init__(
         self,
-        mode: Literal["batched-parallel", "parallel", "serial"] = "serial",
+        mode: Literal["batched-parallel", "parallel", "serial", "thread-queue-parallel"] = "serial",
         max_batch_size: int | None = None,
         max_batch_time: int | None = None,
         max_workers: int | None = None,
@@ -105,6 +115,7 @@ class ResultsStrategyFactory(ProcessingStrategyFactory[KafkaPayload], Generic[T,
     ) -> None:
         self.mode = mode
         metric_tags = {"identifier": self.identifier, "mode": self.mode}
+        self.result_processor = self.result_processor_cls()
         if mode == "batched-parallel":
             self.batched_parallel = True
             self.parallel_executor = ThreadPoolExecutor(max_workers=max_workers)
@@ -117,6 +128,13 @@ class ResultsStrategyFactory(ProcessingStrategyFactory[KafkaPayload], Generic[T,
             if num_processes is None:
                 num_processes = multiprocessing.cpu_count()
             self.multiprocessing_pool = MultiprocessingPool(num_processes)
+        if mode == "thread-queue-parallel":
+            self.thread_queue_parallel = True
+            self.queue_pool = FixedQueuePool(
+                result_processor=self.result_processor,
+                identifier=self.identifier,
+                num_queues=max_workers or 20,  # Number of parallel queues
+            )
 
         metrics.incr(
             "remote_subscriptions.result_consumer.start",
@@ -133,8 +151,6 @@ class ResultsStrategyFactory(ProcessingStrategyFactory[KafkaPayload], Generic[T,
         if output_block_size is not None:
             self.output_block_size = output_block_size
 
-        self.result_processor = self.result_processor_cls()
-
     @property
     @abc.abstractmethod
     def topic_for_codec(self) -> Topic:
@@ -164,6 +180,9 @@ class ResultsStrategyFactory(ProcessingStrategyFactory[KafkaPayload], Generic[T,
     def shutdown(self) -> None:
         if self.parallel_executor:
             self.parallel_executor.shutdown()
+        if self.queue_pool:
+            self.queue_pool.shutdown()
+            self.queue_pool = None
 
     def decode_payload(self, topic_for_codec, payload: KafkaPayload | FilteredPayload) -> T | None:
         assert not isinstance(payload, FilteredPayload)
@@ -187,6 +206,8 @@ class ResultsStrategyFactory(ProcessingStrategyFactory[KafkaPayload], Generic[T,
             return self.create_thread_parallel_worker(commit)
         if self.parallel:
             return self.create_multiprocess_worker(commit)
+        if self.thread_queue_parallel:
+            return self.create_thread_queue_parallel_worker(commit)
         else:
             return self.create_serial_worker(commit)
 
@@ -220,6 +241,23 @@ class ResultsStrategyFactory(ProcessingStrategyFactory[KafkaPayload], Generic[T,
             next_step=batch_processor,
         )
 
+    def create_thread_queue_parallel_worker(
+        self, commit: Commit
+    ) -> ProcessingStrategy[KafkaPayload]:
+        assert self.queue_pool is not None
+
+        def commit_offsets(offsets: dict[Partition, int]):
+            # We add + 1 here because the committed offset should represent the next offset to read from
+            commit_data = {partition: offset + 1 for partition, offset in offsets.items()}
+            commit(commit_data)
+
+        return SimpleQueueProcessingStrategy(
+            queue_pool=self.queue_pool,
+            decoder=partial(self.decode_payload, self.topic_for_codec),
+            grouping_fn=self.build_payload_grouping_key,
+            commit_function=commit_offsets,
+        )
+
     def partition_message_batch(self, message: Message[ValuesBatch[KafkaPayload]]) -> list[list[T]]:
         """
         Takes a batch of messages and partitions them based on the `build_payload_grouping_key` method.
diff --git a/tests/sentry/remote_subscriptions/__init__.py b/tests/sentry/remote_subscriptions/__init__.py
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/tests/sentry/remote_subscriptions/consumers/__init__.py b/tests/sentry/remote_subscriptions/consumers/__init__.py
new file mode 100644
index 00000000000..e69de29bb2d
diff --git a/tests/sentry/remote_subscriptions/consumers/test_queue_consumer.py b/tests/sentry/remote_subscriptions/consumers/test_queue_consumer.py
new file mode 100644
index 00000000000..b94b9c0d485
--- /dev/null
+++ b/tests/sentry/remote_subscriptions/consumers/test_queue_consumer.py
@@ -0,0 +1,421 @@
+"""
+Tests for the thread-queue-parallel result consumer implementation.
+"""
+
+import threading

Leaderboard for this case

easy · 7 of 9 bots fully caught it (78% solve rate) · 89% avg coverage.

#BotResultCaughtCoverageFindingsFalse alarms
1 Claude Code (Opus 5.5) solved 2/2 100.0% 6 0
2 cursor solved 2/2 100.0% 2 0
3 Drape (Deep Review) solved 2/2 100.0% 4 0
4 Drape (Review) solved 2/2 100.0% 5 0
5 Mjolnir (Sonnet 5.5) solved 2/2 100.0% 6 0
6 Mjolnir (Opus 5.5) solved 2/2 100.0% 8 1
7 qodo solved 2/2 100.0% 5 1
8 greptile partial 1/2 50.0% 3 1
9 Cubic partial 1/2 50.0% 7 2

Bot results

Claude Code (Opus 5.5) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d4 high ✓ off by one
d5 high ✓ exact
valid · matches d5 · impact high
valid · impact medium
Arroyo's commit callback is invoked from the background _commit_loop thread, which runs until the strategy's close() is called; Arroyo's commit expects to run on the consumer thread, so this races with consumer state and revocation. [grounded: diff] [consensus 3/3: valid=3]
valid · matches d4 · impact high
valid · impact medium
The contiguous scan breaks at the first offset missing from all_offsets. A real Kafka offset gap (control markers, compaction) is never filled, so commits for that partition stop permanently and all_offsets keeps growing. [grounded: diff] [consensus 3/4: valid=3, invalid=1]
valid · impact low
One FixedQueuePool, and so one OffsetTracker, is shared by every strategy the factory creates, and nothing clears last_committed, all_offsets or outstanding on revoke or reassign. Stale state can suppress commits for offsets at or below the old last_committed, or block commits behind a leftover offset. [grounded: diff] [consensus 3/3: valid=3]
valid · impact low
Workers are told to stop before the queues are shut down, so they exit after their current item and leave queued work behind. The commit loop makes one last pass before the workers finish, and nothing commits afterwards, so the latest completed work is reprocessed after a restart. [grounded: diff] [consensus 3/3: valid=3]
raw comments (6)
src/sentry/remote_subscriptions/consumers/queue_consumer.py:336
On a partition rebalance, `close()` (and `terminate()`) shuts down the `FixedQueuePool`, but the factory reuses that same pool. `factory.queue_pool` is never reset, so `create_thread_queue_parallel_worker` hands the dead pool to the next strategy. After that, every `work_queue.put()` raises `queue.ShutDown`. The except branch in `submit()` catches it and marks the offset completed, so later messages are silently dropped and their offsets committed. The first rebalance permanently stops processing and loses data.
src/sentry/remote_subscriptions/consumers/queue_consumer.py:277
The Arroyo `commit` callback is called from a separate background thread (`_commit_loop`), not from the consumer thread in `poll()`/`submit()`. Arroyo's commit, which stages offsets on the consumer, is not thread-safe. This thread also keeps committing after partitions are revoked until `close()` runs, so it can commit offsets for partitions this consumer no longer owns. It also races with the consumer's own commit and close handling. Committing from `poll()` would avoid both problems.
src/sentry/remote_subscriptions/consumers/queue_consumer.py:183
Each queue is an unbounded `queue.Queue()`, and `submit()` never blocks or raises `MessageRejected`. So there is no backpressure, even though the docstring says there is. When processing falls behind, the consumer keeps reading from Kafka and buffering in memory without limit, and `OffsetTracker.all_offsets` grows too. Under sustained lag this can exhaust memory.
src/sentry/remote_subscriptions/consumers/queue_consumer.py:87
`get_committable_offsets` assumes offsets are contiguous: it stops at the first offset missing from `all_offsets`. Kafka offsets can have gaps, for example from transaction control markers, compaction, or a consumer seek. After one gap, nothing past it is ever committed for that partition. Commits stall for good, and `all_offsets` grows without bound because `mark_committed` never runs.
src/sentry/remote_subscriptions/consumers/queue_consumer.py:42
`OffsetTracker` state (`all_offsets`, `outstanding`, `last_committed`) is never cleared when partitions are revoked or reassigned. If a partition comes back starting at an offset at or below the stale `last_committed`, for example after another consumer committed less, those offsets are skipped and not committed correctly. Leftover outstanding offsets from the revoked assignment can also block commits.
src/sentry/remote_subscriptions/consumers/queue_consumer.py:336
`close()`/`join()` do not wait for queued items to finish or flush a final commit before shutting down. Each worker join is capped at 5 seconds, and the workers are daemon threads. Work that was in flight or queued at shutdown is abandoned, and the last batch of completed offsets is not committed, so those messages are reprocessed after a restart.
Cubic · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d4 high ✓ off by one
d5 high ✗ —
invalid
A FilteredPayload message still carries a BrokerValue as message.value. The decoder's AssertionError is caught, and the except branch adds and completes that offset, so the partition keeps advancing. [invalid: contradiction] [consensus 4/5: invalid=4, valid=1]
invalid
ResultProcessor.__call__ already swallows exceptions, so every existing mode (serial, batched) also commits past failed messages. Completing the offset in `finally` matches that intended behavior, and the test docstring says errors should not block commits. [invalid: contradiction] [consensus 5/5: invalid=5]
valid · impact medium
The queues have no maxsize and submit never blocks or rejects messages, so there is no backpressure. Memory grows without limit when processing falls behind, which contradicts the 'Natural backpressure' claim in the docstring. [grounded: diff] [consensus 5/5: valid=5]
valid · matches d4 · impact high
valid · impact low
Worker threads drain the queues while items are still being submitted, and the processor here is trivial. All items can be consumed before get_stats() runs, so this test can fail intermittently. [grounded: diff] [consensus 5/5: valid=5]
valid · impact low
The commit thread runs every second, independently of the test. It can commit a partial offset while the last item has been dequeued but not yet completed. The loop then exits early and the `== 105` assertion fails. [grounded: diff] [consensus 5/5: valid=5]
valid · impact low
Exceptions are swallowed and `finally` completes both offsets, so the commit loop would commit 102. The assertion that nothing was committed holds only because the test finishes before the 1-second commit tick, and it contradicts the docstring. [grounded: diff] [consensus 5/5: valid=5]
raw comments (8)
src/sentry/remote_subscriptions/consumers/queue_consumer.py:297
<!-- cubic:v=8be86228-b1b5-42a9-b5e9-0d8285c6ea03 --> <!-- cubic:review-run=82032aae-1432-5e16-b8be-b12f781d7b8d --> <!-- metadata:{"confidence":9} --> P1: When the stale-message wrapper forwards a `FilteredPayload`, this path rejects it and drops its partition offsets because the recovery branch only handles `BrokerValue`. Handle filtered payload metadata and mark those offsets complete so stale-only partitions continue advancing. <details> <summary>Prompt for AI agents</summary> ```text Check if this issue is valid — if so, understand the root cause and fix it. At src/sentry/remote_subscriptions/consumers/queue_consumer.py, line 297: <comment>When the stale-message wrapper forwards a `FilteredPayload`, this path rejects it and drops its partition offsets because the recovery branch only handles `BrokerValue`. Handle filtered payload metadata and mark those offsets complete so stale-only partitions continue advancing.</comment> <file context> @@ -0,0 +1,345 @@ + try: + result = self.decoder(message.payload) + + assert isinstance(message.value, BrokerValue) + partition = message.value.partition + offset = message.value.offset </file context> ``` </details>
src/sentry/remote_subscriptions/consumers/queue_consumer.py:149
<!-- cubic:v=ccb6b7f3-370a-4d02-9c52-25858de63d56 --> <!-- cubic:review-run=82032aae-1432-5e16-b8be-b12f781d7b8d --> <!-- metadata:{"confidence":10} --> P1: When `result_processor` raises, `finally` still marks the failed Kafka offset complete, so the commit loop can acknowledge the failed message and later contiguous messages. Complete offsets only after successful processing, and retain or retry failed work instead of treating it as complete. <details> <summary>Prompt for AI agents</summary> ```text Check if this issue is valid — if so, understand the root cause and fix it. At src/sentry/remote_subscriptions/consumers/queue_consumer.py, line 149: <comment>When `result_processor` raises, `finally` still marks the failed Kafka offset complete, so the commit loop can acknowledge the failed message and later contiguous messages. Complete offsets only after successful processing, and retain or retry failed work instead of treating it as complete.</comment> <file context> @@ -0,0 +1,345 @@ + "Unexpected error in queue worker", extra={"worker_id": self.worker_id} + ) + finally: + self.offset_tracker.complete_offset(work_item.partition, work_item.offset) + metrics.gauge( + "remote_subscriptions.queue_worker.queue_depth", </file context> ``` </details>
src/sentry/remote_subscriptions/consumers/result_consumer.py:133
<!-- cubic:v=cda85ae0-748d-4172-b9a1-f65bc99891e7 --> <!-- cubic:review-run=82032aae-1432-5e16-b8be-b12f781d7b8d --> <!-- metadata:{"confidence":9} --> P2: When result processing falls behind, this mode queues messages without limit because `FixedQueuePool` uses unbounded queues. Add bounded queues and backpressure before enabling this mode in production. <details> <summary>Prompt for AI agents</summary> ```text Check if this issue is valid — if so, understand the root cause and fix it. At src/sentry/remote_subscriptions/consumers/result_consumer.py, line 133: <comment>When result processing falls behind, this mode queues messages without limit because `FixedQueuePool` uses unbounded queues. Add bounded queues and backpressure before enabling this mode in production.</comment> <file context> @@ -117,6 +128,13 @@ def __init__( self.multiprocessing_pool = MultiprocessingPool(num_processes) + if mode == "thread-queue-parallel": + self.thread_queue_parallel = True + self.queue_pool = FixedQueuePool( + result_processor=self.result_processor, + identifier=self.identifier, </file context> ``` </details>
src/sentry/remote_subscriptions/consumers/queue_consumer.py:185
<!-- cubic:v=4fc32292-d53a-42e5-99b5-3fbad47e0dd6 --> <!-- cubic:review-run=82032aae-1432-5e16-b8be-b12f781d7b8d --> <!-- metadata:{"confidence":10} --> P2: The worker queues are unbounded, so `submit()` never applies backpressure when processing falls behind and memory can grow with Kafka lag. Use a bounded queue and make its capacity configurable. <details> <summary>Prompt for AI agents</summary> ```text Check if this issue is valid — if so, understand the root cause and fix it. At src/sentry/remote_subscriptions/consumers/queue_consumer.py, line 185: <comment>The worker queues are unbounded, so `submit()` never applies backpressure when processing falls behind and memory can grow with Kafka lag. Use a bounded queue and make its capacity configurable.</comment> <file context> @@ -0,0 +1,345 @@ + self.workers: list[OrderedQueueWorker[T]] = [] + + for i in range(num_queues): + work_queue: queue.Queue[WorkItem[T]] = queue.Queue() + self.queues.append(work_queue) + </file context> ``` </details>
tests/sentry/remote_subscriptions/consumers/test_queue_consumer.py:200
<!-- cubic:v=3265a2bc-9f81-43d4-a621-7a0ca00f682a --> <!-- cubic:review-run=82032aae-1432-5e16-b8be-b12f781d7b8d --> <!-- metadata:{"confidence":6} --> P2: In `test_stats_reporting`, the assertion `stats["total_items"] > 0` right after the submit loop is racy. The pool's worker threads consume items from the queues concurrently, and the submit loop does real work (constructing each WorkItem/Message), so the workers can drain all 10 items before `get_stats()` is called, making `total_items` 0 and failing the assertion intermittently. Use an event/flag that records whether any item was observed queued, or assert on the final drained state instead of an immediate `> 0` check. <details> <summary>Prompt for AI agents</summary> ```text Check if this issue is valid — if so, understand the root cause and fix it. At tests/sentry/remote_subscriptions/consumers/test_queue_consumer.py, line 200: <comment>In `test_stats_reporting`, the assertion `stats["total_items"] > 0` right after the submit loop is racy. The pool's worker threads consume items from the queues concurrently, and the submit loop does real work (constructing each WorkItem/Message), so the workers can drain all 10 items before `get_stats()` is called, making `total_items` 0 and failing the assertion intermittently. Use an event/flag that records whether any item was observed queued, or assert on the final drained state instead of an immediate `> 0` check.</comment> <file context> @@ -0,0 +1,421 @@ + self.pool.submit(group_key, work_item) + + stats = self.pool.get_stats() + assert stats["total_items"] > 0 + assert "queue_depths" in stats + assert len(stats["queue_depths"]) == 3 </file context> ``` </details>
tests/sentry/uptime/consumers/test_results_consumer.py:1870
<!-- cubic:v=090ad80d-0b47-4e85-9d0a-0d3a94451fb4 --> <!-- cubic:review-run=82032aae-1432-5e16-b8be-b12f781d7b8d --> <!-- metadata:{"confidence":6} --> P2: test_thread_queue_parallel_offset_commit can be flaky: the wait loop breaks when `total_items == 0 and len(committed_offsets) > 0`, but total_items is the queue depth, not completion. A worker can have dequeued the last message but not yet called complete_offset() when the 1-second commit thread fires a partial commit, so the loop breaks early and `assert committed_offsets[test_partition] == 105` fails. Break on the actual final commit value (or poll until committed_offsets[partition] reaches 105) instead of queue depth, and apply the same fix to the offset_gaps and error_handling loops. <details> <summary>Prompt for AI agents</summary> ```text Check if this issue is valid — if so, understand the root cause and fix it. At tests/sentry/uptime/consumers/test_results_consumer.py, line 1870: <comment>test_thread_queue_parallel_offset_commit can be flaky: the wait loop breaks when `total_items == 0 and len(committed_offsets) > 0`, but total_items is the queue depth, not completion. A worker can have dequeued the last message but not yet called complete_offset() when the 1-second commit thread fires a partial commit, so the loop breaks early and `assert committed_offsets[test_partition] == 105` fails. Break on the actual final commit value (or poll until committed_offsets[partition] reaches 105) instead of queue depth, and apply the same fix to the offset_gaps and error_handling loops.</comment> <file context> @@ -1675,6 +1686,461 @@ def test_parallel_grouping(self, mock_process_group) -> None: + time.sleep(0.1) + + assert test_partition in committed_offsets + assert committed_offsets[test_partition] == 105 + + factory.shutdown() </file context> ``` </details>
tests/sentry/uptime/consumers/test_results_consumer.py:1922
<!-- cubic:v=e3fae107-8928-416c-9cb1-651232f6f7b4 --> <!-- cubic:review-run=82032aae-1432-5e16-b8be-b12f781d7b8d --> <!-- metadata:{"confidence":8} --> P2: test_thread_queue_parallel_error_handling asserts the opposite of its stated intent and of the implementation. Its docstring says errors shouldn't block offset commits for other messages, but the final assertion `assert len(committed_offsets) == 0 or test_partition not in committed_offsets` asserts that nothing was committed at all. In OrderedQueueWorker.run(), the processing exception is swallowed and complete_offset() always runs in `finally`, so both offsets complete and the commit loop commits offset 102. The assertion only holds because the test checks committed_offsets before the 1-second commit thread fires, so the test is flaky and verifies the opposite behavior of what it claims. Align the assertion with the intended behavior (e.g. assert the successful message's offset is committed despite the earlier error). <details> <summary>Prompt for AI agents</summary> ```text Check if this issue is valid — if so, understand the root cause and fix it. At tests/sentry/uptime/consumers/test_results_consumer.py, line 1922: <comment>test_thread_queue_parallel_error_handling asserts the opposite of its stated intent and of the implementation. Its docstring says errors shouldn't block offset commits for other messages, but the final assertion `assert len(committed_offsets) == 0 or test_partition not in committed_offsets` asserts that nothing was committed at all. In OrderedQueueWorker.run(), the processing exception is swallowed and complete_offset() always runs in `finally`, so both offsets complete and the commit loop commits offset 102. The assertion only holds because the test checks committed_offsets before the 1-second commit thread fires, so the test is flaky and verifies the opposite behavior of what it claims. Align the assertion with the intended behavior (e.g. assert the successful message's offset is committed despite the earlier error).</comment> <file context> @@ -1675,6 +1686,461 @@ def test_parallel_grouping(self, mock_process_group) -> None: + time.sleep(0.1) + + assert mock_processor_call.call_count == 2 + assert len(committed_offsets) == 0 or test_partition not in committed_offsets + + factory.shutdown() </file context> ``` </details>
PR-level
<!-- cubic:review-summary:start --> **7 issues found** across 7 files <!-- cubic:review-summary:end --> <details> <summary>Prompt for AI agents (unresolved issues)</summary> ```text Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately. <file name="src/sentry/remote_subscriptions/consumers/result_consumer.py"> <violation number="1" location="src/sentry/remote_subscriptions/consumers/result_consumer.py:133"> P2: When result processing falls behind, this mode queues messages without limit because `FixedQueuePool` uses unbounded queues. Add bounded queues and backpressure before enabling this mode in production.</violation> </file> <file name="src/sentry/remote_subscriptions/consumers/queue_consumer.py"> <violation number="1" location="src/sentry/remote_subscriptions/consumers/queue_consumer.py:149"> P1: When `result_processor` raises, `finally` still marks the failed Kafka offset complete, so the commit loop can acknowledge the failed message and later contiguous messages. Complete offsets only after successful processing, and retain or retry failed work instead of treating it as complete.</violation> <violation number="2" location="src/sentry/remote_subscriptions/consumers/queue_consumer.py:185"> P2: The worker queues are unbounded, so `submit()` never applies backpressure when processing falls behind and memory can grow with Kafka lag. Use a bounded queue and make its capacity configurable.</violation> <violation number="3" location="src/sentry/remote_subscriptions/consumers/queue_consumer.py:297"> P1: When the stale-message wrapper forwards a `FilteredPayload`, this path rejects it and drops its partition offsets because the recovery branch only handles `BrokerValue`. Handle filtered payload metadata and mark those offsets complete so stale-only partitions continue advancing.</violation> </file> <file name="tests/sentry/remote_subscriptions/consumers/test_queue_consumer.py"> <violation number="1" location="tests/sentry/remote_subscriptions/consumers/test_queue_consumer.py:200"> P2: In `test_stats_reporting`, the assertion `stats["total_items"] > 0` right after the submit loop is racy. The pool's worker threads consume items from the queues concurrently, and the submit loop does real work (constructing each WorkItem/Message), so the workers can drain all 10 items before `get_stats()` is called, making `total_items` 0 and failing the assertion intermittently. Use an event/flag that records whether any item was observed queued, or assert on the final drained state instead of an immediate `> 0` check.</violation> </file> <file name="tests/sentry/uptime/consumers/test_results_consumer.py"> <violation number="1" location="tests/sentry/uptime/consumers/test_results_consumer.py:1870"> P2: test_thread_queue_parallel_offset_commit can be flaky: the wait loop breaks when `total_items == 0 and len(committed_offsets) > 0`, but total_items is the queue depth, not completion. A worker can have dequeued the last message but not yet called complete_offset() when the 1-second commit thread fires a partial commit, so the loop breaks early and `assert committed_offsets[test_partition] == 105` fails. Break on the actual final commit value (or poll until committed_offsets[partition] reaches 105) instead of queue depth, and apply the same fix to the offset_gaps and error_handling loops.</violation> <violation number="2" location="tests/sentry/uptime/consumers/test_results_consumer.py:1922"> P2: test_thread_queue_parallel_error_handling asserts the opposite of its stated intent and of the implementation. Its docstring says errors shouldn't block offset commits for other messages, but the final assertion `assert len(committed_offsets) == 0 or test_partition not in committed_offsets` asserts that nothing was committed at all. In OrderedQueueWorker.run(), the processing exception is swallowed and complete_offset() always runs in `finally`, so both offsets complete and the commit loop commits offset 102. The assertion only holds because the test checks committed_offsets before the 1-second commit thread fires, so the test is flaky and verifies the opposite behavior of what it claims. Align the assertion with the intended behavior (e.g. assert the successful message's offset is committed despite the earlier error).</violation> </file> ``` </details> <sub>Reply with feedback, questions, or to request a fix.<br /><br />[Re-trigger cubic](https://www.cubic.dev/action/re-review/pr/drape-io/crbench-cubic-2026-09/95/ai_pr_review_github_delivery_ca36ea2bc88b6db1509e10b8d528188e2e80ad2701373322e9d4e1c7ff131a18?returnTo=https%3A%2F%2Fgithub.com%2Fdrape-io%2Fcrbench-cubic-2026-09%2Fpull%2F95)</sub> <!-- cubic:review-post:ai_pr_review_github_delivery_ca36ea2bc88b6db1509e10b8d528188e2e80ad2701373322e9d4e1c7ff131a18:10dc6b92dd902163b1c51bda47da90d68ffc33d5:82032aae-1432-5e16-b8be-b12f781d7b8d -->
cursor · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d4 high ✓ off by one
d5 high ✓ exact
valid · matches d5 · impact high
valid · matches d4 · impact high
raw comments (3)
src/sentry/remote_subscriptions/consumers/queue_consumer.py:345
### Strategy close kills shared queue pool **High Severity** <!-- DESCRIPTION START --> `FixedQueuePool` is created once on `ResultsStrategyFactory` and reused across partition assignments, but `SimpleQueueProcessingStrategy.close()`, `terminate()`, and `join()` all call `queue_pool.shutdown()`. Arroyo closes the strategy and then calls `create_with_partitions()` again on rebalance. After that, `put()` fails on the shut-down queues, `submit()` marks the offset complete without running the processor, and results are skipped. `join()` also calls `close()` instead of draining in-flight work, and `shutdown()` sets `worker.shutdown` before the queues empty, so leftover items are dropped. <!-- DESCRIPTION END --> <!-- BUGBOT_BUG_ID: 85518e31-a1f8-4637-90ca-3846b407d04c --> <!-- LOCATIONS START src/sentry/remote_subscriptions/consumers/queue_consumer.py#L334-L345 src/sentry/remote_subscriptions/consumers/result_consumer.py#L130-L137 src/sentry/remote_subscriptions/consumers/queue_consumer.py#L230-L243 LOCATIONS END --> <details> <summary>Additional Locations (2)</summary> - [`src/sentry/remote_subscriptions/consumers/result_consumer.py#L130-L137`](https://github.com/drape-io/crbench-cursor-2026-09/blob/3b630533684ca2b856f81c945cc03d1aab6043a1/src/sentry/remote_subscriptions/consumers/result_consumer.py#L130-L137) - [`src/sentry/remote_subscriptions/consumers/queue_consumer.py#L230-L243`](https://github.com/drape-io/crbench-cursor-2026-09/blob/3b630533684ca2b856f81c945cc03d1aab6043a1/src/sentry/remote_subscriptions/consumers/queue_consumer.py#L230-L243) </details> <div><a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9DVVJTT1IiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjkxNjUyZDU3LWQxZTgtNDQxYy1iZDBmLWY3N2JlYWQ0Y2JkZCIsImVuY3J5cHRpb25LZXkiOiJVZzZqQVVkYldHeE1qUGRpSWp4cnB0elRlYWwzbzA2RWxtWS1YU3lWamlZIiwiYnJhbmNoIjoidDItc2VudHJ5LTk1NjMzIiwicmVwb093bmVyIjoiZHJhcGUtaW8iLCJyZXBvTmFtZSI6ImNyYmVuY2gtY3Vyc29yLTIwMjYtMDkiLCJwcm92aWRlciI6ImdpdGh1YiJ9fQ" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/agents?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9XRUIiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjkxNjUyZDU3LWQxZTgtNDQxYy1iZDBmLWY3N2JlYWQ0Y2JkZCIsImVuY3J5cHRpb25LZXkiOiJVZzZqQVVkYldHeE1qUGRpSWp4cnB0elRlYWwzbzA2RWxtWS1YU3lWamlZIiwiYnJhbmNoIjoidDItc2VudHJ5LTk1NjMzIiwicmVwb093bmVyIjoiZHJhcGUtaW8iLCJyZXBvTmFtZSI6ImNyYmVuY2gtY3Vyc29yLTIwMjYtMDkiLCJwck51bWJlciI6NzcsImNvbW1pdFNoYSI6IjNiNjMwNTMzNjg0Y2EyYjg1NmY4MWM5NDVjYzAzZDFhYWI2MDQzYTEiLCJwcm92aWRlciI6ImdpdGh1YiJ9fQ" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-web-light.png"><img alt="Fix in Web" width="99" height="28" src="https://cursor.com/assets/images/fix-in-web-dark.png"></picture></a></div> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 3b630533684ca2b856f81c945cc03d1aab6043a1. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
src/sentry/remote_subscriptions/consumers/queue_consumer.py:212
### Queue pool has no backpressure **Medium Severity** <!-- DESCRIPTION START --> `FixedQueuePool` builds unbounded `queue.Queue()` instances, and `submit()` always `put()`s without raising `MessageRejected`. The strategy docstring claims natural backpressure when queues fill up, but nothing bounds queue depth or pauses the Arroyo consumer, so lag can grow memory without limit. <!-- DESCRIPTION END --> <!-- BUGBOT_BUG_ID: 92d9f8e5-b316-46c0-b72d-e48867ea03c5 --> <!-- LOCATIONS START src/sentry/remote_subscriptions/consumers/queue_consumer.py#L183-L212 src/sentry/remote_subscriptions/consumers/queue_consumer.py#L252-L253 LOCATIONS END --> <details> <summary>Additional Locations (1)</summary> - [`src/sentry/remote_subscriptions/consumers/queue_consumer.py#L252-L253`](https://github.com/drape-io/crbench-cursor-2026-09/blob/3b630533684ca2b856f81c945cc03d1aab6043a1/src/sentry/remote_subscriptions/consumers/queue_consumer.py#L252-L253) </details> <div><a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9DVVJTT1IiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OmEzOWI3ZmVkLWQzNmMtNGYzNC1hMWIwLTRkOTYyZjg1MTMxNSIsImVuY3J5cHRpb25LZXkiOiJrZE50UGZTdGV6Q2wwdmlSMEtrLTZoM1ZEdmhzSEVYOUE3dEhvTHpIV1kwIiwiYnJhbmNoIjoidDItc2VudHJ5LTk1NjMzIiwicmVwb093bmVyIjoiZHJhcGUtaW8iLCJyZXBvTmFtZSI6ImNyYmVuY2gtY3Vyc29yLTIwMjYtMDkiLCJwcm92aWRlciI6ImdpdGh1YiJ9fQ" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/agents?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9XRUIiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OmEzOWI3ZmVkLWQzNmMtNGYzNC1hMWIwLTRkOTYyZjg1MTMxNSIsImVuY3J5cHRpb25LZXkiOiJrZE50UGZTdGV6Q2wwdmlSMEtrLTZoM1ZEdmhzSEVYOUE3dEhvTHpIV1kwIiwiYnJhbmNoIjoidDItc2VudHJ5LTk1NjMzIiwicmVwb093bmVyIjoiZHJhcGUtaW8iLCJyZXBvTmFtZSI6ImNyYmVuY2gtY3Vyc29yLTIwMjYtMDkiLCJwck51bWJlciI6NzcsImNvbW1pdFNoYSI6IjNiNjMwNTMzNjg0Y2EyYjg1NmY4MWM5NDVjYzAzZDFhYWI2MDQzYTEiLCJwcm92aWRlciI6ImdpdGh1YiJ9fQ" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-web-light.png"><img alt="Fix in Web" width="99" height="28" src="https://cursor.com/assets/images/fix-in-web-dark.png"></picture></a></div> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 3b630533684ca2b856f81c945cc03d1aab6043a1. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
PR-level
<!-- BUGBOT_REVIEW --> Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues. <!-- BUGBOT_FIX_ALL --> <a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9BTExfSU5fQ1VSU09SIiwiZGF0YSI6eyJyZWRpc0tleSI6ImJ1Z2JvdC1tdWx0aToxYzNmYThiNi00ZDAyLTRkMGUtODc5ZS0xNTg4YjQ3NWJiN2UiLCJlbmNyeXB0aW9uS2V5IjoiYmpLV29DZ3YwQkp1RDJyNU05X1BJYjF1Sk5Ea1ZYeDByek5yN0ptbnB1QSIsImJyYW5jaCI6InQyLXNlbnRyeS05NTYzMyIsInJlcG9Pd25lciI6ImRyYXBlLWlvIiwicmVwb05hbWUiOiJjcmJlbmNoLWN1cnNvci0yMDI2LTA5IiwicHJvdmlkZXIiOiJnaXRodWIifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix All in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a> <!-- /BUGBOT_FIX_ALL --> <!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_BEGIN --> <sup>❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the [Cursor dashboard](https://www.cursor.com/dashboard/bugbot).</sup> <!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_END --> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 3b630533684ca2b856f81c945cc03d1aab6043a1. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
Drape (Deep Review) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d4 high ✓ off by one
d5 high ✓ off by one
valid · matches d4 · impact high
valid · impact medium
close() sets shutdown_event, so the commit loop runs one last commit and exits before queue_pool.shutdown() runs. Workers loop on `while not self.shutdown`, so they exit without draining their queues. Items still queued, and any in-flight item that finishes after that last commit, never get committed and will be reprocessed. [grounded: tree src/sentry/remote_subscriptions/consumers/queue_consumer.py] [consensus 3/3: valid=3]
valid · impact medium
The factory hands the same FixedQueuePool, and so the same OffsetTracker, to every new strategy, and nothing ever prunes partitions that were revoked. If a revoked partition's offset completes after the last commit, it shows up in the next strategy's commit. Arroyo rejects staging offsets for unassigned partitions, so mark_committed is skipped and the same failing commit repeats on every loop. [grounded: tree src/sentry/remote_subscriptions/consumers/queue_consumer.py] [consensus 3/3: valid=3]
valid · matches d5 · impact high
raw comments (4)
src/sentry/remote_subscriptions/consumers/queue_consumer.py:182
**Queues are unbounded, so nothing applies backpressure** Each queue is a plain `queue.Queue()` with no maxsize, and `submit`/`poll` never raise `MessageRejected`. When processing falls behind, the consumer keeps reading Kafka and buffers everything in memory, despite the docstring promising backpressure. Give the queues a maxsize and raise `MessageRejected` from `submit` when the target queue is full. ``` src/sentry/remote_subscriptions/consumers/queue_consumer.py:181 self.queues: list[queue.Queue[WorkItem[T]]] = [] src/sentry/remote_subscriptions/consumers/queue_consumer.py:182 self.workers: list[OrderedQueueWorker[T]] = [] src/sentry/remote_subscriptions/consumers/queue_consumer.py:208 queue_index = self.get_queue_for_group(group_key) src/sentry/remote_subscriptions/consumers/queue_consumer.py:209 work_queue = self.queues[queue_index] src/sentry/remote_subscriptions/consumers/queue_consumer.py:324 message.value.partition, message.value.offset src/sentry/remote_subscriptions/consumers/queue_consumer.py:325 ) src/sentry/remote_subscriptions/consumers/queue_consumer.py:327 def poll(self) -> None: src/sentry/remote_subscriptions/consumers/queue_consumer.py:328 stats = self.queue_pool.get_stats() src/sentry/remote_subscriptions/consumers/queue_consumer.py:329 metrics.gauge( src/sentry/remote_subscriptions/consumers/queue_consumer.py:330 "remote_subscriptions.queue_pool.total_queued", ```
src/sentry/remote_subscriptions/consumers/queue_consumer.py:344
**Shutdown abandons queued work and its final offsets** On revoke or shutdown, close() makes its last commit before stopping workers. Workers then exit without draining their queues, so queued items and in-flight completions are never committed and get reprocessed. Make join() drain the queues (honouring timeout) and commit once more after the workers finish. ``` src/sentry/remote_subscriptions/consumers/queue_consumer.py:333 ) src/sentry/remote_subscriptions/consumers/queue_consumer.py:335 def close(self) -> None: src/sentry/remote_subscriptions/consumers/queue_consumer.py:336 self.shutdown_event.set() src/sentry/remote_subscriptions/consumers/queue_consumer.py:337 self.commit_thread.join(timeout=5.0) src/sentry/remote_subscriptions/consumers/queue_consumer.py:262 commit_function: Callable[[dict[Partition, int]], None], src/sentry/remote_subscriptions/consumers/queue_consumer.py:263 ) -> None: src/sentry/remote_subscriptions/consumers/queue_consumer.py:264 self.queue_pool = queue_pool src/sentry/remote_subscriptions/consumers/queue_consumer.py:265 self.decoder = decoder src/sentry/remote_subscriptions/consumers/queue_consumer.py:266 self.grouping_fn = grouping_fn src/sentry/remote_subscriptions/consumers/queue_consumer.py:127 def run(self) -> None: ```
src/sentry/remote_subscriptions/consumers/queue_consumer.py:73
**Revoked partitions block all later commits** The shared `OffsetTracker` keeps state for partitions after they're revoked. The next strategy's commit loop still includes them, arroyo rejects staging offsets for unassigned partitions, and since `mark_committed` never runs, every later commit fails too. Clear tracker state for partitions that aren't in the new assignment when `create_with_partitions`/`close` runs. ``` src/sentry/remote_subscriptions/consumers/queue_consumer.py:43 def __init__(self) -> None: src/sentry/remote_subscriptions/consumers/queue_consumer.py:44 self.all_offsets: dict[Partition, set[int]] = defaultdict(set) src/sentry/remote_subscriptions/consumers/queue_consumer.py:45 self.outstanding: dict[Partition, set[int]] = defaultdict(set) src/sentry/remote_subscriptions/consumers/queue_consumer.py:46 self.last_committed: dict[Partition, int] = {} src/sentry/remote_subscriptions/consumers/queue_consumer.py:73 committable = {} src/sentry/remote_subscriptions/consumers/queue_consumer.py:280 if committable: src/sentry/remote_subscriptions/consumers/queue_consumer.py:281 metrics.incr( src/sentry/remote_subscriptions/consumers/queue_consumer.py:282 "remote_subscriptions.queue_pool.offsets_committed", src/sentry/remote_subscriptions/consumers/queue_consumer.py:283 len(committable), src/sentry/remote_subscriptions/consumers/queue_consumer.py:284 tags={"identifier": self.queue_pool.identifier}, ```
src/sentry/remote_subscriptions/consumers/queue_consumer.py:336
**Results silently dropped after first rebalance** `close()` shuts down the shared `queue_pool` that later strategies reuse. After a rebalance, `put` raises `ShutDown`, and `submit` swallows it and marks the offset complete, so messages are committed unprocessed. Let the factory's `shutdown()` own the pool, or create one pool per strategy. ``` src/sentry/remote_subscriptions/consumers/queue_consumer.py:333 ) src/sentry/remote_subscriptions/consumers/queue_consumer.py:335 def close(self) -> None: src/sentry/remote_subscriptions/consumers/queue_consumer.py:336 self.shutdown_event.set() src/sentry/remote_subscriptions/consumers/queue_consumer.py:237 try: src/sentry/remote_subscriptions/consumers/queue_consumer.py:302 self.queue_pool.offset_tracker.add_offset(partition, offset) src/sentry/remote_subscriptions/consumers/queue_consumer.py:303 self.queue_pool.offset_tracker.complete_offset(partition, offset) src/sentry/remote_subscriptions/consumers/queue_consumer.py:304 return src/sentry/remote_subscriptions/consumers/queue_consumer.py:306 group_key = self.grouping_fn(result) src/sentry/remote_subscriptions/consumers/queue_consumer.py:308 work_item = WorkItem( src/sentry/remote_subscriptions/consumers/queue_consumer.py:309 partition=partition, ```
Drape (Review) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d4 high ✓ off by one
d5 high ✓ exact
valid · matches d4 · impact high
valid · matches d5 · impact high
valid · impact medium
The FixedQueuePool and its OffsetTracker are created once in the factory and shared by every strategy, and nothing clears partition state when partitions are revoked. The arroyo `commit` callable is called from a daemon thread rather than the consumer thread, which is not how arroyo's consumer-side commit is meant to be used. [grounded: diff] [consensus 3/3: valid=3]
valid · impact low
FixedQueuePool.shutdown sets worker.shutdown=True before queue shutdown, so a worker exits its loop after the current item and leaves items still queued. close() also stops and joins the commit thread before calling queue_pool.shutdown(), so there is no final commit after the workers drain. [grounded: diff] [consensus 3/3: valid=3]
valid · impact low
In submit's exception handler, the offset is added and immediately marked complete. A message whose grouping_fn or submission fails is therefore skipped and committed, which contradicts the docstring's 'No item is lost' and 'Only commits offsets after successful processing'. The behavior matches the existing serial mode, so the real-world impact is low. [grounded: diff] [consensus 3/3: valid=3]
raw comments (5)
src/sentry/remote_subscriptions/consumers/queue_consumer.py:176
**Unbounded queues give no backpressure** The docstring promises backpressure, but each worker queue is a plain `queue.Queue()` with no maxsize, and `submit` never rejects. A slow processor lets the consumer read without limit. Use bounded queues and raise arroyo MessageRejected from submit when full. ``` src/sentry/remote_subscriptions/consumers/queue_consumer.py:176 ) -> None: src/sentry/remote_subscriptions/consumers/queue_consumer.py:177 self.result_processor = result_processor src/sentry/remote_subscriptions/consumers/queue_consumer.py:204 def submit(self, group_key: str, work_item: WorkItem[T]) -> None: src/sentry/remote_subscriptions/consumers/queue_consumer.py:205 """ ```
src/sentry/remote_subscriptions/consumers/queue_consumer.py:340
**Rebalance kills the shared worker pool** The queue pool is created once in the factory, but each strategy's close() shuts down that shared pool. After a rebalance the next strategy submits to queues whose workers have exited. Shut the pool down only in the factory's shutdown(); strategy close() should just stop its commit thread. ``` src/sentry/remote_subscriptions/consumers/queue_consumer.py:333 ) src/sentry/remote_subscriptions/consumers/queue_consumer.py:335 def close(self) -> None: src/sentry/remote_subscriptions/consumers/queue_consumer.py:336 self.shutdown_event.set() src/sentry/remote_subscriptions/consumers/queue_consumer.py:337 self.commit_thread.join(timeout=5.0) src/sentry/remote_subscriptions/consumers/queue_consumer.py:338 self.queue_pool.shutdown() src/sentry/remote_subscriptions/consumers/queue_consumer.py:340 def terminate(self) -> None: src/sentry/remote_subscriptions/consumers/queue_consumer.py:341 self.shutdown_event.set() src/sentry/remote_subscriptions/consumers/queue_consumer.py:342 self.queue_pool.shutdown() src/sentry/remote_subscriptions/consumers/queue_consumer.py:344 def join(self, timeout: float | None = None) -> None: src/sentry/remote_subscriptions/consumers/queue_consumer.py:345 self.close() ```
src/sentry/remote_subscriptions/consumers/queue_consumer.py:282
**Commit thread commits stale and revoked offsets** The shared OffsetTracker is never cleared on rebalance, so the commit thread keeps committing partitions no longer assigned. It also calls commit from a background thread, not the consumer thread. Call commit from poll() on the consumer thread; scope the tracker to the strategy's partitions. ``` src/sentry/remote_subscriptions/consumers/queue_consumer.py:268 self.shutdown_event = threading.Event() src/sentry/remote_subscriptions/consumers/queue_consumer.py:270 self.commit_thread = threading.Thread(target=self._commit_loop, daemon=True) src/sentry/remote_subscriptions/consumers/queue_consumer.py:271 self.commit_thread.start() src/sentry/remote_subscriptions/consumers/queue_consumer.py:273 def _commit_loop(self) -> None: src/sentry/remote_subscriptions/consumers/queue_consumer.py:274 while not self.shutdown_event.is_set(): src/sentry/remote_subscriptions/consumers/queue_consumer.py:275 try: src/sentry/remote_subscriptions/consumers/queue_consumer.py:276 self.shutdown_event.wait(1.0) src/sentry/remote_subscriptions/consumers/queue_consumer.py:278 committable = self.queue_pool.offset_tracker.get_committable_offsets() src/sentry/remote_subscriptions/consumers/queue_consumer.py:280 if committable: src/sentry/remote_subscriptions/consumers/queue_consumer.py:281 metrics.incr( ```
src/sentry/remote_subscriptions/consumers/queue_consumer.py:239
**Shutdown drops queued items, no final commit** `shutdown` sets `worker.shutdown = True`, and workers check it before each get, so they exit without draining the queue despite `immediate=False`. `close()` also stops the commit thread before the workers finish. Rely on ShutDown to stop workers, then do a final commit after joining them. ``` src/sentry/remote_subscriptions/consumers/queue_consumer.py:136 with sentry_sdk.start_transaction( src/sentry/remote_subscriptions/consumers/queue_consumer.py:137 op="queue_worker.process", src/sentry/remote_subscriptions/consumers/queue_consumer.py:231 def shutdown(self) -> None: src/sentry/remote_subscriptions/consumers/queue_consumer.py:232 """Gracefully shutdown all workers.""" ```
src/sentry/remote_subscriptions/consumers/queue_consumer.py:264
**Docstring promises no loss, but errors are committed** The docstring says no item is lost and offsets commit only after success. But `submit` catches any exception, such as `grouping_fn` raising, and marks the offset complete anyway. Don't complete the offset in the submit error path, or correct the docstring. ``` src/sentry/remote_subscriptions/consumers/queue_consumer.py:255 """ src/sentry/remote_subscriptions/consumers/queue_consumer.py:257 def __init__( src/sentry/remote_subscriptions/consumers/queue_consumer.py:258 self, src/sentry/remote_subscriptions/consumers/queue_consumer.py:259 queue_pool: FixedQueuePool[T], ```
greptile · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d4 high ✓ exact
d5 high ✗ —
invalid
The result_processor passed to the pool is a ResultProcessor whose __call__ swallows all exceptions, so it never raises into the worker. Failed results are already committed in the existing serial mode too, so this is the established at-most-once design, not a new loss introduced here. [invalid: contradiction] [consensus 3/3: invalid=3]
valid · impact low
The worker loop is `while not self.shutdown`, so once the flag is set a worker exits after its current item and leaves queued items unprocessed, which defeats Queue.shutdown(immediate=False). close() also stops the commit thread before the pool shuts down, so offsets completed during shutdown are never committed. This causes replays and duplicate work on shutdown or rebalance. [grounded: diff] [consensus 3/3: valid=3]
valid · matches d4 · impact high
raw comments (4)
src/sentry/remote_subscriptions/consumers/queue_consumer.py:148
<a href="#"><img alt="P1" src="https://greptile-static-assets.s3.amazonaws.com/badges/p1.svg?v=9" align="top"></a> **Failed work gets committed** When `result_processor` raises, this `finally` block still marks the offset complete. The commit loop can then advance past a result whose side effects never succeeded, permanently discarding it; the new error-handling test expects processing failures to remain uncommitted. The submission error handler at lines 317–325 has the same problem when grouping or queue submission fails.
src/sentry/remote_subscriptions/consumers/queue_consumer.py:244
<a href="#"><img alt="P1" src="https://greptile-static-assets.s3.amazonaws.com/badges/p1.svg?v=9" align="top"></a> **Shutdown abandons pending work** A graceful shutdown sets each worker's stop flag before calling `Queue.shutdown(immediate=False)`, so workers can exit without consuming remaining items. Strategy shutdown also stops the commit thread before shutting down the workers. When work is queued or in flight, processing side effects can remain unfinished and newly completed offsets can remain uncommitted, causing delayed replays and potentially duplicate side effects.
src/sentry/remote_subscriptions/consumers/queue_consumer.py:184
<a href="#"><img alt="P1" src="https://greptile-static-assets.s3.amazonaws.com/badges/p1.svg?v=9" align="top"></a> **Queues grow without bound** Each worker queue is created without a maximum size, while `submit()` and `poll()` never throttle ingestion. If Kafka delivery outpaces a worker—especially when a busy group routes most traffic to one queue—`WorkItem` objects accumulate without limit instead of applying the documented backpressure, which can exhaust process memory during sustained lag.
PR-level
<h3>Greptile Summary</h3> The PR introduces a `thread-queue-parallel` result-consumer mode with per-group ordered worker queues and asynchronous offset tracking and commits. - Adds the new mode to the uptime consumer CLI and strategy factory. - Adds queue workers, grouping, metrics, lifecycle handling, and offset tracking. - Adds unit and Kafka-backed tests for ordering, concurrency, commits, gaps, and shutdown. - The new path currently acknowledges failed work, does not drain or finalize commits safely during shutdown, and provides no bounded backpressure. <h3>Confidence Score: 2/5</h3> The PR is not safe to merge until failed work remains uncommitted, graceful shutdown drains and finalizes pending work, and worker queues apply bounded backpressure. Three blocking behaviors remain in the new processing mode: failed results become committable, shutdown can abandon queued work and stop commits prematurely, and sustained producer/worker imbalance can grow queues without limit. **Files Needing Attention:** src/sentry/remote_subscriptions/consumers/queue_consumer.py <details><summary><h3>Important Files Changed</h3></summary> | Filename | Overview | |----------|----------| | src/sentry/remote_subscriptions/consumers/queue_consumer.py | Implements the new ordered queue strategy, but its failure, backpressure, and shutdown semantics can lose, defer, or duplicate result processing. | | src/sentry/remote_subscriptions/consumers/result_consumer.py | Integrates the new mode into the result-consumer factory and adapts completed offsets to Kafka next-offset commits. | | src/sentry/consumers/__init__.py | Exposes the new processing mode through the uptime consumer CLI. | | tests/sentry/remote_subscriptions/consumers/test_queue_consumer.py | Covers ordering and offset tracking but does not verify draining pending work during shutdown or bounded backpressure. | | tests/sentry/uptime/consumers/test_results_consumer.py | Adds uptime and Kafka integration coverage, including an expectation that conflicts with the implementation's failed-offset behavior. | </details> <details open><summary><h3>Flowchart</h3></summary> ```mermaid %%{init: {'theme': 'neutral'}}%% flowchart LR K[Kafka message] --> D[Decode and derive group key] D --> H[Hash group key] H --> Q[Unbounded per-worker FIFO queue] Q --> W[Single ordered worker] W --> O[Mark offset complete] O --> C[Background commit loop] C --> K W -. processing failure is also completed .-> O S[Shutdown] --> X[Stop commit thread] X --> Y[Set worker stop flags] Y -. pending queue entries remain .-> Q ``` </details> <!-- greptile_other_comments_section --> <sub>Reviews (1): Last reviewed commit: ["Update src/sentry/consumers/\_\_init\_\_.py ..."](https://github.com/drape-io/crbench-greptile-2026-09/commit/7787e154960705e951f38e7635d5c5864ac2e699) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=64289043)</sub>
Mjolnir (Opus 5.5) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d4 high ✓ off by one
d5 high ✓ exact
valid · matches d5 · impact high
valid · impact high
The commit callback is called from a daemon thread. The OffsetTracker belongs to the factory-level pool and is never cleared for revoked partitions, so a commit that fails is retried every second and mark_committed never runs. [grounded: tree src/sentry/remote_subscriptions/consumers/queue_consumer.py] [consensus 3/3: valid=3]
valid · impact medium
The queues have no maxsize and submit never raises MessageRejected, so nothing applies backpressure even though the docstring says it does. [grounded: tree src/sentry/remote_subscriptions/consumers/queue_consumer.py] [consensus 3/3: valid=3]
valid · impact medium
The contiguity check walks the numeric range and stops at the first offset that was never received. Real Kafka gaps (transaction markers, compaction) would therefore block commits for that partition permanently. [grounded: tree src/sentry/remote_subscriptions/consumers/queue_consumer.py] [consensus 3/4: valid=3, invalid=1]
invalid
When shutdown_event is set during wait(1.0), the wait returns immediately and the loop still runs get_committable_offsets and the commit before checking the event again. So there is a final commit pass, and the claimed loss of a second of completed results does not happen. [invalid: contradiction] [consensus 3/3: invalid=3]
valid · impact critical
The strategy's close() shuts down the pool owned by the factory. After a rebalance, the next strategy gets the same pool, and put() raises queue.ShutDown. The broad except catches it and marks the offset complete, so messages are dropped while their offsets are committed. [grounded: tree src/sentry/remote_subscriptions/consumers/queue_consumer.py] [consensus 3/3: valid=3]
valid · impact high
Arroyo's Commit callable is called from a separate thread while the main thread may be inside poll or handling a revocation, so the two race on the consumer's commit state. [grounded: tree src/sentry/remote_subscriptions/consumers/queue_consumer.py] [consensus 3/3: valid=3]
valid · matches d4 · impact high
raw comments (8)
src/sentry/remote_subscriptions/consumers/queue_consumer.py:334
[P1] Rebalance shuts down the shared queue pool, so later messages are dropped and committed without being processed One `FixedQueuePool` is created in `ResultsStrategyFactory.__init__` and reused by every strategy that `create_with_partitions` returns. On each partition revocation, Arroyo calls `close()`/`join()` on the current strategy, and `close()` calls `self.queue_pool.shutdown()`. That stops every worker thread and shuts down every queue. After reassignment, the new `SimpleQueueProcessingStrategy` gets the same dead pool. `work_queue.put` then raises `queue.ShutDown`, and the broad `except` in `submit` logs it and marks the offset complete. From the first rebalance on (e.g. any deploy or scale event), the consumer silently skips every uptime result while still committing its offsets. The strategy should not own the pool's lifecycle; that belongs in the factory's `shutdown()`. Alternatively, create a new pool for each strategy
src/sentry/remote_subscriptions/consumers/queue_consumer.py:276
[P1] Offsets are committed from a background thread, including offsets for revoked partitions _commit_loop` runs on its own thread and calls Arroyo's `commit` callback. That callback stages and commits offsets on the consumer, which is not thread-safe and normally runs only on the main thread during `poll`/`submit`, so the two threads can race during polls and rebalances. The `OffsetTracker` is also shared across strategies and is never cleared on revocation. Workers that finish an item from a revoked partition make that partition committable again in the next strategy's commit loop. Arroyo then raises for the unassigned partition, the exception is swallowed, and `mark_committed` is skipped. The same failing dict is retried every second, so commits stall for all partitions. Commits should happen in `poll()` on the main thread, and tracker state for revoked partitions should be dropped
src/sentry/remote_subscriptions/consumers/queue_consumer.py:186
[P2] Unbounded queues give no backpressure Each worker queue is created as `queue.Queue()` with no `maxsize`, and `submit` never raises `MessageRejected`. When processing falls behind (slow DB, a hot group key), Arroyo keeps consuming and enqueuing without limit, and memory grows until the process OOMs. The class docstring says "Natural backpressure when queues fill up", but this code never applies any
src/sentry/remote_subscriptions/consumers/queue_consumer.py:87
[P2] Any gap in offsets stops commits for that partition permanently get_committable_offsets` requires every offset from `last_committed + 1` upward to be present in `all_offsets`, and breaks at the first missing one. Kafka offsets are not guaranteed to be contiguous; transaction control markers and compaction both leave gaps. After one gap, nothing past it can be committed again, and `all_offsets` grows without bound. Contiguity should be checked against the offsets the consumer actually received, not against `range(start, max_offset + 1)
src/sentry/remote_subscriptions/consumers/queue_consumer.py:334
[P2] Shutdown makes no final commit, so finished work is reprocessed close()` sets `shutdown_event` and joins the commit thread. The thread exits without one last `get_committable_offsets()`/commit pass, and the queues are not drained first. Up to about one second of completed results (plus anything still queued) is never committed on every revocation or shutdown. Those results are reprocessed by the next owner of the partition, which can produce duplicate uptime state transitions
src/sentry/remote_subscriptions/consumers/queue_consumer.py:333
[P1] Rebalance shuts down the shared queue pool, so the next strategy drops every message `queue_pool` lives on the factory and is created once in `ResultsStrategyFactory.__init__`. `SimpleQueueProcessingStrategy.close()` (also called again by `join()`) calls `self.queue_pool.shutdown()`, which shuts down every queue and stops the worker threads. Arroyo closes and joins the current strategy on every partition revocation, then builds a new one with `create_with_partitions`. That new strategy gets the same pool, because `create_thread_queue_parallel_worker` only checks that `self.queue_pool` is not None. After the first rebalance (deploy, scale-up, consumer restart in the group), `work_queue.put` raises `queue.ShutDown`. The broad `except Exception` in `submit` catches it, logs it, and marks the offset complete, and the commit loop then commits it. Every later uptime result is silently dropped without being processed. The pool should only be shut down from the factory's `shutdown()`, or a new pool should be created for each strategy.
src/sentry/remote_subscriptions/consumers/queue_consumer.py:276
[P1] Offsets are committed from a background thread through Arroyo's non-thread-safe commit callback `_commit_loop` runs on a separate daemon thread and calls `self.commit_function(committable)`, which ends up in Arroyo's `Commit` callable. That callable changes the StreamProcessor's internal commit state and stages and commits offsets on the underlying KafkaConsumer. Arroyo expects it to be called only from the consumer's poll/run thread, from within `submit`/`poll`. Calling it at the same time as `poll()` is a data race on the consumer. During a revocation, the thread can also stage offsets for partitions that are no longer assigned, which makes the consumer raise or commit stale offsets. The commit thread is also left running until `close()`, so it can overlap the revocation callback. Committable offsets should be collected in the background but committed from `poll()` on the main thread.
src/sentry/remote_subscriptions/consumers/queue_consumer.py:182
[P2] Unbounded work queues give no backpressure Each queue is created as `queue.Queue()` with no `maxsize`, and `submit` never raises `MessageRejected`. The strategy therefore keeps accepting messages from Kafka however far behind the workers are. When processing is slower than ingest (for example, a slow DB or a backlog after downtime), memory grows without limit until the consumer is OOM-killed. The class docstring says it provides "Natural backpressure when queues fill up", but nothing does that: there is no bounded queue and no rejection when queues are full.
Mjolnir (Sonnet 5.5) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d4 high ✓ off by one
d5 high ✓ exact
valid · matches d5 · impact high
valid · impact medium
The contiguous-offset scan breaks at the first offset missing from all_offsets. Kafka offsets can have gaps (compaction, transaction markers), so commits stall at a gap. Tracker state is also never reset on revocation. [grounded: tree src/sentry/remote_subscriptions/consumers/queue_consumer.py] [consensus 3/3: valid=3]
valid · impact medium
The queues are unbounded and submit never raises MessageRejected, so nothing implements the backpressure the docstring promises and memory can grow without limit. [grounded: tree src/sentry/remote_subscriptions/consumers/queue_consumer.py] [consensus 3/3: valid=3]
valid · impact medium
The Arroyo commit callback is called from a daemon thread, while the StreamProcessor drives the same consumer and commit state from the main thread without synchronization. [grounded: tree src/sentry/remote_subscriptions/consumers/queue_consumer.py] [consensus 3/3: valid=3]
valid · impact high
Strategy close/terminate shuts down the pool, which the factory created once and shares across strategies. After a rebalance the new strategy reuses the dead pool: put raises queue.ShutDown, submit's except block catches it, and the offset is marked complete and committed. The message is dropped silently. [grounded: tree src/sentry/remote_subscriptions/consumers/queue_consumer.py] [consensus 3/3: valid=3]
valid · matches d4 · impact high
raw comments (6)
src/sentry/remote_subscriptions/consumers/queue_consumer.py:335
[P1] Strategy close() permanently shuts down the factory-owned queue pool `FixedQueuePool` is created once in `ResultsStrategyFactory.__init__` and shared by every strategy that `create_with_partitions` returns. `SimpleQueueProcessingStrategy.close()`, `terminate()` and `join()` call `queue_pool.shutdown()`. That sets each worker's `shutdown` flag and shuts down the queues, and nothing recreates them. Arroyo closes the strategy on every rebalance and then builds a new one. The new strategy reuses the dead pool, so `queue.put` raises `queue.ShutDown`. The broad `except Exception` in `submit` catches it, logs it, and marks the offset complete. The commit loop then commits those offsets, so after the first rebalance every message is silently dropped and its offset committed. The pool should live and die with the factory (`ResultsStrategyFactory.shutdown`), or be rebuilt per strategy.
src/sentry/remote_subscriptions/consumers/queue_consumer.py:85
[P2] Offset tracker stalls on offset gaps and keeps stale state across rebalances `get_committable_offsets` only advances while every integer offset in `[start, max]` is in `all_offsets`. A gap in a partition's offsets (compaction, transaction markers, any skipped offset) makes the loop break at the gap every time, so nothing past it is ever committed and the sets grow without bound. `last_committed` and `all_offsets` are also never cleared when partitions are revoked. If a partition is reassigned and replays from an offset at or below the stale `last_committed`, `start` points to an offset that is never added, so commits for that partition stop until the consumer passes the old high-water mark.
src/sentry/remote_subscriptions/consumers/queue_consumer.py:185
[P2] Unbounded queues give no backpressure, despite the docstring Each worker queue is an unbounded `queue.Queue()`, and `submit` never blocks, raises `MessageRejected`, or pauses. `poll` only emits a gauge. If processing is slower than ingestion, or one hot group's queue backs up, the consumer keeps pulling from Kafka and buffers every message in memory. The `SimpleQueueProcessingStrategy` docstring says "Natural backpressure when queues fill up", but nothing implements it. This risks unbounded memory growth and large replay windows on crash.
src/sentry/remote_subscriptions/consumers/queue_consumer.py:268
[P3] Offsets are committed from a background thread `commit_function` is Arroyo's `commit` callback, and `_commit_loop` calls it from a separate daemon thread. The Arroyo `StreamProcessor` and its Kafka consumer are driven from the main thread, and the commit callback stages and commits offsets on that same consumer and its commit-policy state. Calling it concurrently with `poll` and `submit`, or during a rebalance, is unsynchronized. Committing from `poll()` on the consumer thread would avoid the race.
src/sentry/remote_subscriptions/consumers/queue_consumer.py:335
[P1] Strategy close/terminate shuts down the factory-shared queue pool, so every consumer rebalance disables processing and drops messages `FixedQueuePool` is created once in `ResultsStrategyFactory.__init__` and passed to every `SimpleQueueProcessingStrategy` that `create_with_partitions` builds. Arroyo calls `close()`/`terminate()` on the strategy on every partition revoke or rebalance. Those methods call `self.queue_pool.shutdown()`, which sets the worker shutdown flags, calls `shutdown()` on every queue and joins the workers. The next strategy built after the rebalance reuses the same dead pool. Its `queue_pool.submit()` then calls `put()` on a shut-down queue and raises `queue.ShutDown`. `SimpleQueueProcessingStrategy.submit` catches that exception, logs it, and immediately calls `add_offset` and `complete_offset` for the message. The commit loop then commits those offsets, so after the first rebalance (deploy, scale event or broker rebalance) results are silently skipped and their offsets committed. The pool should be shut down only from the factory's `shutdown()`, not from the per-assignment strategy. Pool state such as the offset tracker should also be reset on revoke.
src/sentry/remote_subscriptions/consumers/queue_consumer.py:185
[P2] Queues are unbounded, so there is no backpressure despite the documented guarantee Each worker queue is created with `queue.Queue()` (no `maxsize`), and `SimpleQueueProcessingStrategy.submit` never raises `MessageRejected` or blocks. The class docstring claims "Natural backpressure when queues fill up", but that is not true. If the result processor is slower than ingestion, or a hot group key maps to one queue, the consumer keeps polling and enqueuing without limit. Memory grows, `all_offsets` and `outstanding` grow with it, and the queued backlog is lost on restart. Bound the queues and reject or block in `submit` when they are full.
qodo · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d4 high ✓ exact
d5 high ✓ exact
invalid
The result processor actually used here is a ResultProcessor instance. Its __call__ already catches and logs every exception, so a processing failure never reaches the worker. Serial mode works the same way: it logs failures and commits past them. Skipping failed results is the existing design, not a regression from this change. [invalid: contradiction] [consensus 3/3: invalid=3]
valid · matches d5 · impact high
valid · impact medium
FixedQueuePool.shutdown sets each worker's shutdown flag before shutting down the queues, so `while not self.shutdown` exits after the current item and leaves queued items unprocessed. close() also sets shutdown_event and joins the commit thread before shutting down the pool, so offsets completed during shutdown are never committed. [grounded: diff] [consensus 3/3: valid=3]
valid · matches d4 · impact high
valid · impact low
`max_workers or 20` turns 0 into 20. A negative value makes range() create no queues, so self.queues[...] raises IndexError. Submit's broad except then adds and completes the offset, so the message is committed without ever being processed. It needs a misconfiguration to trigger, but that value is reachable from the CLI. [grounded: diff] [consensus 3/3: valid=3]
raw comments (7)
src/sentry/remote_subscriptions/consumers/queue_consumer.py:149
<img src="https://img.shields.io/badge/High-634FD1?style=flat-square" height="20px" alt="Action required"> 1\. Failed processing is still committed <code>🐞 Bug</code> <code>≡ Correctness</code> <pre> <b><i>OrderedQueueWorker.run</i></b> calls <b><i>complete_offset</i></b> from <b><i>finally</i></b> even when <b><i>result_processor</i></b> raises an exception. The commit loop subsequently advances beyond that message, so any transient or permanent result-processing failure loses the result instead of leaving it available for retry. </pre> <details> <summary><strong>Agent Prompt</strong></summary> ``` ## Issue description Worker processing exceptions are logged but their offsets are still marked complete, allowing failed results to be committed and permanently skipped. ## Fix Focus Areas - src/sentry/remote_subscriptions/consumers/queue_consumer.py[135-149] - src/sentry/remote_subscriptions/consumers/queue_consumer.py[273-289] ## Recommended Fix Move `complete_offset` onto the successful processing path and preserve failed offsets as outstanding. Ensure failure handling stops or retries the affected group without allowing later offsets to be committed past the failed result. ``` <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> </details>
src/sentry/remote_subscriptions/consumers/result_consumer.py:136
<img src="https://img.shields.io/badge/High-634FD1?style=flat-square" height="20px" alt="Action required"> 2\. Rebalances discard later results <code>🐞 Bug</code> <code>☼ Reliability</code> <pre> <b><i>ResultsStrategyFactory</i></b> creates one <b><i>FixedQueuePool</i></b> and passes that same pool to every strategy, while each strategy&#x27;s <b><i>close</i></b> permanently shuts its workers down. After the first assignment change closes the old strategy, later submissions reach stopped queues and the broad submission handler marks their offsets complete, allowing unprocessed results to be committed. </pre> <details> <summary><strong>Agent Prompt</strong></summary> ``` ## Issue description A factory-owned queue pool is reused across processing-strategy generations even though closing one strategy permanently shuts that pool down, causing later assignments to lose or stall results. ## Fix Focus Areas - src/sentry/remote_subscriptions/consumers/result_consumer.py[131-137] - src/sentry/remote_subscriptions/consumers/result_consumer.py[244-259] - src/sentry/remote_subscriptions/consumers/queue_consumer.py[335-338] ## Recommended Fix Give each strategy generation its own queue pool, workers, offset tracker, and commit callback. Make that strategy solely responsible for draining and closing its pool so assignment generations cannot share stopped workers or offset state. ``` <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> </details>
src/sentry/remote_subscriptions/consumers/queue_consumer.py:234
<img src="https://img.shields.io/badge/High-634FD1?style=flat-square" height="20px" alt="Action required"> 3\. Shutdown abandons queued results <code>🐞 Bug</code> <code>☼ Reliability</code> <pre> <b><i>FixedQueuePool.shutdown</i></b> sets every worker&#x27;s loop flag before asking the queues to shut down, so each worker exits after at most its current item rather than draining pending entries. <b><i>SimpleQueueProcessingStrategy.close</i></b> also stops its only commit loop before shutting down the workers, leaving work completed during closure without a final commit and forcing replay after ordinary shutdowns. </pre> <details> <summary><strong>Agent Prompt</strong></summary> ``` ## Issue description The shutdown sequence stops workers and the commit loop before queued and in-flight work has drained, abandoning pending results and missing final offset commits. ## Fix Focus Areas - src/sentry/remote_subscriptions/consumers/queue_consumer.py[127-149] - src/sentry/remote_subscriptions/consumers/queue_consumer.py[231-243] - src/sentry/remote_subscriptions/consumers/queue_consumer.py[335-345] ## Recommended Fix Stop accepting new submissions, let workers drain all queued entries, wait for in-flight processing, perform a final synchronous commit of completed offsets, and only then stop and join workers and the commit mechanism. Honor the timeout supplied to `join` throughout this sequence. ``` <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> </details>
src/sentry/remote_subscriptions/consumers/queue_consumer.py:186
<img src="https://img.shields.io/badge/High-634FD1?style=flat-square" height="20px" alt="Action required"> 4\. Worker backlogs can exhaust memory <code>🐞 Bug</code> <code>☼ Reliability</code> <pre> <b><i>FixedQueuePool</i></b> constructs each worker queue with the default unlimited capacity and <b><i>submit</i></b> always appends without applying backpressure. Whenever Kafka ingestion remains faster than the fixed workers, decoded messages and their original payloads accumulate without a bound until the consumer exhausts memory. </pre> <details> <summary><strong>Agent Prompt</strong></summary> ``` ## Issue description The fixed worker pool uses unbounded queues, allowing ingestion to accumulate an unlimited in-memory backlog when processing falls behind. ## Fix Focus Areas - src/sentry/remote_subscriptions/consumers/queue_consumer.py[171-186] - src/sentry/remote_subscriptions/consumers/queue_consumer.py[204-212] - src/sentry/remote_subscriptions/consumers/queue_consumer.py[327-333] ## Recommended Fix Configure a bounded capacity for every work queue and make strategy polling or submission pause consumption when capacity is exhausted. Ensure shutdown and error handling distinguish intentional backpressure from a failed submission and never acknowledge work that was not queued. ``` <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> </details>
src/sentry/remote_subscriptions/consumers/result_consumer.py:136
<img src="https://img.shields.io/badge/Medium-634FD1?style=flat-square" height="20px" alt="Remediation recommended"> 5\. Invalid worker counts discard results <code>🐞 Bug</code> <code>≡ Correctness</code> <pre> The new mode passes <b><i>max_workers</i></b> directly as the number of queues without validating that it is positive, while zero is silently replaced by twenty through <b><i>or</i></b>. A negative command-line value creates no queues, causing every group lookup to fail and the broad submission handler to mark each unprocessed message complete. </pre> <details> <summary><strong>Agent Prompt</strong></summary> ``` ## Issue description The thread-queue mode accepts non-positive worker counts, which either silently changes the requested value or creates a pool with no workers and acknowledges every failed submission. ## Fix Focus Areas - src/sentry/consumers/__init__.py[138-143] - src/sentry/remote_subscriptions/consumers/result_consumer.py[131-137] - src/sentry/remote_subscriptions/consumers/queue_consumer.py[171-202] ## Recommended Fix Validate at configuration or factory construction that an explicitly supplied worker count is greater than zero. Use the default only when the value is `None`, and fail startup with a clear configuration error for zero or negative values. ``` <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> </details>
PR-level
<h3>PR Summary by Qodo</h3> Add ordered thread-queue parallelism to remote subscription consumers <code>✨ Enhancement</code> <code>🧪 Tests</code> <code>🕐 40+ Minutes</code> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <details> <summary>AI Description</summary> <dl> <dd> <br/> ><pre> >• Adds grouped thread-queue parallel processing while preserving per-subscription result order. >• Tracks contiguous Kafka offsets for asynchronous commits across concurrent workers. >• Covers queue behavior, uptime integration, shutdown, failures, and real Kafka commits. ></pre> </dd> </dl> </details> <details> <summary>Diagram</summary> <dl> <dd> <br/> ```mermaid sequenceDiagram participant K as Kafka Consumer participant F as Strategy Factory participant S as Queue Strategy participant Q as Fixed Queue Pool participant W as Queue Worker participant R as Result Processor participant O as Offset Tracker F->>S: Create queue mode K->>S: Submit payload S->>Q: Enqueue by group Q->>W: FIFO dispatch W->>R: Process result W->>O: Mark complete S->>O: Read safe offsets O-->>S: Contiguous offsets S->>K: Commit next offsets ``` </dd> </dl> </details> <details> <summary>High-Level Assessment</summary> <dl> <dd> <br/> >The following are alternative approaches to this PR: <details> <summary><b>1. Keyed Kafka partitioning</b></summary> <dl> <dd> - ➕ Uses Kafka partition ordering instead of custom in-process ordering. - ➕ Scales concurrency across consumer instances. - ➕ Keeps offset ownership aligned with Kafka partitions. - ➖ Requires producers to key messages by subscription. - ➖ May require topic migration or repartitioning. - ➖ Hot subscriptions can create partition imbalance. </dd> </dl> </details> <details> <summary><b>2. Arroyo-native keyed strategy</b></summary> <dl> <dd> - ➕ Could reuse framework lifecycle, backpressure, and rebalance handling. - ➕ Reduces custom thread and offset-management code. - ➖ No existing repository strategy provides the required keyed FIFO behavior directly. - ➖ May require upstream Arroyo changes and delay deployment. </dd> </dl> </details> >**Recommendation:** The fixed keyed queue pool is a reasonable incremental approach because it adds per-subscription ordering without changing producers or Kafka topology. Prefer keyed Kafka partitioning for a longer-term distributed scaling model; for this implementation, reviewers should focus on failure semantics, rebalance behavior, and ensuring shutdown drains work before offsets advance. </dd> </dl> </details> <details> <summary> Files changed (7) <code> +1276 / -6 </code> </summary> <dl> <dd> <br/> <details> <summary>Enhancement (3) <code> +388 / -5 </code></summary> <dl> <dd> <details> <summary>__init__.py<code>Expose thread-queue parallel uptime consumer mode</code> <code>+2/-2</code></summary> <br/> >Expose thread-queue parallel uptime consumer mode > ><pre> >• Adds &#x27;thread-queue-parallel&#x27; to the uptime consumer mode choices. Generalizes the worker-limit help text to cover all parallel modes. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/73/files#diff-e5ee943f84dc1d6870ab01db646066449461e8f35ebdfab60048a4b3b6f73c59'>src/sentry/consumers/__init__.py</a> </details> <details> <summary>queue_consumer.py<code>Implement grouped FIFO queue processing strategy</code> <code>+345/-0</code></summary> <br/> >Implement grouped FIFO queue processing strategy > ><pre> >• Introduces keyed fixed queues, dedicated worker threads, and partition-aware contiguous offset tracking. Adds periodic asynchronous commits, processing metrics, tracing, and worker-pool shutdown handling. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/73/files#diff-833fb1d1f07cdf4208473ed234b716ee7cd0d13feb98e79513c0291336b104e4'>src/sentry/remote_subscriptions/consumers/queue_consumer.py</a> </details> <details> <summary>result_consumer.py<code>Integrate queue parallelism into result strategy factory</code> <code>+41/-3</code></summary> <br/> >Integrate queue parallelism into result strategy factory > ><pre> >• Adds the &#x27;thread-queue-parallel&#x27; factory mode and initializes a queue pool using the configured worker count. Connects payload decoding and grouping to the queue strategy, translates completed offsets into Kafka next-read offsets, and shuts down the pool with the factory. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/73/files#diff-d25c9b2410f097a594b1d7c91c401657675064c56d9fea09f0cc6fa5a62ac971'>src/sentry/remote_subscriptions/consumers/result_consumer.py</a> </details> </dd> </dl> </details> <details> <summary>Tests (4) <code> +888 / -1 </code></summary> <dl> <dd> <details> <summary>__init__.py<code>Add remote subscriptions test package marker</code> <code>+0/-0</code></summary> <br/> >Add remote subscriptions test package marker > ><pre> >• Adds the package initializer for remote-subscription tests. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/73/files#diff-f5795212127f7d522382d2701c104b76ff9af80ed289d92e449904ed35a012a4'>tests/sentry/remote_subscriptions/__init__.py</a> </details> <details> <summary>__init__.py<code>Add consumer test package marker</code> <code>+0/-0</code></summary> <br/> >Add consumer test package marker > ><pre> >• Adds the package initializer for remote-subscription consumer tests. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/73/files#diff-eb27ca22a86bb0cbdb14480eb6c333e98e8575bf299e54a1390b4303a3cce3a8'>tests/sentry/remote_subscriptions/consumers/__init__.py</a> </details> <details> <summary>test_queue_consumer.py<code>Test queue workers and contiguous offset tracking</code> <code>+421/-0</code></summary> <br/> >Test queue workers and contiguous offset tracking > ><pre> >• Covers offset tracking across partitions, deterministic group assignment, FIFO ordering, concurrent group processing, queue metrics, invalid payloads, offset gaps, and factory integration. Exercises asynchronous commits and queue-pool lifecycle behavior. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/73/files#diff-19b1139f3cfc3baf038b442c0c6ae9b86459588c03e3b207803fa570367777a0'>tests/sentry/remote_subscriptions/consumers/test_queue_consumer.py</a> </details> <details> <summary>test_results_consumer.py<code>Validate queue mode through uptime and Kafka integrations</code> <code>+467/-1</code></summary> <br/> >Validate queue mode through uptime and Kafka integrations > ><pre> >• Adds uptime consumer coverage for ordered and concurrent processing, commits, offset gaps, failures, and graceful shutdown. Adds a Kafka-backed integration test that verifies the consumer group receives the expected committed offset. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/73/files#diff-20565d043b15f7a19e0d98baeab19413e00bd1bda5222e78fd13eba35b295af8'>tests/sentry/uptime/consumers/test_results_consumer.py</a> </details> </dd> </dl> </details> </dd> </dl> </details>
PR-level
<h3>Code Review by Qodo</h3> <code>🐞 Bugs (5)</code> <code>📘 Rule violations (0)</code> <code>📜 Skill insights (0)</code> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <br/> <img src="https://img.shields.io/badge/High-634FD1?style=flat-square" height="20px" alt="Action required"> <details> <summary> 1. Failed processing is still committed <code>🐞 Bug</code> <code>≡ Correctness</code></summary> <br/> > <details open> ><summary>Description</summary> ><br/> > ><pre> ><b><i>OrderedQueueWorker.run</i></b> calls <b><i>complete_offset</i></b> from <b><i>finally</i></b> even when <b><i>result_processor</i></b> raises >an exception. The commit loop subsequently advances beyond that message, so any transient or >permanent result-processing failure loses the result instead of leaving it available for retry. ></pre> ></details> > <details> ><summary>Code</summary> ><br/> > ><code>[src/sentry/remote_subscriptions/consumers/queue_consumer.py[R148-149]](https://github.com/drape-io/crbench-qodo-2026-09/pull/73/files#diff-833fb1d1f07cdf4208473ed234b716ee7cd0d13feb98e79513c0291336b104e4R148-R149)</code> > >```diff >+ finally: >+ self.offset_tracker.complete_offset(work_item.partition, work_item.offset) >``` ></details> > <details> ><summary>Evidence</summary> ><br/> > ><pre> >The worker catches processor exceptions at lines 144-147 but unconditionally completes the same >offset at lines 148-149. The commit loop treats completed offsets as committable and the production >callback advances Kafka to the following offset. ></pre> > > <code>[src/sentry/remote_subscriptions/consumers/queue_consumer.py[135-149]](https://github.com/drape-io/crbench-qodo-2026-09/blob/44487111137dba118bc8e253c7c1da2472962244/src/sentry/remote_subscriptions/consumers/queue_consumer.py/#L135-L149)</code> > <code>[src/sentry/remote_subscriptions/consumers/queue_consumer.py[273-289]](https://github.com/drape-io/crbench-qodo-2026-09/blob/44487111137dba118bc8e253c7c1da2472962244/src/sentry/remote_subscriptions/consumers/queue_consumer.py/#L273-L289)</code> > <code>[src/sentry/remote_subscriptions/consumers/result_consumer.py[249-252]](https://github.com/drape-io/crbench-qodo-2026-09/blob/44487111137dba118bc8e253c7c1da2472962244/src/sentry/remote_subscriptions/consumers/result_consumer.py/#L249-L252)</code> ></details> > <details> ><summary>Agent prompt</summary> ><br/> > >``` >The issue below was found during a code review. Follow the provided context and guidance below and implement a solution > >## Issue description >Worker processing exceptions are logged but their offsets are still marked complete, allowing failed results to be committed and permanently skipped. > >## Fix Focus Areas >- src/sentry/remote_subscriptions/consumers/queue_consumer.py[135-149] >- src/sentry/remote_subscriptions/consumers/queue_consumer.py[273-289] > >## Recommended Fix >Move `complete_offset` onto the successful processing path and preserve failed offsets as outstanding. Ensure failure handling stops or retries the affected group without allowing later offsets to be committed past the failed result. >``` > <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> ></details> <hr/> </details> <details> <summary> 2. Rebalances discard later results <code>🐞 Bug</code> <code>☼ Reliability</code></summary> <br/> > <details open> ><summary>Description</summary> ><br/> > ><pre> ><b><i>ResultsStrategyFactory</i></b> creates one <b><i>FixedQueuePool</i></b> and passes that same pool to every strategy, >while each strategy&#x27;s <b><i>close</i></b> permanently shuts its workers down. After the first assignment change >closes the old strategy, later submissions reach stopped queues and the broad submission handler >marks their offsets complete, allowing unprocessed results to be committed. ></pre> ></details> > <details> ><summary>Code</summary> ><br/> > ><code>[src/sentry/remote_subscriptions/consumers/result_consumer.py[R133-136]](https://github.com/drape-io/crbench-qodo-2026-09/pull/73/files#diff-d25c9b2410f097a594b1d7c91c401657675064c56d9fea09f0cc6fa5a62ac971R133-R136)</code> > >```diff >+ self.queue_pool = FixedQueuePool( >+ result_processor=self.result_processor, >+ identifier=self.identifier, >+ num_queues=max_workers or 20, # Number of parallel queues >``` ></details> > <details> ><summary>Evidence</summary> ><br/> > ><pre> >The factory constructs only one pool and repeatedly passes it through ><b><i>create_thread_queue_parallel_worker</i></b>. Strategy closure shuts that shared pool permanently, while ><b><i>submit</i></b> catches subsequent queue failures and records those offsets as completed. ></pre> > > <code>[src/sentry/remote_subscriptions/consumers/result_consumer.py[131-137]](https://github.com/drape-io/crbench-qodo-2026-09/blob/44487111137dba118bc8e253c7c1da2472962244/src/sentry/remote_subscriptions/consumers/result_consumer.py/#L131-L137)</code> > <code>[src/sentry/remote_subscriptions/consumers/result_consumer.py[200-212]](https://github.com/drape-io/crbench-qodo-2026-09/blob/44487111137dba118bc8e253c7c1da2472962244/src/sentry/remote_subscriptions/consumers/result_consumer.py/#L200-L212)</code> > <code>[src/sentry/remote_subscriptions/consumers/result_consumer.py[244-259]](https://github.com/drape-io/crbench-qodo-2026-09/blob/44487111137dba118bc8e253c7c1da2472962244/src/sentry/remote_subscriptions/consumers/result_consumer.py/#L244-L259)</code> > <code>[src/sentry/remote_subscriptions/consumers/queue_consumer.py[231-243]](https://github.com/drape-io/crbench-qodo-2026-09/blob/44487111137dba118bc8e253c7c1da2472962244/src/sentry/remote_subscriptions/consumers/queue_consumer.py/#L231-L243)</code> > <code>[src/sentry/remote_subscriptions/consumers/queue_consumer.py[317-325]](https://github.com/drape-io/crbench-qodo-2026-09/blob/44487111137dba118bc8e253c7c1da2472962244/src/sentry/remote_subscriptions/consumers/queue_consumer.py/#L317-L325)</code> > <code>[src/sentry/remote_subscriptions/consumers/queue_consumer.py[335-338]](https://github.com/drape-io/crbench-qodo-2026-09/blob/44487111137dba118bc8e253c7c1da2472962244/src/sentry/remote_subscriptions/consumers/queue_consumer.py/#L335-L338)</code> ></details> > <details> ><summary>Agent prompt</summary> ><br/> > >``` >The issue below was found during a code review. Follow the provided context and guidance below and implement a solution > >## Issue description >A factory-owned queue pool is reused across processing-strategy generations even though closing one strategy permanently shuts that pool down, causing later assignments to lose or stall results. > >## Fix Focus Areas >- src/sentry/remote_subscriptions/consumers/result_consumer.py[131-137] >- src/sentry/remote_subscriptions/consumers/result_consumer.py[244-259] >- src/sentry/remote_subscriptions/consumers/queue_consumer.py[335-338] > >## Recommended Fix >Give each strategy generation its own queue pool, workers, offset tracker, and commit callback. Make that strategy solely responsible for draining and closing its pool so assignment generations cannot share stopped workers or offset state. >``` > <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> ></details> <hr/> </details> <details> <summary> 3. Shutdown abandons queued results <code>🐞 Bug</code> <code>☼ Reliability</code></summary> <br/> > <details open> ><summary>Description</summary> ><br/> > ><pre> ><b><i>FixedQueuePool.shutdown</i></b> sets every worker&#x27;s loop flag before asking the queues to shut down, so >each worker exits after at most its current item rather than draining pending entries. ><b><i>SimpleQueueProcessingStrategy.close</i></b> also stops its only commit loop before shutting down the >workers, leaving work completed during closure without a final commit and forcing replay after >ordinary shutdowns. ></pre> ></details> > <details> ><summary>Code</summary> ><br/> > ><code>[src/sentry/remote_subscriptions/consumers/queue_consumer.py[R231-234]](https://github.com/drape-io/crbench-qodo-2026-09/pull/73/files#diff-833fb1d1f07cdf4208473ed234b716ee7cd0d13feb98e79513c0291336b104e4R231-R234)</code> > >```diff >+ def shutdown(self) -> None: >+ """Gracefully shutdown all workers.""" >+ for worker in self.workers: >+ worker.shutdown = True >``` ></details> > <details> ><summary>Evidence</summary> ><br/> > ><pre> >Workers continue only while their shutdown flag is false, but shutdown sets that flag before >draining the queues. Close first ends the commit thread and only afterward invokes this non-draining >worker shutdown, while <b><i>join</i></b> ignores its supplied timeout and repeats the same sequence. ></pre> > > <code>[src/sentry/remote_subscriptions/consumers/queue_consumer.py[127-149]](https://github.com/drape-io/crbench-qodo-2026-09/blob/44487111137dba118bc8e253c7c1da2472962244/src/sentry/remote_subscriptions/consumers/queue_consumer.py/#L127-L149)</code> > <code>[src/sentry/remote_subscriptions/consumers/queue_consumer.py[231-243]](https://github.com/drape-io/crbench-qodo-2026-09/blob/44487111137dba118bc8e253c7c1da2472962244/src/sentry/remote_subscriptions/consumers/queue_consumer.py/#L231-L243)</code> > <code>[src/sentry/remote_subscriptions/consumers/queue_consumer.py[273-289]](https://github.com/drape-io/crbench-qodo-2026-09/blob/44487111137dba118bc8e253c7c1da2472962244/src/sentry/remote_subscriptions/consumers/queue_consumer.py/#L273-L289)</code> > <code>[src/sentry/remote_subscriptions/consumers/queue_consumer.py[335-345]](https://github.com/drape-io/crbench-qodo-2026-09/blob/44487111137dba118bc8e253c7c1da2472962244/src/sentry/remote_subscriptions/consumers/queue_consumer.py/#L335-L345)</code> ></details> > <details> ><summary>Agent prompt</summary> ><br/> > >``` >The issue below was found during a code review. Follow the provided context and guidance below and implement a solution > >## Issue description >The shutdown sequence stops workers and the commit loop before queued and in-flight work has drained, abandoning pending results and missing final offset commits. > >## Fix Focus Areas >- src/sentry/remote_subscriptions/consumers/queue_consumer.py[127-149] >- src/sentry/remote_subscriptions/consumers/queue_consumer.py[231-243] >- src/sentry/remote_subscriptions/consumers/queue_consumer.py[335-345] > >## Recommended Fix >Stop accepting new submissions, let workers drain all queued entries, wait for in-flight processing, perform a final synchronous commit of completed offsets, and only then stop and join workers and the commit mechanism. Honor the timeout supplied to `join` throughout this sequence. >``` > <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> ></details> <hr/> </details> <details><summary><ins><strong>View high (1)</strong></ins></summary><br/> <details> <summary> 4. Worker backlogs can exhaust memory <code>🐞 Bug</code> <code>☼ Reliability</code></summary> <br/> > <details open> ><summary>Description</summary> ><br/> > ><pre> ><b><i>FixedQueuePool</i></b> constructs each worker queue with the default unlimited capacity and <b><i>submit</i></b> >always appends without applying backpressure. Whenever Kafka ingestion remains faster than the fixed >workers, decoded messages and their original payloads accumulate without a bound until the consumer >exhausts memory. ></pre> ></details> > <details> ><summary>Code</summary> ><br/> > ><code>[src/sentry/remote_subscriptions/consumers/queue_consumer.py[R184-186]](https://github.com/drape-io/crbench-qodo-2026-09/pull/73/files#diff-833fb1d1f07cdf4208473ed234b716ee7cd0d13feb98e79513c0291336b104e4R184-R186)</code> > >```diff >+ for i in range(num_queues): >+ work_queue: queue.Queue[WorkItem[T]] = queue.Queue() >+ self.queues.append(work_queue) >``` ></details> > <details> ><summary>Evidence</summary> ><br/> > ><pre> >Every queue is instantiated without <b><i>maxsize</i></b>, and submissions use unconditional <b><i>put</i></b> with no >consumer-level capacity check. The strategy&#x27;s <b><i>poll</i></b> merely emits queue-depth metrics, so it does >not slow ingestion as backlog grows. ></pre> > > <code>[src/sentry/remote_subscriptions/consumers/queue_consumer.py[181-186]](https://github.com/drape-io/crbench-qodo-2026-09/blob/44487111137dba118bc8e253c7c1da2472962244/src/sentry/remote_subscriptions/consumers/queue_consumer.py/#L181-L186)</code> > <code>[src/sentry/remote_subscriptions/consumers/queue_consumer.py[204-212]](https://github.com/drape-io/crbench-qodo-2026-09/blob/44487111137dba118bc8e253c7c1da2472962244/src/sentry/remote_subscriptions/consumers/queue_consumer.py/#L204-L212)</code> > <code>[src/sentry/remote_subscriptions/consumers/queue_consumer.py[293-315]](https://github.com/drape-io/crbench-qodo-2026-09/blob/44487111137dba118bc8e253c7c1da2472962244/src/sentry/remote_subscriptions/consumers/queue_consumer.py/#L293-L315)</code> > <code>[src/sentry/remote_subscriptions/consumers/queue_consumer.py[327-333]](https://github.com/drape-io/crbench-qodo-2026-09/blob/44487111137dba118bc8e253c7c1da2472962244/src/sentry/remote_subscriptions/consumers/queue_consumer.py/#L327-L333)</code> ></details> > <details> ><summary>Agent prompt</summary> ><br/> > >``` >The issue below was found during a code review. Follow the provided context and guidance below and implement a solution > >## Issue description >The fixed worker pool uses unbounded queues, allowing ingestion to accumulate an unlimited in-memory backlog when processing falls behind. > >## Fix Focus Areas >- src/sentry/remote_subscriptions/consumers/queue_consumer.py[171-186] >- src/sentry/remote_subscriptions/consumers/queue_consumer.py[204-212] >- src/sentry/remote_subscriptions/consumers/queue_consumer.py[327-333] > >## Recommended Fix >Configure a bounded capacity for every work queue and make strategy polling or submission pause consumption when capacity is exhausted. Ensure shutdown and error handling distinguish intentional backpressure from a failed submission and never acknowledge work that was not queued. >``` > <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> ></details> <hr/> </details> </details> <br/> <img src="https://img.shields.io/badge/Medium-634FD1?style=flat-square" height="20px" alt="Remediation recommended"> <details> <summary> 5. Invalid worker counts discard results <code>🐞 Bug</code> <code>≡ Correctness</code></summary> <br/> > <details open> ><summary>Description</summary> ><br/> > ><pre> >The new mode passes <b><i>max_workers</i></b> directly as the number of queues without validating that it is >positive, while zero is silently replaced by twenty through <b><i>or</i></b>. A negative command-line value >creates no queues, causing every group lookup to fail and the broad submission handler to mark each >unprocessed message complete. ></pre> ></details> > <details> ><summary>Code</summary> ><br/> > ><code>[src/sentry/remote_subscriptions/consumers/result_consumer.py[R133-136]](https://github.com/drape-io/crbench-qodo-2026-09/pull/73/files#diff-d25c9b2410f097a594b1d7c91c401657675064c56d9fea09f0cc6fa5a62ac971R133-R136)</code> > >```diff >+ self.queue_pool = FixedQueuePool( >+ result_processor=self.result_processor, >+ identifier=self.identifier, >+ num_queues=max_workers or 20, # Number of parallel queues >``` ></details> > <details> ><summary>Evidence</summary> ><br/> > ><pre> >The CLI accepts any integer, the factory maps zero to twenty and forwards negative values unchanged, >and <b><i>FixedQueuePool</i></b> creates workers with <b><i>range(num_queues)</i></b>. With a negative count the empty queue >list makes group submission fail, after which the strategy completes the failed message&#x27;s offset. ></pre> > > <code>[src/sentry/consumers/__init__.py[138-143]](https://github.com/drape-io/crbench-qodo-2026-09/blob/44487111137dba118bc8e253c7c1da2472962244/src/sentry/consumers/__init__.py/#L138-L143)</code> > <code>[src/sentry/remote_subscriptions/consumers/result_consumer.py[131-137]](https://github.com/drape-io/crbench-qodo-2026-09/blob/44487111137dba118bc8e253c7c1da2472962244/src/sentry/remote_subscriptions/consumers/result_consumer.py/#L131-L137)</code> > <code>[src/sentry/remote_subscriptions/consumers/queue_consumer.py[171-202]](https://github.com/drape-io/crbench-qodo-2026-09/blob/44487111137dba118bc8e253c7c1da2472962244/src/sentry/remote_subscriptions/consumers/queue_consumer.py/#L171-L202)</code> > <code>[src/sentry/remote_subscriptions/consumers/queue_consumer.py[306-325]](https://github.com/drape-io/crbench-qodo-2026-09/blob/44487111137dba118bc8e253c7c1da2472962244/src/sentry/remote_subscriptions/consumers/queue_consumer.py/#L306-L325)</code> ></details> > <details> ><summary>Agent prompt</summary> ><br/> > >``` >The issue below was found during a code review. Follow the provided context and guidance below and implement a solution > >## Issue description >The thread-queue mode accepts non-positive worker counts, which either silently changes the requested value or creates a pool with no workers and acknowledges every failed submission. > >## Fix Focus Areas >- src/sentry/consumers/__init__.py[138-143] >- src/sentry/remote_subscriptions/consumers/result_consumer.py[131-137] >- src/sentry/remote_subscriptions/consumers/queue_consumer.py[171-202] > >## Recommended Fix >Validate at configuration or factory construction that an explicitly supplied worker count is greater than zero. Use the default only when the value is `None`, and fail startup with a clear configuration error for zero or negative values. >``` > <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> ></details> <hr/> </details> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <!-- qodo-context:start --> <details><summary><strong>Context sources</strong></summary> <div>Review mode: <code>⚖️ Balanced</code>: Downgraded extended -&gt; standard: change is below the extended eligibility bar (hunks 17/18, lines 1282/200; both must reach the floor). Router rationale: This introduces substantial concurrent queueing, ordering, offset-commit, shutdown, and consumer-integration logic across multiple independent paths, creating a high density of subtle behavioral defects.</div> <!-- qodo-context:end --> </details> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <!-- qodo-daily-tip:start --> <details> <summary><strong>Tip of the day</strong></summary> <br/> <pre>💡 Did you know, you can reply &#x27;qodo&#x27; on any finding to push back, ask questions, or dig deeper</pre> <a href="https://docs.qodo.ai/tips-and-tricks">More tips ↗</a> | <a href="https://app.qodo.ai/configurations?tab=display-preferences">Customize Qodo ↗</a> | <a href="https://docs.qodo.ai">Qodo docs ↗</a> </details> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <!-- qodo-daily-tip:end --> <!-- https://github.com/drape-io/crbench-qodo-2026-09/commit/44487111137dba118bc8e253c7c1da2472962244 --> <a href="https://www.qodo.ai"><img src="https://www.qodo.ai/wp-content/uploads/2025/03/qodo-logo.svg" width="80" alt="Qodo Logo"></a>