feat(sig): custom agents
Tier 2 medium python · cross-file · real merged PR from PostHog/posthog #58847
Ground-truth defects
In the new signals auto-start path (_resolve_autostart_assignee in products/signals/backend/auto_start.py), candidate users are filtered with User.objects.filter(..., teams__id=team_id). User.teams is a @cached_property on the User model, not an ORM relation, so this lookup raises Django FieldError ("Cannot resolve keyword 'teams' into field") every time the path reaches the query. The exception is swallowed upstream, so report generation keeps working but the auto-start path silently creates zero implementation tasks. It is cross-file because the teams cached_property lives in the user model file while the broken filter is in the changed auto_start.py code.
The change under review
diff (truncated — full diff)
new file mode 100644 @@ -0,0 +1,117 @@ +# GitHub integration setup + +How the GitHub App integration works and what you need to configure to get it +running — both the **GitHub App** side and your **local env**. This package +(`posthog/api/github_callback/`) handles the install/OAuth callbacks that create +the integration records. + +## What gets created + +There are two integration records, created by two related flows: + +| Record | Model | Created by | Used by | +| ------------------------------------- | ------------------------------------ | ----------------------------------------------------------------- | ---------------------------------------------- | +| **Team** `Integration(kind="github")` | `posthog/models/integration.py` | `team_services.create_team_github_integration_from_oauth_code` | Tasks/Code, signals custom agents, deployments | +| **Personal** `UserIntegration` | `posthog/models/user_integration.py` | `personal_finish.github_link_complete` (`/complete/github-link/`) | per-user GitHub linking | + +Both are created from a GitHub App **installation** plus a user-to-server OAuth +**code**. The App-as-App JWT (for installation tokens) is signed in +`github_integration_base.client_request` using the **Client ID as the JWT +issuer** — there is no separate numeric App ID env var. + +## GitHub App configuration + +Each engineer needs their own dev GitHub App (GitHub → Settings → Developer +Settings → GitHub Apps → New GitHub App). Assuming the default +`SITE_URL=http://localhost:8010` (the Caddy-fronted port you browse — not 8000): + +| App field | Value | Why | +| ---------------------------------------------------------- | ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Setup URL** (Post installation) | `http://localhost:8010/integrations/github/callback` | Where GitHub returns the browser after install (carries `installation_id`). Frontend `integrationsLogic` route → creates the **team** integration. Tick **Redirect on update**. | +| **Callback URL** (user authorization) | `http://localhost:8010/complete/github-link/` | The **personal** user-link return route (`github_link_complete`). | +| **Request user authorization (OAuth) during installation** | ✅ enabled | The team flow exchanges an OAuth `code`; without it, no `code` is returned and creation fails. | +| **Webhook → Active** | ❌ off for local | GitHub can't reach `localhost`. Only needed for inbound events (PR/issue/installation). If enabled, set the same secret as `GITHUB_WEBHOOK_SECRET` and point it at `…/webhooks/github`. | +| **Client secret** | generate one | Required for the OAuth code exchange (`GITHUB_APP_CLIENT_SECRET`). | +| **Private key** | generate a `.pem` | App-JWT signing → installation tokens (`GITHUB_APP_PRIVATE_KEY`). | + +`http://` localhost is fine — the callback flow uses `SITE_URL` verbatim (no +https rewrite), and a public URL is only required for inbound webhooks. The +install URL is built from the slug: +`https://github.com/apps/{GITHUB_APP_SLUG}/installations/new`, so the slug must +match your App exactly. + +### Permissions + +| Permission | Access | Purpose | +| ------------- | ------------------ | --------------------------------------------------- | +| Metadata | Read | Required for all GitHub Apps | +| Contents | Read (R/W to push) | Read files / repo discovery; R/W to create branches | +| Pull requests | Read & Write | Only if creating/updating PRs | + +Optional: Issues (R/W), Workflows (R/W). + +## Local env (`.env.local`) + +Set these to match your App. `.env.local` is loaded by `bin/start` (precedence: +shell env > `.env.local` > `.env.development` > `.env.services`); ad-hoc +`manage.py` runs outside `bin/start` won't have them. + +```bash +# OAuth Client ID — starts with Iv1. or Iv23 (NOT the numeric App ID). +# Doubles as the App-JWT issuer, so no separate GITHUB_APP_ID is needed. +GITHUB_APP_CLIENT_ID=Iv1.xxxxxxxxxxxxxxxx +GITHUB_APP_CLIENT_SECRET=your_client_secret +GITHUB_APP_SLUG=your-app-slug # URL-friendly name in github.com/apps/<slug> +GITHUB_APP_PRIVATE_KEY="-----BEGIN RSA PRIVATE KEY-----\n...\n-----END RSA PRIVATE KEY-----" + +# Only if you enable the webhook (must match the App's webhook secret): +# GITHUB_WEBHOOK_SECRET=... +``` + +Literal `\n` in the private key is fine — `client_request` converts them to real +newlines. `GITHUB_APP_SLUG` and `GITHUB_WEBHOOK_SECRET` are instance settings +(env-backed), so they can alternatively be set under instance settings in the DB. + +## Connecting (creating the integration record) + +Env vars alone create nothing — you must complete the install flow: + +1. Start the stack via `bin/start` / `hogli start`. +2. Go to `http://localhost:8010/project/<id>/settings/project-integrations`. +3. Connect **GitHub Integration** → you're sent to + `github.com/apps/<slug>/installations/new`. +4. Install the App on the account/org that owns your target repos. +5. GitHub returns you to `/integrations/github/callback` with `installation_id` + - `code`; the frontend POSTs them and the `Integration(kind="github")` row is + created (and the repository cache begins syncing). + +## Callback routes + +| Route | Handler | Flow | +| ------------------------------- | ----------------------------------------------------- | ------------------ | +| `/integrations/github/callback` | `integrationsLogic` (frontend) → `Integration` create | team integration | +| `/complete/github-link/` | `personal_finish.github_link_complete` | personal user-link | +| `/webhooks/github` (+ `/pr`) | `posthog/urls.py:github_webhook` | inbound events | + +The team code exchange (`team_services`) calls `github_user_from_code(code)` +**without** a `redirect_uri`, so GitHub does not enforce a redirect_uri match on +the token exchange — only the browser landing URL (the App's Setup/Callback URL) +matters. + +## Verify / troubleshoot + +```bash +# Integration present on a team? +.flox/cache/venv/bin/python manage.py shell -c \ + "from posthog.models.integration import Integration; print(Integration.objects.filter(kind='github').values_list('team_id', flat=True))" +``` + +| Symptom | Likely cause | +| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `Failed to exchange the OAuth code — ensure GITHUB_APP_CLIENT_SECRET is configured` | **Misleading message** — it fires for _any_ failed exchange, not just a missing secret. `team_services` raises it whenever `github_user_from_code()` returns `None`; the real reason is logged as a warning (`code exchange returned no access_token` with GitHub's `error_description`). See the rows below. | +| ↳ GitHub `error: incorrect_client_credentials` | **`GITHUB_APP_CLIENT_ID` holds the numeric App ID, not the OAuth Client ID** (most common), or the client secret is wrong/stale/from a different App. The Client ID starts with `Iv1.`/`Iv23`; the numeric App ID will not work for the token exchange. | +| ↳ GitHub `error: bad_verification_code` | The OAuth `code` expired (~10 min) or was already used — restart the connect flow to get a fresh one. | +| `GITHUB_APP_CLIENT_ID is not configured` | Env var unset, or running outside `bin/start` | +| Install redirects somewhere wrong / 404 | Setup URL not set to `…/integrations/github/callback` | +| No `code` returned → creation fails | "Request user authorization during installation" not enabled | +| Webhook signature verification fails | `GITHUB_WEBHOOK_SECRET` doesn't match the App's webhook secret | @@ -247,6 +247,7 @@ def test_workflows_remain_unchanged(self): "EmitEvalSignalWorkflow", "RunSignalsScoutWorkflow", "SignalsScoutCoordinatorWorkflow", + "CustomSignalAgentWorkflow", ] actual_workflow_names = [w.__name__ for w in SIGNALS_PRODUCT_WORKFLOWS] assert len(actual_workflow_names) == len(expected_workflows), ( @@ -299,6 +300,7 @@ def test_activities_remain_unchanged(self): "wait_for_signal_in_clickhouse_activity", "fetch_enabled_signals_scout_runs_activity", "run_signals_scout_activity", + "run_custom_signal_agent_activity", ] actual_activity_names = [a.__name__ for a in SIGNALS_PRODUCT_ACTIVITIES] assert len(actual_activity_names) == len(expected_activities), ( @@ -814,7 +814,7 @@ python manage.py enable_signals_autonomy <team_id> <priority> <emails> ### Auto-Start Flow -Runs inside `_maybe_autostart_task_for_report()` in `temporal/agentic/report.py`, called after artefact persistence in `run_agentic_report_activity`. +Runs inside `maybe_autostart_implementation_task()` in `backend/auto_start.py`, called after artefact persistence in both `run_agentic_report_activity` (the signals pipeline) and `run_custom_signal_agent_activity` (custom agents). **Guard clause** — all must pass: @@ -823,7 +823,7 @@ Runs inside `_maybe_autostart_task_for_report()` in `temporal/agentic/report.py` - Report has suggested reviewers - No existing `SignalReportTask` with `relationship=implementation` for this report -**User selection** via `_resolve_autostart_assignee()`: +**User selection** via `_resolve_autostart_assignee()` in `backend/auto_start.py`: 1. Map reviewer GitHub logins to PostHog user IDs via social auth (preserving reviewer relevance order) 2. Single query: fetch `User` objects whose ID is in that list **and** who have a `SignalUserAutonomyConfig` row (joined via `select_related`) @@ -847,11 +847,11 @@ Runs inside `_maybe_autostart_task_for_report()` in `temporal/agentic/report.py` All report ↔ task relationships are tracked via `SignalReportTask`: -| Relationship | Created when | Created where | -| ---------------- | ----------------------------------------------------- | -------------------------------------------------------------------- | -| `research` | Immediately after the research sandbox session starts | `run_multi_turn_research()` in `report_generation/research.py` | -| `implementation` | After auto-starting a coding task | `_maybe_autostart_task_for_report()` in `temporal/agentic/report.py` | -| `repo_selection` | Reserved for future use | Not yet created anywhere | +| Relationship | Created when | Created where | +| ---------------- | ----------------------------------------------------- | ------------------------------------------------------------------ | +| `research` | Immediately after the research sandbox session starts | `run_multi_turn_research()` in `report_generation/research.py` | +| `implementation` | After auto-starting a coding task | `maybe_autostart_implementation_task()` in `backend/auto_start.py` | +| `repo_selection` | Reserved for future use | Not yet created anywhere | Both `report` and `task` FKs cascade on delete — deleting a report or task cleans up the relationship rows automatically. new file mode 100644 @@ -0,0 +1,185 @@ +from __future__ import annotations + +from typing import TypedDict + +import structlog + +from posthog.models import Team, User +from posthog.sync import database_sync_to_async + +from products.signals.backend.models import SignalReportTask, SignalTeamConfig, SignalUserAutonomyConfig +from products.signals.backend.report_generation.research import ( + ActionabilityAssessment, + ActionabilityChoice, + Priority, + PriorityAssessment, +) +from products.signals.backend.report_generation.resolve_reviewers import resolve_org_github_login_to_users +from products.signals.backend.slack_inbox_notifications import POSTHOG_CODE_INBOX_DEEP_LINK_SCHEME +from products.tasks.backend.models import Task + +logger = structlog.get_logger(__name__) + + +class ReviewerContent(TypedDict): + github_login: str + github_name: str | None + relevant_commits: list[dict] + + +_PRIORITY_RANK: dict[Priority, int] = { + Priority.P0: 0, + Priority.P1: 1, + Priority.P2: 2, + Priority.P3: 3, + Priority.P4: 4, +} + + +def _priority_rank(priority: Priority) -> int: + return _PRIORITY_RANK[priority] + + +def _build_autostart_task_description( + *, report_id: str, summary: str, repository: str, priority: PriorityAssessment | None +) -> str: + priority_line = f"Priority: {priority.priority.value}\nReason: {priority.explanation}\n\n" if priority else "" + report_deep_link = f"{POSTHOG_CODE_INBOX_DEEP_LINK_SCHEME}://inbox/{report_id}" + return ( + f"{summary}\n\n" + f"{priority_line}" + f"Repository: {repository}\n\n" + "Act on this signal report. Investigate the root cause, implement the fix, and open a PR if appropriate.\n\n" + "When opening the PR, include this report deep link in the description footer, " + "making the footer '*Created with [PostHog Code](https://posthog.com/code?ref=pr) " + f"from [an inbox report]({report_deep_link}).' - " + "so the human reviewer can jump straight to it." + ) + + +def _resolve_autostart_assignee( + team_id: int, + report_priority: Priority, + reviewers_content: list[ReviewerContent], + team_default_priority: Priority, +) -> User | None: + """Return the first suggested reviewer whose effective priority threshold allows auto-start. + + Walks *reviewers_content* in order (most relevant first). For each reviewer + that maps to an org member with an autonomy config, resolves their effective + threshold (personal setting, falling back to the team default) and checks + whether the report's priority is high enough (lower rank = higher priority). + Returns the first matching ``User``, or ``None`` if nobody qualifies. + """ + login_to_user = resolve_org_github_login_to_users( + team_id, (str(r["github_login"]) for r in reviewers_content if r.get("github_login")) + ) + report_rank = _priority_rank(report_priority) + + # Map reviewer github logins to user IDs (preserving reviewer order) + candidate_user_ids: list[int] = [] + for reviewer in reviewers_content: + login = reviewer.get("github_login") + if not login: + continue + login = login.lower() + candidate = login_to_user.get(login) + if isinstance(candidate, User): + candidate_user_ids.append(candidate.id) + + if not candidate_user_ids: + return None + + # Single query: fetch users who have an autonomy config, joined eagerly + users_with_config = { + u.id: u + for u in User.objects.filter( + id__in=candidate_user_ids, + signal_autonomy_config__isnull=False, + teams__id=team_id, + ).select_related("signal_autonomy_config") + } + + # Walk in reviewer order (most relevant first) + for uid in candidate_user_ids: + user = users_with_config.get(uid) + if user is None: + continue + config: SignalUserAutonomyConfig = user.signal_autonomy_config + effective_threshold = ( + Priority(config.autostart_priority) if config.autostart_priority else team_default_priority + ) + if report_rank <= _priority_rank(effective_threshold): + return user + + return None + + +async def maybe_autostart_implementation_task( + *, + team_id: int, + report_id: str, + repository: str, + title: str, + summary: str, + actionability: ActionabilityAssessment, + reviewers_content: list[ReviewerContent], + priority: PriorityAssessment | None, +) -> None: + """Start an implementation Task for a SignalReport if autonomy + priority allow it. + + Idempotent: skipped if an IMPLEMENTATION task already exists for the report, + if the report is not immediately actionable, if it's already addressed, if + priority is missing, if there are no suggested reviewers, or if no reviewer's + autonomy threshold is met. + + Both the agentic signals pipeline (``temporal/agentic/report.py``) and the + custom agent activity (``temporal/custom_agent.py``) call this after persisting + their report and artefacts. Callers should wrap this in try/except so an + autostart failure does not fail the report itself. + """ + task_exists = await SignalReportTask.objects.filter( + team_id=team_id, report_id=report_id, relationship=SignalReportTask.Relationship.IMPLEMENTATION + ).aexists() + if ( + actionability.actionability != ActionabilityChoice.IMMEDIATELY_ACTIONABLE + or actionability.already_addressed + or priority is None + or not reviewers_content + or task_exists + ): + return + + team = await Team.objects.select_related("organization").aget(id=team_id) + team_config = await SignalTeamConfig.objects.filter(team_id=team_id).afirst() + team_default_priority = Priority(team_config.default_autostart_priority) if team_config else Priority.P0 + + task_user = await database_sync_to_async(_resolve_autostart_assignee, thread_sensitive=False)( + team_id, priority.priority, reviewers_content, team_default_priority + ) + if task_user is None: + return + + task = await database_sync_to_async(Task.create_and_run, thread_sensitive=False)( + team=team, + title=title, + description=_build_autostart_task_description( + report_id=report_id, summary=summary, repository=repository, priority=priority + ), + origin_product=Task.OriginProduct.SIGNAL_REPORT, + user_id=task_user.id, + repository=repository, + signal_report_id=report_id, + posthog_mcp_scopes="read_only", + interaction_origin="signal_report", # Makes the agent auto-push and open a draft PR + ) + task_run = await task.runs.order_by("-created_at").afirst() + if task_run is None: + raise RuntimeError(f"Task {task.id} auto-started without producing a TaskRun") + + await SignalReportTask.objects.acreate( + team_id=team_id, + report_id=report_id, + task=task, + relationship=SignalReportTask.Relationship.IMPLEMENTATION, + ) new file mode 100644 @@ -0,0 +1,282 @@ +# Custom Signal Agent + +Reference doc for the `products/signals/backend/custom_agent/` package: a thin abstraction that lets PostHog engineers ship one-off Signals-style agents that produce final READY reports in the Code Inbox without writing Temporal, sandbox, repo-selection, or persistence code. + +Status: feature-complete locally. No production pilot yet. No automated tests — the test file was deleted along with the `session_factory` DI seam because it was contorting the public interface; manual testing through the management command is the current bar. + +## What this is for + +You want to write an LLM-backed agent that: + +- Runs against a PostHog team (with or without a target GitHub repository). +- Optionally clones one repo into a sandbox, runs `gh`, SQL, MCP, etc. +- Produces a single final report (title, description, actionability, priority, assignees). +- Surfaces in the Code Inbox as a `SignalReport(status=READY)` with compatible artefacts. +- Optionally auto-starts a draft PR via the shared autonomy / priority logic. + +Pick this layer (instead of writing your own Temporal workflow) when you want one shared Temporal workflow + sandbox session lifecycle + persistence to do the boring work. + +## Module layout + +```text +products/signals/backend/ + custom_agent/ + __init__.py # public re-exports (includes run_agent / arun_agent) + base.py # CustomSignalAgent base class — Temporal-free SDK + schemas.py # workflow IO, identifier validation, assignee schema + loader.py # dotted-path import + identity validation for the activity + persistence.py # direct READY-report creation + artefacts + task link + examples/ + cookie_poem_agent.py # canonical minimal example (NO_REPO, one send()) + cursed_comment_agent.py # realistic example (repo research + agentic resolvers) + auto_start.py # shared with the agentic signals pipeline + temporal/custom_agent.py # workflow + activity + run_agent/arun_agent launchers + management/commands/ + run_custom_agent_example.py # CLI smoke test (--agent cookie_poem|cursed_comment) +``` + +`base.py` has zero `temporalio` / `posthog.temporal` imports. The agent is a +plain async Python object that you construct, drive via :py:meth:`start`, and +discard. The Temporal workflow + activity + launcher functions in +`temporal/custom_agent.py` are _one_ way to wrap that lifecycle — future +wrappers (in-process runners, Temporal schedules, queue consumers) can live +alongside without touching the SDK. + +## Public API + +### Subclass contract + +A custom agent is a `CustomSignalAgent` subclass that implements two methods: + +```python +class MyAgent(CustomSignalAgent): + @classmethod + def identifier(cls) -> tuple[str, str]: + return ("my_product", "my_type") # both must match [a-z0-9][a-z0-9_-]* + + async def run(self) -> bool: + result = await self.send("Do the thing.", MyOutputModel) + self.register_title(result.title) + self.register_description(result.body) + self.register_actionability(ActionabilityAssessment(...)) + self.register_priority(PriorityAssessment(...)) + self.register_assignees([CustomAgentAssignee(github_login="oliver")]) + return True # finalize and persist a trailing report +``` + +`run()` returns a bool: `True` finalizes a trailing report from the registered components; any falsy value emits no trailing report (use this when all reports were already emitted via `report_and_continue()`, or when the run intentionally produces nothing). `title` and `description` must be registered before any finalization point — they have **no default resolver**. Missing either raises `MissingReportComponentError` before any default-resolver LLM calls run. + +The class must live in an importable top-level module (no nested classes, no `__main__`); the Temporal launcher captures the dotted path automatically. + +### Starting an agent (Temporal-backed) + +```python +from products.signals.backend.temporal.custom_agent import run_agent, arun_agent + +handle = run_agent(MyAgent, team, initial_prompt, *, repository=None, id=None, model=None) +# or, in async code: +handle = await arun_agent(MyAgent, team, initial_prompt, *, repository=None, id=None, model=None) +``` + +Both are fire-and-forget module-level functions defined in `temporal/custom_agent.py`. They are **not** re-exported from `custom_agent/__init__.py` — doing so would create a circular import (the SDK package is Temporal-free; the Temporal wrapper depends on it, not the other way around). Import them from `temporal.custom_agent` directly. They start the shared `signals-custom-agent` Temporal workflow and return a `CustomAgentRunHandle(workflow_id, run_id, started)`. Reusing the same optional `id` while a workflow with that id is still running returns `started=False` instead of erroring; all other identity bits (`product`, `type`, `team_id`) are already known to the caller. + +The base class itself does not know about Temporal. Other framework wrappers (in-process runners for tests/scripts, Temporal schedules for periodic runs) can be added next to `run_agent` / `arun_agent` without changing the SDK. + +### `send()` + +```python +async def send( + self, + prompt: str, + output_model: type[T], # pydantic BaseModel + *, + label: str | None = None, + validation_retries: int | None = None, +) -> T +``` + +- First call lazily creates the `MultiTurnSession`. Later calls use `send_followup_raw`. +- Always wraps your prompt with `output_model.model_json_schema()` and "return JSON only". +- The first call also prepends an initial preamble with `initial_prompt`, repo context, and a "untrusted evidence" safety blurb. +- On validation failure, sends the validation error back and asks for a corrected response. Default `validation_retries` is 3 (overridable per call, including `0`). After exhaustion raises `CustomAgentValidationError(label, model_name, error, last_raw_text)`. + +### Register functions + +Plain setters. No coercion, no normalization, no overwrite guard: + +```python +register_title(title: str) # also enforces max_title_length (hard cap) +register_description(description: str) +register_actionability(ActionabilityAssessment) +register_priority(PriorityAssessment) +register_assignees(list[CustomAgentAssignee]) +``` + +`register_title` enforces `self.max_title_length` (default 255) as a hard cap. `title` and `description` have no default resolver, so they must be registered explicitly before any finalization point. + +If you call the same `register_*` twice, the second call silently wins. That's fine; build your `run()` so it doesn't happen accidentally. + +### `report_and_continue()` + +```python
Leaderboard for this case
medium · 5 of 9 bots fully caught it (56% solve rate) · 56% avg coverage.
| # | Bot | Result | Caught | Coverage | Findings | False alarms |
|---|---|---|---|---|---|---|
| 1 | Claude Code (Opus 5.5) | solved | 1/1 | 100.0% | 3 | 0 |
| 2 | Drape (Deep Review) | solved | 1/1 | 100.0% | 6 | 0 |
| 3 | Mjolnir (Opus 5.5) | solved | 1/1 | 100.0% | 6 | 0 |
| 4 | Mjolnir (Sonnet 5.5) | solved | 1/1 | 100.0% | 2 | 0 |
| 5 | qodo | solved | 1/1 | 100.0% | 3 | 0 |
| 6 | Cubic | miss | 0/1 | 0.0% | 7 | 0 |
| 7 | cursor | miss | 0/1 | 0.0% | 1 | 0 |
| 8 | Drape (Review) | miss | 0/1 | 0.0% | 4 | 0 |
| 9 | greptile | miss | 0/1 | 0.0% | 5 | 0 |
Bot results
Claude Code (Opus 5.5) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
| Defect | Severity | Found | Grade |
|---|---|---|---|
| d1 | high | ✓ | exact |
start()'s finally block calls session.end() with the default status="completed" even when run(), a resolver or persistence raised. MultiTurnSession.end's docstring says failed runs must pass status="failed", so a failed custom-agent run is recorded as a completed TaskRun. start_raw only handles failures on the first turn. [grounded: diff] [consensus 3/3: valid=3]
Linking several reports to the same task is the documented design ('Each persisted report links to the same task'). The finding names no consumer that assumes a 1:1 mapping, and a repo search found only IMPLEMENTATION-relationship lookups by task. The claimed breakage is hypothetical. [uncertain: no evidence quoted] [consensus 3/4: uncertain=3, invalid=1]
raw comments (3)
The old per-user check `user.teams.filter(id=team_id).exists()` was replaced with an ORM filter `teams__id=team_id` on `User.objects`. `User.teams` is a `cached_property` that returns a Team queryset (posthog/models/user.py:306). It is not a model field or reverse relation; no relation from Team to User uses related_name 'teams'. So `User.objects.filter(teams__id=...)` raises `FieldError: Cannot resolve keyword 'teams'` whenever there is at least one candidate reviewer. Both callers wrap autostart in try/except, so the error is only logged and autostart of implementation tasks silently stops working for every signal report and custom agent report. The swap also drops the access-control filtering the property applied (private-project restrictions).
`start()` always calls `self._session.end()` in `finally` with the default `status="completed"`, even when `run()`, a resolver, validation retries, or persistence raised. `MultiTurnSession.end` documents that failed runs must pass `status="failed"`, because otherwise the TaskRun is recorded as completed, which corrupts run-status metrics and misleads triage. Failed custom agent runs will show up as successful sandbox runs.
Every report from a multi-report run (`report_and_continue`) creates a `SignalReportTask` with relationship RESEARCH that points to the same `task_id` (the one shared session task). If the UI/APIs assume one research task per report or a 1:1 mapping from task to report, lookups by task can return several reports or the wrong one.
Cubic · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
| Defect | Severity | Found | Grade |
|---|---|---|---|
| d1 | high | ✗ | — |
This asks for a docstring change about who owns teardown. It is documentation, not a behaviour bug. The actual mis-reporting of status is covered by finding 2. [grounded: tree products/tasks/backend/services/custom_prompt_multi_turn_runner.py] [consensus 5/5: minor=5]
A request for test coverage. It does not claim the code misbehaves. [grounded: tree products/signals/backend/auto_start.py] [consensus 5/5: minor=5]
`end()` defaults to status="completed". The `finally` in `start()` calls it whatever happened, so a run that raises (including a `CustomAgentValidationError` from `send`) is recorded as completed. [grounded: tree products/signals/backend/custom_agent/base.py] [consensus 5/5: valid=5]
`rpartition(".")` leaves no dot in `class_name`, so this guard can never fire. A nested path falls through to the generic module-import error instead. [grounded: tree products/signals/backend/custom_agent/loader.py] [consensus 5/5: valid=5]
`__init__.py` does not export `run_agent`/`arun_agent`, and the same doc's 'Starting an agent' section says they are not re-exported, so the layout comment is wrong. [grounded: tree products/signals/backend/custom_agent/PLAN.md] [consensus 5/5: valid=5]
The GitHub-integration requirement is intentional and documented in PLAN.md. The finding amounts to asking for an extra note in the command's docstring. [consensus 3/5: minor=3, invalid=2]
`select_repository.py` imports temporalio and `posthog.temporal.common`. That contradicts PLAN.md's claim that base.py has no temporalio imports, and it adds a circular-import risk through the `temporal` package's `__init__`, which imports `custom_agent.base`. [grounded: diff] [consensus 5/5: valid=5]
raw comments (8)
<!-- cubic:v=b26d69a0-a9e7-411a-8fb4-a09c5a5bbcdf -->
<!-- cubic:review-run=78a91415-23b8-59c8-b7a7-151a54cfec90 -->
<!-- metadata:{"confidence":6} -->
P2: `start_raw` moves extract/validate out to the caller but its docstring doesn't state that the caller now owns teardown. `start` ends the session with `status="failed"` when the first turn fails to parse; a `start_raw` caller that lets validation raise leaves the TaskRun unreported or (in base.py's `finally`) reported as `completed`. Document the contract in the docstring: on validation failure the caller must call `end(status="failed", error=...)`.
<details>
<summary>Prompt for AI agents</summary>
```text
Check if this issue is valid — if so, understand the root cause and fix it. At products/tasks/backend/services/custom_prompt_multi_turn_runner.py, line 109:
<comment>`start_raw` moves extract/validate out to the caller but its docstring doesn't state that the caller now owns teardown. `start` ends the session with `status="failed"` when the first turn fails to parse; a `start_raw` caller that lets validation raise leaves the TaskRun unreported or (in base.py's `finally`) reported as `completed`. Document the contract in the docstring: on validation failure the caller must call `end(status="failed", error=...)`.</comment>
<file context>
@@ -60,19 +60,62 @@ async def start(
+ internal: bool = False,
+ on_task_run_created: Callable[[TaskRun], Awaitable[None]] | None = None,
+ ) -> tuple[MultiTurnSession, str]:
+ """Start a multi-turn sandbox session and return the first raw agent response.
+
+ `on_task_run_created`, if given, is awaited once the `TaskRun` exists but
</file context>
```
</details>
```suggestion
"""Start a multi-turn sandbox session and return the first raw agent response.
Callers own parsing and teardown: unlike `start`, a validation failure here is not
reported to the workflow, so call `end(status="failed", error=...)` yourself.
`on_task_run_created`, if given, is awaited once the `TaskRun` exists but
BEFORE the agent's first turn runs — see `start` for the rationale.
"""
```
<!-- cubic:v=2c69cea0-1d84-4499-aa6b-3487aa08301b -->
<!-- cubic:review-run=78a91415-23b8-59c8-b7a7-151a54cfec90 -->
<!-- metadata:{"confidence":7} -->
P2: The new `auto_start.py` module ships without tests, unlike its neighbours in `products/signals/backend/test/`. The two behaviours most likely to regress are untested: the threshold comparison in `_resolve_autostart_assignee` (`report_rank <= _priority_rank(effective_threshold)`, including the `team_default_priority` fallback when `autostart_priority` is null) and the idempotency guard in `maybe_autostart_implementation_task`. Add `products/signals/backend/test/test_auto_start.py` covering: P0 report with no config, user override lower/higher than team default, `already_addressed`/non-actionable skip, and the second call for the same report not creating another task.
<details>
<summary>Prompt for AI agents</summary>
```text
Check if this issue is valid — if so, understand the root cause and fix it. At products/signals/backend/auto_start.py, line 60:
<comment>The new `auto_start.py` module ships without tests, unlike its neighbours in `products/signals/backend/test/`. The two behaviours most likely to regress are untested: the threshold comparison in `_resolve_autostart_assignee` (`report_rank <= _priority_rank(effective_threshold)`, including the `team_default_priority` fallback when `autostart_priority` is null) and the idempotency guard in `maybe_autostart_implementation_task`. Add `products/signals/backend/test/test_auto_start.py` covering: P0 report with no config, user override lower/higher than team default, `already_addressed`/non-actionable skip, and the second call for the same report not creating another task.</comment>
<file context>
@@ -0,0 +1,185 @@
+ )
+
+
+def _resolve_autostart_assignee(
+ team_id: int,
+ report_priority: Priority,
</file context>
```
</details>
<!-- cubic:v=de7c7d15-d915-4d74-add7-9ab5deff62dc -->
<!-- cubic:review-run=78a91415-23b8-59c8-b7a7-151a54cfec90 -->
<!-- metadata:{"confidence":8} -->
P2: When `run()` or finalization raises, the `finally` block still ends the session with the default `status="completed"`, so a failed custom-agent run records its `TaskRun` as completed. Track the exception and call `await self._session.end(status="failed", error=str(exc))` on the failure path; the activity does not end the session itself, so this `finally` is the only place the terminal status is signalled.
<details>
<summary>Prompt for AI agents</summary>
```text
Check if this issue is valid — if so, understand the root cause and fix it. At products/signals/backend/custom_agent/base.py, line 374:
<comment>When `run()` or finalization raises, the `finally` block still ends the session with the default `status="completed"`, so a failed custom-agent run records its `TaskRun` as completed. Track the exception and call `await self._session.end(status="failed", error=str(exc))` on the failure path; the activity does not end the session itself, so this `finally` is the only place the terminal status is signalled.</comment>
<file context>
@@ -0,0 +1,627 @@
+ finally:
+ if self._session is not None:
+ try:
+ await self._session.end()
+ except Exception:
+ logger.warning("custom signal agent session cleanup failed", exc_info=True)
</file context>
```
</details>
<!-- cubic:v=cd44f5e5-d29d-4168-8422-c8b9e2b2dc73 -->
<!-- cubic:review-run=78a91415-23b8-59c8-b7a7-151a54cfec90 -->
<!-- metadata:{"confidence":9} -->
P3: The `if "." in class_name` guard never fires: `rpartition(".")` splits at the last dot, so `class_name` cannot contain one. Nested paths like `pkg.mod.Outer.Inner` therefore skip this branch and surface the misleading 'Could not import custom signal agent module' error at line 31 instead of the intended top-level-class message. Remove the unreachable branch, or detect nesting before the split.
<details>
<summary>Prompt for AI agents</summary>
```text
Check if this issue is valid — if so, understand the root cause and fix it. At products/signals/backend/custom_agent/loader.py, line 25:
<comment>The `if "." in class_name` guard never fires: `rpartition(".")` splits at the last dot, so `class_name` cannot contain one. Nested paths like `pkg.mod.Outer.Inner` therefore skip this branch and surface the misleading 'Could not import custom signal agent module' error at line 31 instead of the intended top-level-class message. Remove the unreachable branch, or detect nesting before the split.</comment>
<file context>
@@ -0,0 +1,59 @@
+ module_path, separator, class_name = agent_path.strip().rpartition(".")
+ if not separator or not module_path or not class_name:
+ raise CustomAgentLoadError(f"Invalid agent_path {agent_path!r}; expected 'module.ClassName'")
+ if "." in class_name:
+ raise CustomAgentLoadError("agent_path must point to a top-level class")
+
</file context>
```
</details>
<!-- cubic:v=397b1f8f-bc7c-4842-aa1a-c869b147eb86 -->
<!-- cubic:review-run=78a91415-23b8-59c8-b7a7-151a54cfec90 -->
<!-- metadata:{"confidence":9} -->
P3: The module-layout comment says `__init__.py` publicly re-exports `run_agent`/`arun_agent`, but the actual `custom_agent/__init__.py` does not export them, and the 'Starting an agent' section (line 81) says the opposite. The layout diagram misleads a reader into importing from the wrong module. Change the comment to note that the Temporal launchers live in `temporal/custom_agent.py` and are intentionally not re-exported.
<details>
<summary>Prompt for AI agents</summary>
```text
Check if this issue is valid — if so, understand the root cause and fix it. At products/signals/backend/custom_agent/PLAN.md, line 24:
<comment>The module-layout comment says `__init__.py` publicly re-exports `run_agent`/`arun_agent`, but the actual `custom_agent/__init__.py` does not export them, and the 'Starting an agent' section (line 81) says the opposite. The layout diagram misleads a reader into importing from the wrong module. Change the comment to note that the Temporal launchers live in `temporal/custom_agent.py` and are intentionally not re-exported.</comment>
<file context>
@@ -0,0 +1,282 @@
+```text
+products/signals/backend/
+ custom_agent/
+ __init__.py # public re-exports (includes run_agent / arun_agent)
+ base.py # CustomSignalAgent base class — Temporal-free SDK
+ schemas.py # workflow IO, identifier validation, assignee schema
</file context>
```
</details>
```suggestion
__init__.py # public re-exports (SDK only; run_agent/arun_agent live in temporal/custom_agent.py)
```
<!-- cubic:v=987f15e3-6d29-49a2-8eb4-e3680afbfbbb -->
<!-- cubic:review-run=78a91415-23b8-59c8-b7a7-151a54cfec90 -->
<!-- metadata:{"confidence":6} -->
P3: The docstring calls the cookie poem example "no repo, fully static — the minimal example", but `CustomSignalAgent.start()` always resolves `user_id` through `resolve_user_id_for_team`, which raises when the team has no GitHub integration. The documented minimal example therefore fails on a GitHub-less team; either state the GitHub-integration requirement in the usage block or add a `--user-id` option to the command.
<details>
<summary>Prompt for AI agents</summary>
```text
Check if this issue is valid — if so, understand the root cause and fix it. At products/signals/backend/management/commands/run_custom_agent_example.py, line 7:
<comment>The docstring calls the cookie poem example "no repo, fully static — the minimal example", but `CustomSignalAgent.start()` always resolves `user_id` through `resolve_user_id_for_team`, which raises when the team has no GitHub integration. The documented minimal example therefore fails on a GitHub-less team; either state the GitHub-integration requirement in the usage block or add a `--user-id` option to the command.</comment>
<file context>
@@ -0,0 +1,117 @@
+
+Usage::
+
+ # Cookie poem (no repo, fully static — the minimal example)
+ python manage.py run_custom_agent_example --agent cookie_poem --team-id 1
+
</file context>
```
</details>
<!-- cubic:v=39ab7ba0-7ec4-4008-8228-8306ddb44100 -->
<!-- cubic:review-run=78a91415-23b8-59c8-b7a7-151a54cfec90 -->
<!-- metadata:{"confidence":6} -->
P3: `base.py` imports `products.signals.backend.temporal.agentic.select_repository`, which imports `temporalio` and `posthog.temporal.common.*`, so importing `products.signals.backend.custom_agent` now pulls the Temporal layer into every agent module. That contradicts the package's documented design (`custom_agent/__init__.py`: "Temporal-free SDK"; PLAN.md: "base.py has zero temporalio / posthog.temporal imports"), and the example agents' docstrings warn that agent modules pulling in the Temporal layer at module load time hit partial-load `ImportError`s. Move `GITHUB_ONLY_DOMAINS` (and the other constants base.py needs) into a Temporal-free module.
<details>
<summary>Prompt for AI agents</summary>
```text
Check if this issue is valid — if so, understand the root cause and fix it. At products/signals/backend/custom_agent/base.py, line 31:
<comment>`base.py` imports `products.signals.backend.temporal.agentic.select_repository`, which imports `temporalio` and `posthog.temporal.common.*`, so importing `products.signals.backend.custom_agent` now pulls the Temporal layer into every agent module. That contradicts the package's documented design (`custom_agent/__init__.py`: "Temporal-free SDK"; PLAN.md: "base.py has zero temporalio / posthog.temporal imports"), and the example agents' docstrings warn that agent modules pulling in the Temporal layer at module load time hit partial-load `ImportError`s. Move `GITHUB_ONLY_DOMAINS` (and the other constants base.py needs) into a Temporal-free module.</comment>
<file context>
@@ -0,0 +1,627 @@
+ get_or_create_signals_sandbox_env,
+ resolve_user_id_for_team,
+)
+from products.signals.backend.temporal.agentic.select_repository import GITHUB_ONLY_DOMAINS
+from products.tasks.backend.models import SandboxEnvironment, Task
+from products.tasks.backend.services.custom_prompt_internals import CustomPromptSandboxContext, extract_json_from_text
</file context>
```
</details>
<!-- cubic:review-summary:start -->
**7 issues found** across 23 files
<!-- cubic:review-summary:end -->
<details>
<summary>Prompt for AI agents (unresolved issues)</summary>
```text
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="products/signals/backend/custom_agent/loader.py">
<violation number="1" location="products/signals/backend/custom_agent/loader.py:25">
P3: The `if "." in class_name` guard never fires: `rpartition(".")` splits at the last dot, so `class_name` cannot contain one. Nested paths like `pkg.mod.Outer.Inner` therefore skip this branch and surface the misleading 'Could not import custom signal agent module' error at line 31 instead of the intended top-level-class message. Remove the unreachable branch, or detect nesting before the split.</violation>
</file>
<file name="products/signals/backend/custom_agent/PLAN.md">
<violation number="1" location="products/signals/backend/custom_agent/PLAN.md:24">
P3: The module-layout comment says `__init__.py` publicly re-exports `run_agent`/`arun_agent`, but the actual `custom_agent/__init__.py` does not export them, and the 'Starting an agent' section (line 81) says the opposite. The layout diagram misleads a reader into importing from the wrong module. Change the comment to note that the Temporal launchers live in `temporal/custom_agent.py` and are intentionally not re-exported.</violation>
</file>
<file name="products/signals/backend/management/commands/run_custom_agent_example.py">
<violation number="1" location="products/signals/backend/management/commands/run_custom_agent_example.py:7">
P3: The docstring calls the cookie poem example "no repo, fully static — the minimal example", but `CustomSignalAgent.start()` always resolves `user_id` through `resolve_user_id_for_team`, which raises when the team has no GitHub integration. The documented minimal example therefore fails on a GitHub-less team; either state the GitHub-integration requirement in the usage block or add a `--user-id` option to the command.</violation>
</file>
<file name="products/tasks/backend/services/custom_prompt_multi_turn_runner.py">
<violation number="1" location="products/tasks/backend/services/custom_prompt_multi_turn_runner.py:109">
P2: `start_raw` moves extract/validate out to the caller but its docstring doesn't state that the caller now owns teardown. `start` ends the session with `status="failed"` when the first turn fails to parse; a `start_raw` caller that lets validation raise leaves the TaskRun unreported or (in base.py's `finally`) reported as `completed`. Document the contract in the docstring: on validation failure the caller must call `end(status="failed", error=...)`.</violation>
</file>
<file name="products/signals/backend/auto_start.py">
<violation number="1" location="products/signals/backend/auto_start.py:60">
P2: The new `auto_start.py` module ships without tests, unlike its neighbours in `products/signals/backend/test/`. The two behaviours most likely to regress are untested: the threshold comparison in `_resolve_autostart_assignee` (`report_rank <= _priority_rank(effective_threshold)`, including the `team_default_priority` fallback when `autostart_priority` is null) and the idempotency guard in `maybe_autostart_implementation_task`. Add `products/signals/backend/test/test_auto_start.py` covering: P0 report with no config, user override lower/higher than team default, `already_addressed`/non-actionable skip, and the second call for the same report not creating another task.</violation>
</file>
<file name="products/signals/backend/custom_agent/base.py">
<violation number="1" location="products/signals/backend/custom_agent/base.py:31">
P3: `base.py` imports `products.signals.backend.temporal.agentic.select_repository`, which imports `temporalio` and `posthog.temporal.common.*`, so importing `products.signals.backend.custom_agent` now pulls the Temporal layer into every agent module. That contradicts the package's documented design (`custom_agent/__init__.py`: "Temporal-free SDK"; PLAN.md: "base.py has zero temporalio / posthog.temporal imports"), and the example agents' docstrings warn that agent modules pulling in the Temporal layer at module load time hit partial-load `ImportError`s. Move `GITHUB_ONLY_DOMAINS` (and the other constants base.py needs) into a Temporal-free module.</violation>
<violation number="2" location="products/signals/backend/custom_agent/base.py:374">
P2: When `run()` or finalization raises, the `finally` block still ends the session with the default `status="completed"`, so a failed custom-agent run records its `TaskRun` as completed. Track the exception and call `await self._session.end(status="failed", error=str(exc))` on the failure path; the activity does not end the session itself, so this `finally` is the only place the terminal status is signalled.</violation>
</file>
```
</details>
<sub>**Tip**: cubic can generate docs of your entire codebase and keep them up to date. Try it [here](https://docs.cubic.dev/wiki/ai-wiki?utm_source=github).<br /><br />[Re-trigger cubic](https://www.cubic.dev/action/re-review/pr/drape-io/crbench-cubic-2026-09/67/ai_pr_review_github_delivery_97d5540512d9065aa5cc73998fa9c7ee6f93d159dcb2deef82bce8515046a01a?returnTo=https%3A%2F%2Fgithub.com%2Fdrape-io%2Fcrbench-cubic-2026-09%2Fpull%2F67)</sub>
<!-- cubic:review-post:ai_pr_review_github_delivery_97d5540512d9065aa5cc73998fa9c7ee6f93d159dcb2deef82bce8515046a01a:1b277b8016d42f7559cb4844d5aa37710bd7d1c9:78a91415-23b8-59c8-b7a7-151a54cfec90 -->
cursor · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
| Defect | Severity | Found | Grade |
|---|---|---|---|
| d1 | high | ✗ | — |
`start()` runs `self._session.end()` with no arguments in its finally block. `end()` defaults to `status="completed"`, so a failure or cancellation raised from `run()` or finalization is still recorded as a completed TaskRun. This is new code added by the change, and it ignores the `status="failed"` convention the module already uses. [grounded: tree products/signals/backend/custom_agent/base.py] [consensus 5/5: valid=5]
raw comments (2)
### Failed agent sessions marked completed
**Medium Severity**
<!-- DESCRIPTION START -->
`CustomSignalAgent.start` always calls `session.end()` with the default `status="completed"` in `finally`, including when `run()` or finalization raises. Failed or cancelled custom-agent runs then record the sandbox `TaskRun` as successful, which corrupts run-status metrics and hides real failures during triage.
<!-- DESCRIPTION END -->
<!-- BUGBOT_BUG_ID: 230e3399-d4de-440b-b166-7b91e23a6a5d -->
<!-- LOCATIONS START
products/signals/backend/custom_agent/base.py#L370-L376
LOCATIONS END -->
<div><a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9DVVJTT1IiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjVhMDU0YWYyLWY2YmYtNDUwNy04N2Q1LWEzOGZhZTM3ZmQ4YSIsImVuY3J5cHRpb25LZXkiOiI4MWJtU3JnMXYxem5xeVdfOUczaFlrYjJDZUhnM3JuYjZFbDNfcU9EejE0IiwiYnJhbmNoIjoidDItcG9zdGhvZy01ODg0NyIsInJlcG9Pd25lciI6ImRyYXBlLWlvIiwicmVwb05hbWUiOiJjcmJlbmNoLWN1cnNvci0yMDI2LTA5IiwicHJvdmlkZXIiOiJnaXRodWIifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a> <a href="https://cursor.com/agents?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9XRUIiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjVhMDU0YWYyLWY2YmYtNDUwNy04N2Q1LWEzOGZhZTM3ZmQ4YSIsImVuY3J5cHRpb25LZXkiOiI4MWJtU3JnMXYxem5xeVdfOUczaFlrYjJDZUhnM3JuYjZFbDNfcU9EejE0IiwiYnJhbmNoIjoidDItcG9zdGhvZy01ODg0NyIsInJlcG9Pd25lciI6ImRyYXBlLWlvIiwicmVwb05hbWUiOiJjcmJlbmNoLWN1cnNvci0yMDI2LTA5IiwicHJOdW1iZXIiOjY5LCJjb21taXRTaGEiOiJlY2JiNGRiZGUzNjYxMDFlMmFhYjQwNWU5ZTJkZmJhNzE4ZWViYjdiIiwicHJvdmlkZXIiOiJnaXRodWIifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-web-light.png"><img alt="Fix in Web" width="99" height="28" src="https://cursor.com/assets/images/fix-in-web-dark.png"></picture></a></div>
<sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit ecbb4dbde366101e2aab405e9e2dfba718eebb7b. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- BUGBOT_REVIEW -->
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
<!-- BUGBOT_FIX_ALL -->
<a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9BTExfSU5fQ1VSU09SIiwiZGF0YSI6eyJyZWRpc0tleSI6ImJ1Z2JvdC1tdWx0aTo4ZDI3NWZlMS03NTkzLTRmNjAtOWQ1OS1mMmZjNjU2OTFiYzUiLCJlbmNyeXB0aW9uS2V5IjoiTG5hMk5IN2h5RW00SHpvUHNpN2ZNd21FRGFCVHkyUnQ1eTA4a0xyVk9xdyIsImJyYW5jaCI6InQyLXBvc3Rob2ctNTg4NDciLCJyZXBvT3duZXIiOiJkcmFwZS1pbyIsInJlcG9OYW1lIjoiY3JiZW5jaC1jdXJzb3ItMjAyNi0wOSIsInByb3ZpZGVyIjoiZ2l0aHViIn19" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix All in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a>
<!-- /BUGBOT_FIX_ALL -->
<!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_BEGIN -->
<sup>❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the [Cursor dashboard](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_END -->
<sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit ecbb4dbde366101e2aab405e9e2dfba718eebb7b. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
Drape (Deep Review) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
| Defect | Severity | Found | Grade |
|---|---|---|---|
| d1 | high | ✓ | exact |
The finally block in start() calls end() with the default status of completed, whether run() succeeded or raised. The call isn't shielded, unlike the research.py pattern that ends failed sessions as failed. [grounded: tree products/signals/backend/custom_agent/base.py] [consensus 3/3: valid=3]
Custom-agent assignees come straight from the LLM's resolve_assignees output. They are fed to autostart and mapped to any opted-in team member, with no check against commit authors like the agentic pipeline does via resolve_suggested_reviewers. [grounded: diff] [consensus 3/3: valid=3]
The removed guard in report.py had no already_addressed condition; the shared auto_start guard adds one, so the agentic pipeline's behavior changes for reports flagged already_addressed. [grounded: diff] [consensus 3/3: valid=3]
Both type and run id may contain hyphens (the regex allows them), so joining them with a hyphen can make distinct (type, id) pairs collide and silently return started=False. [grounded: diff] [consensus 3/3: valid=3]
The consent check exists only in arun_agent; the --local command path and direct start() calls skip it. The command is DEBUG-only and the plan documents this choice, so impact is low. [grounded: diff] [consensus 3/3: valid=3]
raw comments (6)
**Failed custom agent runs are marked completed**
When `run()` or finalization raises or is cancelled, `start()` still ends the session with the default "completed" status, unshielded. On exception (including CancelledError), call `asyncio.shield(self._session.end(status="failed", error=str(e)))`; call plain `end()` only on success.
```
products/signals/backend/custom_agent/base.py:364 self.user_id = await database_sync_to_async(resolve_user_id_for_team, thread_sensitive=False)(self.team_id)
products/signals/backend/custom_agent/base.py:365 self._resolved_repository = await self._resolve_repository()
products/signals/backend/custom_agent/base.py:366 try:
products/signals/backend/custom_agent/base.py:367 should_finalize = await self.run()
products/signals/backend/report_generation/research.py:628 await session.end()
products/signals/backend/report_generation/research.py:629 except (Exception, asyncio.CancelledError) as e:
products/signals/backend/report_generation/research.py:630 # Shield so the session ending cannot itself be canceled - must complete
products/signals/backend/report_generation/research.py:631 await asyncio.shield(session.end(status="failed", error=str(e)))
products/signals/backend/report_generation/research.py:632 raise
products/tasks/backend/services/custom_prompt_multi_turn_runner.py:252 async def end(self, *, status: str = "completed", error: str | None = None) -> None:
```
**LLM-picked assignees can auto-start tasks as any user**
Autostart uses GitHub logins the LLM chose, not commit authors checked through GitHub as the agentic pipeline does. Repo or prompt content can make it name any opted-in org member. Check logins against commit authorship (e.g. `resolve_suggested_reviewers`) before autostart.
```
products/signals/backend/custom_agent/base.py:487 await maybe_autostart_implementation_task(
products/signals/backend/custom_agent/base.py:488 team_id=self.team_id,
products/signals/backend/custom_agent/base.py:489 report_id=persisted.report_id,
products/signals/backend/custom_agent/base.py:490 repository=repository,
products/signals/backend/temporal/agentic/report.py:154 if not commit_hashes_with_reasons or not repository:
products/signals/backend/temporal/agentic/report.py:155 return []
products/signals/backend/temporal/agentic/report.py:157 resolved = resolve_suggested_reviewers(team_id, repository, commit_hashes_with_reasons)
products/signals/backend/auto_start.py:74 login_to_user = resolve_org_github_login_to_users(
products/signals/backend/auto_start.py:75 team_id, (str(r["github_login"]) for r in reviewers_content if r.get("github_login"))
products/signals/backend/auto_start.py:76 )
```
**Auto-start always fails on the `teams__id` filter**
`User.teams` is a Python property, not a database relation, so this filter raises a FieldError whenever a reviewer maps to an org member. The caller swallows it, so no task ever auto-starts. Restore the per-user `user.teams.filter(id=team_id).exists()` check.
```
products/signals/backend/auto_start.py:95 u.id: u
products/signals/backend/auto_start.py:96 for u in User.objects.filter(
products/signals/backend/auto_start.py:97 id__in=candidate_user_ids,
products/signals/backend/auto_start.py:98 signal_autonomy_config__isnull=False,
posthog/models/user.py:306 @cached_property
posthog/models/user.py:307 def teams(self):
posthog/models/user.py:308 """
posthog/models/user.py:309 All teams the user has access to on any organization, taking into account project based permissioning
posthog/models/user.py:310 """
posthog/models/user.py:311 teams = Team.objects.filter(organization__members=self)
```
**Agentic reports flagged already-addressed no longer auto-start**
Moving the autostart code adds an `actionability.already_addressed` skip, so immediately-actionable agentic reports with that flag used to auto-start and now silently don't. If this is intended, document it and add a test; otherwise remove the check so the refactor keeps the old behavior.
```
products/signals/backend/auto_start.py:143 ).aexists()
products/signals/backend/auto_start.py:144 if (
products/signals/backend/auto_start.py:145 actionability.actionability != ActionabilityChoice.IMMEDIATELY_ACTIONABLE
products/signals/backend/auto_start.py:146 or actionability.already_addressed
products/signals/backend/temporal/agentic/report.py:282 (base) ) -> None:
products/signals/backend/temporal/agentic/report.py:283 (base) task_exists = await SignalReportTask.objects.filter(
products/signals/backend/temporal/agentic/report.py:284 (base) report_id=report_id, relationship=SignalReportTask.Relationship.IMPLEMENTATION
products/signals/backend/temporal/agentic/report.py:285 (base) ).aexists()
products/signals/backend/temporal/agentic/report.py:286 (base) if (
products/signals/backend/temporal/agentic/report.py:287 (base) result.actionability.actionability != ActionabilityChoice.IMMEDIATELY_ACTIONABLE
```
**Different agent runs can share a workflow id**
Types and run ids may contain hyphens and are joined with one, so type `scan` + id `deps-1` collides with type `scan-deps` + id `1`. The second launch silently returns `started=False`. Join type and run id with `:`.
```
products/signals/backend/temporal/custom_agent.py:56 def _workflow_id_for(agent_class: type[CustomSignalAgent], team_id: int, run_id: str) -> str:
products/signals/backend/temporal/custom_agent.py:57 product, type_ = validated_identifier(agent_class)
products/signals/backend/temporal/custom_agent.py:58 return f"signals-custom-agent:{team_id}:{product}:{type_}-{validate_run_id(run_id)}"
products/signals/backend/custom_agent/schemas.py:16 _IDENTIFIER_PART_RE = re.compile(r"^[a-z0-9][a-z0-9_-]*$")
```
**AI consent check skipped when agents run directly**
Only `arun_agent` checks `is_ai_data_processing_approved`. `start()` and the Temporal activity don't check it, so calling `agent.start()` directly (as the `--local` path does) sends team data to the LLM and sandbox without the org's consent. The class even defines `AIDataProcessingNotApprovedError` for this check. Run the consent check at the beginning of `start()`.
```
products/signals/backend/custom_agent/base.py:383 (raises :py:class:`CustomAgentRepositorySelectionError` if the
products/signals/backend/custom_agent/base.py:384 selector picks nothing).
products/signals/backend/custom_agent/base.py:385 """
products/signals/backend/custom_agent/base.py:386 repository = self._repository_input
products/signals/backend/custom_agent/base.py:74 class AIDataProcessingNotApprovedError(RuntimeError):
products/signals/backend/custom_agent/base.py:75 """Raised when the team's organization has not approved AI data processing.
products/signals/backend/custom_agent/base.py:77 Mirrors the consent gate enforced by ``emit_signal`` and the subscription
products/signals/backend/custom_agent/base.py:78 serializer (``organization.is_ai_data_processing_approved``). Custom
products/signals/backend/custom_agent/base.py:79 agents send team data through LLMs and sandboxes, so we refuse to launch
products/signals/backend/custom_agent/base.py:80 when consent is missing.
```
Drape (Review) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
| Defect | Severity | Found | Grade |
|---|---|---|---|
| d1 | high | ✗ | — |
The removed `_maybe_autostart_task_for_report` guard had no already_addressed check. The new shared function adds one, so the signals pipeline no longer auto-starts tasks for reports flagged already_addressed. [grounded: tree products/signals/backend/auto_start.py] [consensus 3/3: valid=3]
The consent check exists only in `arun_agent`. The `--local` path and the activity call `agent.start()` directly and never check `is_ai_data_processing_approved`, which contradicts the error docstring's 'refuse to launch' claim. PLAN.md documents this, but the gap is still real. [grounded: tree products/signals/backend/management/commands/run_custom_agent_example.py] [consensus 3/3: valid=3]
Both type and run_id may contain '-'. So type='a-b', id='c' and type='a', id='b-c' produce the same workflow ID, and the second run is silently reported as started=False. [grounded: tree products/signals/backend/temporal/custom_agent.py] [consensus 3/3: valid=3]
The `finally` block calls `end()` with the default status='completed' even when `run()` or finalization raised. Failed runs are therefore recorded as completed, unlike `MultiTurnSession.start`, which passes status='failed'. [grounded: diff-section products/signals/backend/custom_agent/base.py] [consensus 3/3: valid=3]
raw comments (4)
**Autostart now skips already-addressed reports**
The refactored `maybe_autostart_implementation_task` adds an `actionability.already_addressed` early return. The old agentic pipeline's `_maybe_autostart_task_for_report` had no such check. The existing signals report pipeline now silently stops auto-starting implementation tasks for reports flagged already addressed.
```
products/signals/backend/auto_start.py:143 ).aexists()
products/signals/backend/auto_start.py:144 if (
products/signals/backend/auto_start.py:145 actionability.actionability != ActionabilityChoice.IMMEDIATELY_ACTIONABLE
products/signals/backend/auto_start.py:146 or actionability.already_addressed
products/signals/backend/report_generation/research.py:80 already_addressed: bool = Field(
products/signals/backend/report_generation/research.py:81 description=(
products/signals/backend/report_generation/research.py:82 "Whether the core issue described by this report appears to have been "
products/signals/backend/report_generation/research.py:83 "already fixed or addressed in recent code changes. Tracked separately from `actionability`."
products/signals/backend/report_generation/research.py:84 ),
products/signals/backend/report_generation/research.py:85 )
```
**AI consent gate bypassed outside `arun_agent`**
The consent check lives only in `arun_agent`. `CustomSignalAgent.start()` and the `run_custom_signal_agent_activity` activity never check `is_ai_data_processing_approved`. The `AIDataProcessingNotApprovedError` docstring says custom agents refuse to launch without consent. Run the consent check in `start()` (or at the top of the activity) as well as in `arun_agent`.
```
products/signals/backend/custom_agent/base.py:77 Mirrors the consent gate enforced by ``emit_signal`` and the subscription
products/signals/backend/custom_agent/base.py:78 serializer (``organization.is_ai_data_processing_approved``). Custom
products/signals/backend/custom_agent/base.py:79 agents send team data through LLMs and sandboxes, so we refuse to launch
products/signals/backend/custom_agent/base.py:80 when consent is missing.
products/signals/backend/temporal/emit_eval_signal.py:140 if not organization.is_ai_data_processing_approved:
products/signals/backend/custom_agent/PLAN.md:247 - **AI data processing consent required.** `arun_agent` refuses to launch for organizations with `is_ai_data_processing_approved=False`, raising `AIDataProcessingNotApprovedError`. Mirrors the `emit_signal` gate. Checked only at the launcher boundary; not re-checked inside the activity.
```
**Workflow IDs can collide across type/run_id pairs**
The workflow ID joins type and run id with '-', but '-' is also allowed inside both parts. Different (type, id) pairs can therefore produce the same workflow ID. Use a delimiter that cannot appear in identifier parts, e.g. ':' between type and run id.
```
products/signals/backend/temporal/custom_agent.py:56 def _workflow_id_for(agent_class: type[CustomSignalAgent], team_id: int, run_id: str) -> str:
products/signals/backend/temporal/custom_agent.py:57 product, type_ = validated_identifier(agent_class)
products/signals/backend/custom_agent/schemas.py:16 _IDENTIFIER_PART_RE = re.compile(r"^[a-z0-9][a-z0-9_-]*$")
```
**Failed agent runs marked as completed**
`start()` always calls `self._session.end()` in `finally` with the default status. A run that raised (validation error, missing title, persistence failure) is closed as 'completed', unlike the first-turn failure path, which uses `status="failed"`. Catch the exception in `start()`, call `end(status="failed", error=...)`, then re-raise.
```
products/signals/backend/custom_agent/base.py:276 def register_actionability(self, actionability: ActionabilityAssessment) -> None:
products/signals/backend/custom_agent/base.py:277 self._actionability = actionability
products/signals/backend/custom_agent/base.py:279 def register_priority(self, priority: PriorityAssessment) -> None:
products/signals/backend/custom_agent/base.py:280 self._priority = priority
products/tasks/backend/services/custom_prompt_multi_turn_runner.py:252 async def end(self, *, status: str = "completed", error: str | None = None) -> None:
```
greptile · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
| Defect | Severity | Found | Grade |
|---|---|---|---|
| d1 | high | ✗ | — |
The consent check only runs in arun_agent. The activity loads the team and starts the agent without checking is_ai_data_processing_approved again. If consent is revoked while the workflow is queued, the run still sends team data to the LLM and sandbox. PLAN.md admits this but gives no safeguard. [grounded: tree products/signals/backend/temporal/custom_agent.py] [consensus 3/3: valid=3]
start() always calls end() with no arguments in its finally block. end() defaults to status="completed", so a run that fails after the session starts (validation, missing component or persistence error) is recorded as completed. [grounded: tree products/signals/backend/custom_agent/base.py] [consensus 3/3: valid=3]
Custom-agent reports are written straight to READY. Nothing in this path calls publish_report_completed_activity or dispatch_inbox_slack_notifications, which the summary workflow runs when a report becomes ready. Downstream consumers and suggested reviewers never hear about these reports. [grounded: tree products/signals/backend/custom_agent/persistence.py] [consensus 3/3: valid=3]
This asks for test coverage. It does not claim the code behaves incorrectly. [consensus 3/3: minor=3]
The layout comment says __init__.py re-exports run_agent and arun_agent. The actual __init__.py does not, and PLAN.md's own Public API section says it does not. The doc is wrong and points readers to an import that fails. [grounded: diff-section products/signals/backend/custom_agent/PLAN.md] [consensus 3/5: valid=3, minor=2]
raw comments (6)
<a href="#"><img alt="P1" src="https://greptile-static-assets.s3.amazonaws.com/badges/p1.svg?v=9" align="top"></a> <a href="#"><img alt="security" src="https://greptile-static-assets.s3.amazonaws.com/badges/Security.svg?v=2" align="top"></a> **Consent is not rechecked**
If an organization revokes AI-processing consent after `arun_agent` queues the workflow but before this activity executes, the activity still calls `agent.start()`. That path submits the initial request and repository context to the LLM and sandbox despite the current consent state. Recheck `team.organization.is_ai_data_processing_approved` at the execution boundary before importing or starting the agent.
**How this was verified:** The activity loads the organization but proceeds directly to an agent path that submits team request and repository data to AI without consulting the current consent flag.
<a href="#"><img alt="P2" src="https://greptile-static-assets.s3.amazonaws.com/badges/p2.svg?v=9" align="top"></a> **Failed sessions look successful**
This `finally` block always ends an initialized session with the default `"completed"` status. Validation exhaustion, missing report components, and persistence errors can all occur after session creation, so those failed task runs are recorded as successful and mislead run metrics and operational triage. Track whether execution failed and call `end(status="failed", error=...)` on exception paths.
<a href="#"><img alt="P2" src="https://greptile-static-assets.s3.amazonaws.com/badges/p2.svg?v=9" align="top"></a> **Ready side effects are missing**
Creating the report directly in `READY` state bypasses the normal post-ready orchestration. The custom-agent path never publishes the report-completed event or dispatches inbox Slack notifications, so event-driven consumers and completion telemetry miss these reports, and suggested reviewers are not notified. Run the shared completion and notification side effects for every successfully persisted custom-agent report.
<a href="#"><img alt="P2" src="https://greptile-static-assets.s3.amazonaws.com/badges/p2.svg?v=9" align="top"></a> **Framework lacks automated tests**
This adds a substantial lifecycle covering AI validation retries, repository selection, multi-report persistence, session cleanup, Temporal serialization, and autostart, but relies only on a manual management-command smoke test. Add focused automated coverage for successful runs, validation failures, cleanup status, multi-report resets, and persistence rollback so regressions in this shared framework are detected before deployment.
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
<a href="#"><img alt="P2" src="https://greptile-static-assets.s3.amazonaws.com/badges/p2.svg?v=9" align="top"></a> **Launcher export docs conflict**
This layout entry says `custom_agent/__init__.py` re-exports `run_agent` and `arun_agent`, while the public API section and implementation correctly state that it does not. Correcting the entry prevents developers from following an import path that fails.
```suggestion
__init__.py # public SDK re-exports (Temporal launchers excluded)
```
<h3>Greptile Summary</h3>
This PR adds a reusable custom Signals-agent framework backed by Temporal, shared repository selection and autostart logic, direct READY-report persistence, example agents, and supporting documentation. It also extracts raw-response support from the multi-turn session runner.
- Adds the `CustomSignalAgent` SDK, validation schemas, dynamic loader, persistence layer, Temporal workflow/activity, and local examples.
- Extracts Signals implementation-task autostart into a shared module.
- Generalizes repository selection for free-form custom-agent requests.
- Adds raw first-turn and follow-up APIs while retaining the existing structured multi-turn interface.
- Adds GitHub App integration setup documentation.
<h3>Confidence Score: 3/5</h3>
The PR is not safe to merge until the custom-agent activity enforces current AI-processing consent; the remaining findings concern lifecycle status, missing report side effects, tests, and documentation.
A queued custom-agent workflow can continue into LLM processing after organization consent is revoked because consent is checked only before launch. Failed agent sessions are also recorded as completed, and directly persisted reports bypass completion events and reviewer notifications.
**Files Needing Attention:** products/signals/backend/temporal/custom_agent.py, products/signals/backend/custom_agent/base.py, products/signals/backend/custom_agent/persistence.py, products/signals/backend/custom_agent/PLAN.md
<details open><summary><h3>Security Review</h3></summary>
The custom-agent activity relies solely on launch-time AI-processing consent. A queued run can therefore process team data after consent has been revoked; consent should be revalidated immediately before activity execution.
</details>
<details><summary><h3>Important Files Changed</h3></summary>
| Filename | Overview |
|----------|----------|
| products/signals/backend/custom_agent/base.py | Implements the custom-agent lifecycle, validation, report finalization, and autostart integration; failure paths currently close sessions as completed. |
| products/signals/backend/temporal/custom_agent.py | Adds the shared Temporal launcher, workflow, and activity, but does not revalidate AI-processing consent at execution time. |
| products/signals/backend/custom_agent/persistence.py | Atomically creates READY reports and compatible artefacts but omits normal completion-event and Slack-notification orchestration. |
| products/signals/backend/auto_start.py | Extracts and generalizes implementation-task autostart while preserving the existing eligibility and assignee-selection behavior. |
| products/tasks/backend/services/custom_prompt_multi_turn_runner.py | Adds raw-response session methods while preserving existing structured parsing, polling, callbacks, and internal cleanup. |
| products/signals/backend/report_generation/select_repo.py | Generalizes repository selection to accept caller-rendered request context without regressing the existing Signals wrapper. |
| products/signals/backend/custom_agent/PLAN.md | Documents the new framework and its limitations, but contains a contradictory launcher re-export claim and explicitly omits automated coverage. |
</details>
<details open><summary><h3>Flowchart</h3></summary>
```mermaid
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[run_agent / arun_agent] --> B{AI processing approved?}
B -->|No| C[Reject launch]
B -->|Yes| D[Start signals-custom-agent workflow]
D --> E[run_custom_signal_agent_activity]
E --> F[Import and validate agent class]
F --> G[Resolve user and repository]
G --> H[Run CustomSignalAgent]
H --> I[LLM multi-turn session]
H --> J[Finalize report components]
J --> K[Persist READY SignalReport and artefacts]
K --> L[Optional implementation-task autostart]
K -. missing .-> M[Completion event]
K -. missing .-> N[Slack inbox notification]
```
</details>
<!-- greptile_other_comments_section -->
<sub>Reviews (1): Last reviewed commit: ["Update posthog/api/github\_callback/READM..."](https://github.com/drape-io/crbench-greptile-2026-09/commit/38458eb07bdefcbc979cf07077ccd2217dd07f20) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=64284758)</sub>
Mjolnir (Opus 5.5) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
| Defect | Severity | Found | Grade |
|---|---|---|---|
| d1 | high | ✓ | exact |
start()'s finally block calls session.end() with default status="completed" even when run()/finalization raises, so failed runs get recorded as completed, which is what end()'s docstring warns about. [grounded: tree products/signals/backend/custom_agent/base.py] [consensus 3/3: valid=3]
The diff adds an empty management/commands/__init__.py. That breaks a repo style rule but causes no behavior defect. [grounded: tree CLAUDE.md] [consensus 3/3: minor=3]
User.teams is a cached_property (posthog/models/user.py:307). The Team relations named 'teams' have related_query_name 'team' and point to Organization/Project, not User, so filtering User on teams__id raises FieldError. Callers swallow that error, so autostart never runs. The old code used user.teams.filter(...). [grounded: tree products/signals/backend/auto_start.py] [consensus 3/3: valid=3]
Same issue as finding 1: the session is ended with the default 'completed' status even when run() or finalization raises. [grounded: tree products/signals/backend/custom_agent/base.py] [consensus 3/3: valid=3]
Duplicate style note about the empty __init__.py; it does not claim any behavior defect. [grounded: tree CLAUDE.md] [consensus 3/3: minor=3]
raw comments (6)
[P1] `User.objects.filter(teams__id=...)` raises FieldError, so autostart never runs
`User.teams` is a `@cached_property` defined at `posthog/models/user.py:306-311`. It is not a model field or reverse relation; the `Team` FKs use `related_query_name="team"` on Organization/Project. So `teams__id=team_id` in this ORM filter raises `FieldError: Cannot resolve keyword 'teams'` whenever at least one suggested reviewer maps to an org user. Both callers wrap `maybe_autostart_implementation_task` in a broad `except Exception` that only logs. As a result, implementation-task autostart silently stops working for the existing agentic report pipeline and for the new custom agents. The old code checked membership per user with `user.teams.filter(id=team_id).exists()`. That version also respected project-level access control, which a plain join would lose.
[P2] Agent session is recorded as `completed` even when the run fails
The `finally` block always calls `self._session.end()`, and its default is `status="completed"`. If `run()` raises, or a later step fails, the sandbox `TaskRun` is still marked successful. Examples include a `CustomAgentValidationError` after retries, `MissingReportComponentError`, or a persistence error. `MultiTurnSession.end`'s docstring warns against this because it corrupts run-status metrics and triage. The existing pattern in `report_generation/research.py:628-631` ends with `status="failed", error=str(e)` (shielded) on exceptions, and this code should do the same.
[P3] New empty `__init__.py` added to management commands
This adds an empty `__init__.py`, which the project guidelines forbid ([CLAUDE.md:132](CLAUDE.md) "Python: do not create empty `__init__.py` files"). It isn't needed: `ingest_report_json` and the other commands in this directory were already discovered and loaded without it, because Django's `find_commands` and the import system work with the namespace package.
[P1] `teams__id` is not a queryable relation on `User`, so autostart now always fails
The refactor replaced the per-user `user.teams.filter(id=team_id).exists()` check with `User.objects.filter(..., teams__id=team_id)`. On `User`, `teams` is a `@cached_property` (posthog/models/user.py:306), not a model field or reverse relation. Nothing in posthog/ee/products declares a relation to `User` with a query name of `teams`. Django will raise `FieldError: Cannot resolve keyword 'teams'` when this queryset is evaluated. That happens every time at least one suggested reviewer maps to an org user. Both callers (the agentic report pipeline and the new custom agent) wrap autostart in a try/except that only logs. As a result, implementation tasks are silently never auto-started, which breaks the existing agentic autostart feature. The swap also drops the access-control filtering that the `teams` property applied.
[P2] Session is always ended as `completed`, even when `run()` or finalization raises
`start()` calls `self._session.end()` in its `finally` block using the default `status="completed"`. This still happens when `run()` raises, when `send()` exhausts its validation retries (`CustomAgentValidationError`), or when finalization raises `MissingReportComponentError`. The underlying research `TaskRun` is therefore recorded as successful. `MultiTurnSession.end`'s docstring explicitly says to pass `status="failed"` with an error in these cases so failed runs don't distort run-status metrics. The rest of this PR follows that rule in `start`/`start_raw`. Ending with `status="failed", error=str(exc)` on the exception path would keep the two consistent.
[P3] Unnecessary empty `__init__.py` added to the management commands directory
The `commands/` directory already served more than a dozen commands, including `ingest_report_json`, without an `__init__.py`. Adding this empty file doesn't change behavior, and it goes against the repo rule "Python: do not create empty `__init__.py` files" ([CLAUDE.md](CLAUDE.md), Code Style section).
Mjolnir (Sonnet 5.5) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
| Defect | Severity | Found | Grade |
|---|---|---|---|
| d1 | high | ✓ | exact |
base.py imports auto_start, and auto_start imports report_generation.research. research.py imports temporal.types (I checked the line), which runs temporal/__init__.py. That file now imports temporal/custom_agent.py and agentic/report.py, and both import back into modules that are only partly initialized at that point (base / auto_start). So importing custom_agent before the temporal package, as run_custom_agent_example does, hits an ImportError. The failure point may be report.py's auto_start import rather than custom_agent.py's base import, but either way the claimed failure happens, and this PR introduced the cycle. [grounded: diff] [consensus 3/3: valid=3]
raw comments (2)
[P2] Circular import when `custom_agent` is imported before the `temporal` package
`base.py` imports `auto_start`, which imports `report_generation.research`, which imports `products.signals.backend.temporal.types`. That import runs `temporal/__init__.py` in full, and `temporal/__init__.py` now imports `temporal/custom_agent.py`. `temporal/custom_agent.py` does `from products.signals.backend.custom_agent.base import AIDataProcessingNotApprovedError, CustomSignalAgent`. If `custom_agent.base` is the first module loaded, it is still partially initialized at that point and the names don't exist yet, so this raises `ImportError`. This hits any entry point that imports `products.signals.backend.custom_agent` before the temporal package, such as the new `run_custom_agent_example` command or a third-party agent module. The Temporal worker imports `temporal/__init__` first, so it is unaffected. The `custom_agent/__init__.py` docstring says the package is "Temporal-free", but `base.py` also imports `temporal.agentic`, so it isn't. The function-level `from ...base import CustomSignalAgent` in `loader.py` (`import_agent_class`) looks like a workaround for the same cycle. It has no `# noqa: PLC0415` and breaks the repo's module-level-import rule ([CLAUDE.md](CLAUDE.md) Code Style, "Python imports"). Fix the module structure rather than rely on import order.
[P1] Invalid `teams__id` lookup on User breaks autostart
The new `User.objects.filter(..., teams__id=team_id)` replaces the old `user.teams.filter(id=team_id).exists()` check. `User.teams` is a `cached_property` (posthog/models/user.py:306-311) that returns an RBAC-aware `Team` queryset. It is not a model relation, and no `related_name="teams"` on a Team FK points to User. The ORM will therefore raise `FieldError: Cannot resolve keyword 'teams' into field` whenever `_resolve_autostart_assignee` reaches this query. That happens for any immediately-actionable report with a priority and at least one reviewer who maps to an org user. In the agentic report pipeline the exception is swallowed by the existing try/except around the autostart call, so implementation tasks silently stop auto-starting. Custom agents hit the same failure. The change also drops the project-level access control that `User.teams` applied. Keep the per-user `user.teams.filter(id=team_id).exists()` check, or filter on a real relation such as organization membership plus the team's access rules
qodo · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
| Defect | Severity | Found | Grade |
|---|---|---|---|
| d1 | high | ✓ | exact |
The new activity loads the team with its organization but never checks is_ai_data_processing_approved before agent.start(); the consent gate lives only in arun_agent. PLAN.md admits this ('not re-checked inside the activity'), but documenting the gap does not close it. If approval is revoked after launch, or a workflow is started directly, team data still goes to the LLM. [grounded: tree products/signals/backend/temporal/custom_agent.py] [consensus 3/3: valid=3]
start()'s finally block calls _session.end() with no arguments, and MultiTurnSession.end defaults to status="completed". So when run(), validation, or persistence raises after the session exists, the TaskRun is recorded as completed. MultiTurnSession.start/start_raw instead end failed sessions with status="failed", so this is inconsistent with the existing pattern. [grounded: diff] [consensus 3/3: valid=3]
raw comments (5)
<img src="https://img.shields.io/badge/High-634FD1?style=flat-square" height="20px" alt="Action required">
1\. Unapproved agents still process team data <code>🐞 Bug</code> <code>⛨ Security</code>
<pre>
<b><i>run_custom_signal_agent_activity</i></b> loads the team and starts the agent without checking the
organization's current AI data-processing approval. When a workflow bypasses the convenience
launcher or approval is revoked between launch and activity execution, prompts and repository
context still reach the sandbox and language model.
</pre>
<details>
<summary><strong>Agent Prompt</strong></summary>
```
## Issue description
The Temporal activity executes custom agents without verifying the organization's current AI data-processing approval, allowing queued or directly started workflows to process team data after approval is absent.
## Fix Focus Areas
- products/signals/backend/temporal/custom_agent.py[170-177]
## Recommended Fix
After loading the team and organization, verify `is_ai_data_processing_approved` before constructing or starting the agent. Raise `AIDataProcessingNotApprovedError` when approval is absent so every execution path enforces the gate at processing time.
```
<code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code>
</details>
<img src="https://img.shields.io/badge/High-634FD1?style=flat-square" height="20px" alt="Action required">
2\. Autostart skips every eligible reviewer <code>🐞 Bug</code> <code>≡ Correctness</code>
<pre>
<b><i>_resolve_autostart_assignee</i></b> filters <b><i>User.objects</i></b> with <b><i>teams__id</i></b>, but <b><i>teams</i></b> is a Python
cached property rather than a queryable ORM relation. Whenever a suggested GitHub reviewer resolves
to a user with an autonomy config, Django raises a <b><i>FieldError</i></b>; the caller catches it so report
persistence completes without starting a task.
</pre>
<details>
<summary><strong>Agent Prompt</strong></summary>
```
## Issue description
The assignee lookup uses `teams__id` on `User.objects`, but `User.teams` is an instance-level cached property and cannot be used in ORM filters. This raises `FieldError` before a qualifying reviewer can be selected.
## Fix Focus Areas
- products/signals/backend/auto_start.py[93-101]
## Recommended Fix
Replace the invalid `teams__id=team_id` filter with a queryable set of users that have access to the team. Preserve access-control semantics by deriving the eligible users from `Team.all_users_with_access()` (or its equivalent query), intersecting it with `candidate_user_ids`, and retaining the eager `signal_autonomy_config` join.
```
<code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code>
</details>
<img src="https://img.shields.io/badge/High-634FD1?style=flat-square" height="20px" alt="Action required">
3\. Custom agent failures appear completed <code>🐞 Bug</code> <code>☼ Reliability</code>
<pre>
<b><i>CustomSignalAgent.start</i></b> unconditionally invokes <b><i>self._session.end()</i></b> with its default completed
status in <b><i>finally</i></b>, even when <b><i>run()</i></b>, response validation, report finalization, persistence, or
resolver work raises. Invalid model responses, exhausted validation retries, follow-up failures, and
activity cancellation can escape after <b><i>send()</i></b> or <b><i>start_raw()</i></b> has stored the session, causing the
Temporal activity or workflow to fail while the associated TaskRun remains marked successful.
</pre>
<details>
<summary><strong>Agent Prompt</strong></summary>
```
## Issue description
The custom-agent lifecycle closes every established multi-turn sandbox session as completed, including when agent execution, response validation, report finalization, persistence, resolver work, or cancellation fails. This leaves the associated TaskRun marked successful even though the Temporal activity or workflow fails.
## Fix Focus Areas
- products/signals/backend/custom_agent/base.py[361-376]
- products/tasks/backend/services/custom_prompt_multi_turn_runner.py[252-263]
## Recommended Fix
Track whether `start()` and its final persistence work completed successfully. Call `session.end()` without failure arguments only on success; when execution raises or is cancelled, call `session.end(status="failed", error=str(error))` under `asyncio.shield` as needed for cancellation-sensitive cleanup, then re-raise the original failure, matching the established research-session cleanup behavior.
```
<code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code>
</details>
<h3>PR Summary by Qodo</h3>
Add custom Signals agent framework and Temporal workflow
<code>✨ Enhancement</code> <code>📝 Documentation</code> <code>🧪 Tests</code> <code>🕐 40+ Minutes</code>
<img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider">
<details>
<summary>AI Description</summary>
<dl>
<dd>
<br/>
><pre>
>• Adds reusable custom Signals agent SDK with Temporal execution and READY report persistence.
>• Shares repository selection and autonomous implementation-task creation across Signals pipelines.
>• Provides examples, local smoke tooling, integration documentation, and workflow registration.
></pre>
</dd>
</dl>
</details>
<details>
<summary>Diagram</summary>
<dl>
<dd>
<br/>
```mermaid
sequenceDiagram
actor Caller
participant Launcher as Temporal Launcher
participant Workflow as Custom Workflow
participant Activity as Agent Activity
participant Agent as Agent SDK
participant Sandbox as Multi-turn Sandbox
participant Store as Signals Store
participant Autostart as Task Autostart
Caller->>Launcher: Start agent
Launcher->>Workflow: Validated input
Workflow->>Activity: Execute once
Activity->>Agent: Import and start
Agent->>Sandbox: Select repo and research
Sandbox-->>Agent: Raw responses
Agent->>Store: Persist READY reports
Agent->>Autostart: Start eligible task
```
</dd>
</dl>
</details>
<details>
<summary>High-Level Assessment</summary>
<dl>
<dd>
<br/>
>The following are alternative approaches to this PR:
<details>
<summary><b>1. Dedicated workflow per agent</b></summary>
<dl>
<dd>
- ➕ Provides agent-specific retry policies, timeouts, and operational isolation.
- ➕ Makes each agent's Temporal behavior explicit.
- ➖ Requires repeated workflow and activity registration for every one-off agent.
- ➖ Duplicates sandbox lifecycle, persistence, and report-finalization code.
</dd>
</dl>
</details>
<details>
<summary><b>2. Extend the existing agentic report pipeline</b></summary>
<dl>
<dd>
- ➕ Reuses the established report-generation workflow and its operational coverage.
- ➕ Avoids introducing a parallel agent abstraction.
- ➖ Couples one-off agents to signal grouping and research pipeline assumptions.
- ➖ Makes multi-report and repository-free agents harder to express cleanly.
</dd>
</dl>
</details>
>**Recommendation:** Keep the shared Temporal workflow and Temporal-free SDK: it offers the best balance between reusable infrastructure and lightweight agent implementations. Before production rollout, add focused automated coverage for identifier loading, repository modes, multi-report finalization, persistence, validation retries, and autostart handoff.
</dd>
</dl>
</details>
<details>
<summary> Files changed (23) <code> +2087 / -192 </code> </summary>
<dl>
<dd>
<br/>
<details>
<summary>Enhancement (9) <code> +1277 / -11 </code></summary>
<dl>
<dd>
<details>
<summary>__init__.py<code>Expose the Temporal-free custom agent API</code> <code>+32/-0</code></summary>
<br/>
>Expose the Temporal-free custom agent API
>
><pre>
>• Defines the public SDK exports while intentionally keeping Temporal launchers outside the package to avoid circular dependencies.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-fdf6a35e85cf5014619f9ce536ac26b1fe9703c7d4584c9a1b433f9a74dbb8f1'>products/signals/backend/custom_agent/__init__.py</a>
</details>
<details>
<summary>base.py<code>Implement the CustomSignalAgent SDK</code> <code>+627/-0</code></summary>
<br/>
>Implement the CustomSignalAgent SDK
>
><pre>
>• Adds the subclass lifecycle, multi-turn structured prompting, validation retries, repository modes, component resolvers, multi-report support, persistence integration, cleanup, and best-effort autostart.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-69eae99ad68ca52225def7d21af716d99f56591ff04510e8ee08cee422fbc24e'>products/signals/backend/custom_agent/base.py</a>
</details>
<details>
<summary>loader.py<code>Validate dynamically imported agent classes</code> <code>+59/-0</code></summary>
<br/>
>Validate dynamically imported agent classes
>
><pre>
>• Adds dotted-path importing, subclass checks, top-level class enforcement, and workflow-to-class identity validation.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-89935080aa578dda9161598d9399e942f05e707b007d9490f8d3237514acaed8'>products/signals/backend/custom_agent/loader.py</a>
</details>
<details>
<summary>persistence.py<code>Persist custom-agent reports transactionally</code> <code>+79/-0</code></summary>
<br/>
>Persist custom-agent reports transactionally
>
><pre>
>• Creates READY Signal reports, compatible artefacts, and optional research-task relationships in one database transaction.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-2f865d4aaf086be03e878f089095b15a64c576c9ffc46bbedad114536dd33660'>products/signals/backend/custom_agent/persistence.py</a>
</details>
<details>
<summary>schemas.py<code>Define custom-agent identities and workflow schemas</code> <code>+105/-0</code></summary>
<br/>
>Define custom-agent identities and workflow schemas
>
><pre>
>• Adds validated identifiers, normalized assignees, workflow input/output dataclasses, run handles, and final report structures.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-705816bab304a6987386b5de1340418d0aba9a4f1d0ad543d79fc9620476a126'>products/signals/backend/custom_agent/schemas.py</a>
</details>
<details>
<summary>run_custom_agent_example.py<code>Add local custom-agent smoke command</code> <code>+117/-0</code></summary>
<br/>
>Add local custom-agent smoke command
>
><pre>
>• Adds DEBUG-only execution for example agents through Temporal or in-process, with prompt and repository mode overrides.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-da3b50cb93a56709fecfc9a3296e71c98543d310f82ce80171aeb6c0a6657200'>products/signals/backend/management/commands/run_custom_agent_example.py</a>
</details>
<details>
<summary>custom_agent.py<code>Add shared custom-agent Temporal workflow</code> <code>+194/-0</code></summary>
<br/>
>Add shared custom-agent Temporal workflow
>
><pre>
>• Adds synchronous and asynchronous launchers, consent and identity validation, deterministic workflow IDs, single-attempt execution, heartbeat handling, dynamic class loading, and workflow output collection.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-92625f4d36fff2116332175205c1c78dde2203539b3ce4f576910759cf685673'>products/signals/backend/temporal/custom_agent.py</a>
</details>
<details>
<summary>agent.py<code>Support caller-specific repository-selection step names</code> <code>+2/-1</code></summary>
<br/>
>Support caller-specific repository-selection step names
>
><pre>
>• Allows repository-selection callers to assign distinct task step names while preserving the existing default.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-20bd0bda5c5b5a06dfaae74e009b19a098734accdb1316355c0340761aa30781'>products/tasks/backend/repo_selection/agent.py</a>
</details>
<details>
<summary>custom_prompt_multi_turn_runner.py<code>Expose raw multi-turn agent responses</code> <code>+62/-10</code></summary>
<br/>
>Expose raw multi-turn agent responses
>
><pre>
>• Splits session start and follow-up operations into raw and structured variants, preserving failure cleanup while allowing callers to implement their own validation retries.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-6078db5afad80000627b2459392a9ee6dd6ee0648a9605139fa17d190eefee2a'>products/tasks/backend/services/custom_prompt_multi_turn_runner.py</a>
</details>
</dd>
</dl>
</details>
<details>
<summary>Refactor (3) <code> +235 / -170 </code></summary>
<dl>
<dd>
<details>
<summary>auto_start.py<code>Extract shared implementation-task autostart service</code> <code>+185/-0</code></summary>
<br/>
>Extract shared implementation-task autostart service
>
><pre>
>• Centralizes eligibility checks, reviewer assignment, priority thresholds, task creation, and report-task linking. The service is reusable by standard Signals reports and custom agents.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-a83f1b1e25c7a2c6c69c9d48ca74bef179ad0670f0daca13dbc8b6862e64a0db'>products/signals/backend/auto_start.py</a>
</details>
<details>
<summary>select_repo.py<code>Generalize Signals repository selection</code> <code>+41/-10</code></summary>
<br/>
>Generalize Signals repository selection
>
><pre>
>• Introduces request-based team repository selection with configurable step names while retaining the signal-specific wrapper and null-result behavior.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-575f7a6b022598cf33c3c058196bbe075194c480411d6324f47f76250bd0d07b'>products/signals/backend/report_generation/select_repo.py</a>
</details>
<details>
<summary>report.py<code>Adopt the shared report autostart service</code> <code>+9/-160</code></summary>
<br/>
>Adopt the shared report autostart service
>
><pre>
>• Removes local autostart implementation details and delegates eligible report handling to the shared backend service.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-91c0313c44877e52de26bd9e98926ecd77861da55a2894a38e2f4c900ec83565'>products/signals/backend/temporal/agentic/report.py</a>
</details>
</dd>
</dl>
</details>
<details>
<summary>Tests (1) <code> +2 / -0 </code></summary>
<dl>
<dd>
<details>
<summary>test_module_integrity.py<code>Track custom Signals workflow registration</code> <code>+2/-0</code></summary>
<br/>
>Track custom Signals workflow registration
>
><pre>
>• Adds the custom-agent workflow and activity to the Signals Temporal integrity expectations.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-fdabd5583c388d1284a8d50263222fecae2309965b4a924d803c7c54756b6b64'>posthog/temporal/tests/ai/test_module_integrity.py</a>
</details>
</dd>
</dl>
</details>
<details>
<summary>Documentation (8) <code> +570 / -11 </code></summary>
<dl>
<dd>
<details>
<summary>README.md<code>Document local GitHub App integration setup</code> <code>+117/-0</code></summary>
<br/>
>Document local GitHub App integration setup
>
><pre>
>• Adds setup guidance for team and personal GitHub integrations, required application settings, environment variables, callback routes, permissions, and troubleshooting.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-62d2591e7692d512f93511da3cd6c8be4858a5b5e35127b38b6481f55153a05b'>posthog/api/github_callback/README.md</a>
</details>
<details>
<summary>ARCHITECTURE.md<code>Document shared Signals autostart flow</code> <code>+7/-7</code></summary>
<br/>
>Document shared Signals autostart flow
>
><pre>
>• Updates architecture references to the extracted autostart module and documents its use by both agentic reports and custom agents.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-135165b90f4c75c07760329748e5d25ba929793262132f27692575069fc0e6a3'>products/signals/ARCHITECTURE.md</a>
</details>
<details>
<summary>PLAN.md<code>Document the custom agent framework</code> <code>+282/-0</code></summary>
<br/>
>Document the custom agent framework
>
><pre>
>• Provides comprehensive design, API, lifecycle, persistence, Temporal, repository-selection, autostart, limitations, and extension guidance for custom Signals agents.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-a0e7446062a7c14d6fa38a04346e58ccd2cfc1246ffdbf9c1b0f7701f11e5c62'>products/signals/backend/custom_agent/PLAN.md</a>
</details>
<details>
<summary>__init__.py<code>Describe reference custom agents</code> <code>+10/-0</code></summary>
<br/>
>Describe reference custom agents
>
><pre>
>• Documents the minimal repository-free example and the repository-research example, including how to run them locally.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-0f210e3bb216b596103987e97faf40e035d0fcad15a4feb0d49c9592c21aa717'>products/signals/backend/custom_agent/examples/__init__.py</a>
</details>
<details>
<summary>cookie_poem_agent.py<code>Add minimal custom agent example</code> <code>+62/-0</code></summary>
<br/>
>Add minimal custom agent example
>
><pre>
>• Adds a repository-free example that requests structured poem output and explicitly registers every report component.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-f02b579ce8a76b4784483df06be9b010cdf22cf7bca6b8591ae12dfe257225f8'>products/signals/backend/custom_agent/examples/cookie_poem_agent.py</a>
</details>
<details>
<summary>cursed_comment_agent.py<code>Add repository research agent example</code> <code>+88/-0</code></summary>
<br/>
>Add repository research agent example
>
><pre>
>• Adds a realistic agent that searches a repository for risky comments and delegates actionability, priority, and assignee resolution to the SDK defaults.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-c1971a8bfef600be11d8ee3364fabb984f8dff95b0f192af0a9d462941121f26'>products/signals/backend/custom_agent/examples/cursed_comment_agent.py</a>
</details>
<details>
<summary>AGENTS.md<code>Update autostart management documentation</code> <code>+1/-1</code></summary>
<br/>
>Update autostart management documentation
>
><pre>
>• Points report-ingestion guidance at the new shared implementation-task autostart service.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-d8f9557f9d3cea30218dd0cbb3f865ddcc2320d8420b38b7b8ea94175291369a'>products/signals/backend/management/AGENTS.md</a>
</details>
<details>
<summary>ingest_report_json.py<code>Update ingest command autostart references</code> <code>+3/-3</code></summary>
<br/>
>Update ingest command autostart references
>
><pre>
>• Revises module documentation to identify the extracted shared autostart service used during fixture ingestion.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-0613935be2e896b4bb634210a7c83f734e9854b9fdae71d3e1e47b3bf5d876c4'>products/signals/backend/management/commands/ingest_report_json.py</a>
</details>
</dd>
</dl>
</details>
<details>
<summary>Other (2) <code> +3 / -0 </code></summary>
<dl>
<dd>
<details>
<summary>__init__.py<code>Add custom-agent command package marker</code> <code>+0/-0</code></summary>
<br/>
>Add custom-agent command package marker
>
><pre>
>• Includes the management command package initializer needed alongside the new example command, without adding runtime behavior.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-55fda36f74e46a4088dd4efd651ff164dcd58878b95747cce718c8dd148e102d'>products/signals/backend/management/commands/__init__.py</a>
</details>
<details>
<summary>__init__.py<code>Register custom-agent Temporal components</code> <code>+3/-0</code></summary>
<br/>
>Register custom-agent Temporal components
>
><pre>
>• Adds the custom-agent workflow and activity to the Signals worker registration lists.
></pre>
>
><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-8baedb06389394c6483fc8b494e48635692980818c3d20b767b40e641f0b7a56'>products/signals/backend/temporal/__init__.py</a>
</details>
</dd>
</dl>
</details>
</dd>
</dl>
</details>
<h3>Code Review by Qodo</h3>
<code>🐞 Bugs (3)</code> <code>📘 Rule violations (0)</code> <code>📜 Skill insights (0)</code>
<img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider">
<br/>
<img src="https://img.shields.io/badge/High-634FD1?style=flat-square" height="20px" alt="Action required">
<details>
<summary> 1. Unapproved agents still process team data <code>🐞 Bug</code> <code>⛨ Security</code></summary>
<br/>
> <details open>
><summary>Description</summary>
><br/>
>
><pre>
><b><i>run_custom_signal_agent_activity</i></b> loads the team and starts the agent without checking the
>organization's current AI data-processing approval. When a workflow bypasses the convenience
>launcher or approval is revoked between launch and activity execution, prompts and repository
>context still reach the sandbox and language model.
></pre>
></details>
> <details>
><summary>Code</summary>
><br/>
>
><code>[products/signals/backend/temporal/custom_agent.py[R170-175]](https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-92625f4d36fff2116332175205c1c78dde2203539b3ce4f576910759cf685673R170-R175)</code>
>
>```diff
>+ team = await Team.objects.select_related("organization").aget(pk=inputs.team_id)
>+ agent = agent_class(
>+ team=team,
>+ initial_prompt=inputs.initial_prompt,
>+ repository=inputs.repository,
>+ model=inputs.model,
>```
></details>
> <details>
><summary>Evidence</summary>
><br/>
>
><pre>
>The launcher performs a point-in-time approval check, but the independently registered activity
>proceeds directly from loading the organization to executing <b><i>agent.start()</i></b>. The agent then creates
>trusted-network sandbox sessions that receive the initial request and repository context.
></pre>
>
> <code>[products/signals/backend/temporal/custom_agent.py[95-106]](https://github.com/drape-io/crbench-qodo-2026-09/blob/f4160e25c4dee52e98b14f7454ab0f2e0aee707e/products/signals/backend/temporal/custom_agent.py/#L95-L106)</code>
> <code>[products/signals/backend/temporal/custom_agent.py[156-177]](https://github.com/drape-io/crbench-qodo-2026-09/blob/f4160e25c4dee52e98b14f7454ab0f2e0aee707e/products/signals/backend/temporal/custom_agent.py/#L156-L177)</code>
> <code>[products/signals/backend/custom_agent/base.py[565-575]](https://github.com/drape-io/crbench-qodo-2026-09/blob/f4160e25c4dee52e98b14f7454ab0f2e0aee707e/products/signals/backend/custom_agent/base.py/#L565-L575)</code>
> <code>[products/signals/backend/custom_agent/base.py[595-619]](https://github.com/drape-io/crbench-qodo-2026-09/blob/f4160e25c4dee52e98b14f7454ab0f2e0aee707e/products/signals/backend/custom_agent/base.py/#L595-L619)</code>
></details>
> <details>
><summary>Agent prompt</summary>
><br/>
>
>```
>The issue below was found during a code review. Follow the provided context and guidance below and implement a solution
>
>## Issue description
>The Temporal activity executes custom agents without verifying the organization's current AI data-processing approval, allowing queued or directly started workflows to process team data after approval is absent.
>
>## Fix Focus Areas
>- products/signals/backend/temporal/custom_agent.py[170-177]
>
>## Recommended Fix
>After loading the team and organization, verify `is_ai_data_processing_approved` before constructing or starting the agent. Raise `AIDataProcessingNotApprovedError` when approval is absent so every execution path enforces the gate at processing time.
>```
> <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code>
></details>
<hr/>
</details>
<details>
<summary> 2. Autostart skips every eligible reviewer <code>🐞 Bug</code> <code>≡ Correctness</code></summary>
<br/>
> <details open>
><summary>Description</summary>
><br/>
>
><pre>
><b><i>_resolve_autostart_assignee</i></b> filters <b><i>User.objects</i></b> with <b><i>teams__id</i></b>, but <b><i>teams</i></b> is a Python
>cached property rather than a queryable ORM relation. Whenever a suggested GitHub reviewer resolves
>to a user with an autonomy config, Django raises a <b><i>FieldError</i></b>; the caller catches it so report
>persistence completes without starting a task.
></pre>
></details>
> <details>
><summary>Code</summary>
><br/>
>
><code>[products/signals/backend/auto_start.py[R96-100]](https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-a83f1b1e25c7a2c6c69c9d48ca74bef179ad0670f0daca13dbc8b6862e64a0dbR96-R100)</code>
>
>```diff
>+ for u in User.objects.filter(
>+ id__in=candidate_user_ids,
>+ signal_autonomy_config__isnull=False,
>+ teams__id=team_id,
>+ ).select_related("signal_autonomy_config")
>```
></details>
> <details>
><summary>Evidence</summary>
><br/>
>
><pre>
>The changed query applies a nonexistent ORM relation. <b><i>User.teams</i></b> is implemented as a cached Python
>property, while both agentic and custom-agent callers catch failures from this shared function,
>converting the exception into a missing implementation task.
></pre>
>
> <code>[products/signals/backend/auto_start.py[74-113]](https://github.com/drape-io/crbench-qodo-2026-09/blob/f4160e25c4dee52e98b14f7454ab0f2e0aee707e/products/signals/backend/auto_start.py/#L74-L113)</code>
> <code>[posthog/models/user.py[302-323]](https://github.com/drape-io/crbench-qodo-2026-09/blob/f4160e25c4dee52e98b14f7454ab0f2e0aee707e/posthog/models/user.py/#L302-L323)</code>
> <code>[products/signals/backend/temporal/agentic/report.py[248-267]](https://github.com/drape-io/crbench-qodo-2026-09/blob/f4160e25c4dee52e98b14f7454ab0f2e0aee707e/products/signals/backend/temporal/agentic/report.py/#L248-L267)</code>
> <code>[products/signals/backend/custom_agent/base.py[486-504]](https://github.com/drape-io/crbench-qodo-2026-09/blob/f4160e25c4dee52e98b14f7454ab0f2e0aee707e/products/signals/backend/custom_agent/base.py/#L486-L504)</code>
></details>
> <details>
><summary>Agent prompt</summary>
><br/>
>
>```
>The issue below was found during a code review. Follow the provided context and guidance below and implement a solution
>
>## Issue description
>The assignee lookup uses `teams__id` on `User.objects`, but `User.teams` is an instance-level cached property and cannot be used in ORM filters. This raises `FieldError` before a qualifying reviewer can be selected.
>
>## Fix Focus Areas
>- products/signals/backend/auto_start.py[93-101]
>
>## Recommended Fix
>Replace the invalid `teams__id=team_id` filter with a queryable set of users that have access to the team. Preserve access-control semantics by deriving the eligible users from `Team.all_users_with_access()` (or its equivalent query), intersecting it with `candidate_user_ids`, and retaining the eager `signal_autonomy_config` join.
>```
> <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code>
></details>
<hr/>
</details>
<details>
<summary> 3. Custom agent failures appear completed <code>🐞 Bug</code> <code>☼ Reliability</code></summary>
<br/>
> <details open>
><summary>Description</summary>
><br/>
>
><pre>
><b><i>CustomSignalAgent.start</i></b> unconditionally invokes <b><i>self._session.end()</i></b> with its default completed
>status in <b><i>finally</i></b>, even when <b><i>run()</i></b>, response validation, report finalization, persistence, or
>resolver work raises. Invalid model responses, exhausted validation retries, follow-up failures, and
>activity cancellation can escape after <b><i>send()</i></b> or <b><i>start_raw()</i></b> has stored the session, causing the
>Temporal activity or workflow to fail while the associated TaskRun remains marked successful.
></pre>
></details>
> <details>
><summary>Code</summary>
><br/>
>
><code>[products/signals/backend/custom_agent/base.py[R371-376]](https://github.com/drape-io/crbench-qodo-2026-09/pull/63/files#diff-69eae99ad68ca52225def7d21af716d99f56591ff04510e8ee08cee422fbc24eR371-R376)</code>
>
>```diff
>+ finally:
>+ if self._session is not None:
>+ try:
>+ await self._session.end()
>+ except Exception:
>+ logger.warning("custom signal agent session cleanup failed", exc_info=True)
>```
></details>
> <details>
><summary>Evidence</summary>
><br/>
>
><pre>
>Response parsing and validation occur after <b><i>start_raw()</i></b> has returned and stored the session, so
>subsequent errors flow through <b><i>start()</i></b>'s unconditional cleanup. <b><i>MultiTurnSession.end()</i></b> defaults
>to <b><i>completed</i></b> and explicitly documents that failures must pass <b><i>status="failed"</i></b>; the existing
>Signals research lifecycle follows this contract, while the new custom-agent lifecycle does not.
></pre>
>
> <code>[products/signals/backend/custom_agent/base.py[361-376]](https://github.com/drape-io/crbench-qodo-2026-09/blob/f4160e25c4dee52e98b14f7454ab0f2e0aee707e/products/signals/backend/custom_agent/base.py/#L361-L376)</code>
> <code>[products/tasks/backend/services/custom_prompt_multi_turn_runner.py[252-265]](https://github.com/drape-io/crbench-qodo-2026-09/blob/f4160e25c4dee52e98b14f7454ab0f2e0aee707e/products/tasks/backend/services/custom_prompt_multi_turn_runner.py/#L252-L265)</code>
> <code>[products/signals/backend/report_generation/research.py[541-632]](https://github.com/drape-io/crbench-qodo-2026-09/blob/f4160e25c4dee52e98b14f7454ab0f2e0aee707e/products/signals/backend/report_generation/research.py/#L541-L632)</code>
> <code>[products/signals/backend/custom_agent/base.py[230-250]](https://github.com/drape-io/crbench-qodo-2026-09/blob/f4160e25c4dee52e98b14f7454ab0f2e0aee707e/products/signals/backend/custom_agent/base.py/#L230-L250)</code>
> <code>[products/signals/backend/custom_agent/base.py[613-622]](https://github.com/drape-io/crbench-qodo-2026-09/blob/f4160e25c4dee52e98b14f7454ab0f2e0aee707e/products/signals/backend/custom_agent/base.py/#L613-L622)</code>
> <code>[products/tasks/backend/services/custom_prompt_multi_turn_runner.py[252-263]](https://github.com/drape-io/crbench-qodo-2026-09/blob/f4160e25c4dee52e98b14f7454ab0f2e0aee707e/products/tasks/backend/services/custom_prompt_multi_turn_runner.py/#L252-L263)</code>
></details>
> <details>
><summary>Agent prompt</summary>
><br/>
>
>```
>The issue below was found during a code review. Follow the provided context and guidance below and implement a solution
>
>## Issue description
>The custom-agent lifecycle closes every established multi-turn sandbox session as completed, including when agent execution, response validation, report finalization, persistence, resolver work, or cancellation fails. This leaves the associated TaskRun marked successful even though the Temporal activity or workflow fails.
>
>## Fix Focus Areas
>- products/signals/backend/custom_agent/base.py[361-376]
>- products/tasks/backend/services/custom_prompt_multi_turn_runner.py[252-263]
>
>## Recommended Fix
>Track whether `start()` and its final persistence work completed successfully. Call `session.end()` without failure arguments only on success; when execution raises or is cancelled, call `session.end(status="failed", error=str(error))` under `asyncio.shield` as needed for cancellation-sensitive cleanup, then re-raise the original failure, matching the established research-session cleanup behavior.
>```
> <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code>
></details>
<hr/>
</details>
<img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider">
<!-- qodo-context:start -->
<details><summary><strong>Context sources</strong></summary>
<div>Review mode: <code>🧠 Deep</code>: This adds a substantial custom-agent SDK and Temporal workflow, refactors shared repository/autostart paths, and changes multi-turn session behavior across many independent runtime code paths with no automated tests.</div>
<!-- qodo-context:end -->
</details>
<img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider">
<!-- qodo-daily-tip:start -->
<details>
<summary><strong>Tip of the day</strong></summary>
<br/>
<pre>💡 Did you know, you can reply 'qodo' on any finding to push back, ask questions, or dig deeper</pre>
<a href="https://docs.qodo.ai/tips-and-tricks">More tips ↗</a> | <a href="https://app.qodo.ai/configurations?tab=display-preferences">Customize Qodo ↗</a> | <a href="https://docs.qodo.ai">Qodo docs ↗</a>
</details>
<img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider">
<!-- qodo-daily-tip:end -->
<!-- https://github.com/drape-io/crbench-qodo-2026-09/commit/f4160e25c4dee52e98b14f7454ab0f2e0aee707e -->
<a href="https://www.qodo.ai"><img src="https://www.qodo.ai/wp-content/uploads/2025/03/qodo-logo.svg" width="80" alt="Qodo Logo"></a>