crbench//

← all cases

Narrowing a shared visibility flag hides sync-method/field/primary-key columns for non-SQL sources

Tier 2 hard typescript · cross-file · real merged PR from PostHog/posthog #60155

Ground-truth defects

d1 · correctness · high
The change narrows shouldShowSyncColumns with `&& selectedConnector.supportsColumnSelection`, correct for column-selection but the same flag also gates the Sync-method, Sync-field, and Primary-key columns. supportsColumnSelection is false for all non-SQL sources (Stripe, Hubspot, REST), so those three controls silently disappeared for every non-SQL source during table setup.

The change under review

diff (truncated — full diff)
diff --git a/frontend/src/queries/schema.json b/frontend/src/queries/schema.json
index 973204ab2f12..7e57b2bba0b2 100644
--- a/frontend/src/queries/schema.json
+++ b/frontend/src/queries/schema.json
@@ -37974,6 +37974,11 @@
                     },
                     "type": "array"
                 },
+                "supportsColumnSelection": {
+                    "default": false,
+                    "description": "Whether the source-creation wizard should expose the per-column projection picker. Mirrors `SQLSource.supports_column_selection` so the wizard doesn't show a picker for drivers that ignore `enabled_columns` at sync time.",
+                    "type": "boolean"
+                },
                 "unreleasedSource": {
                     "type": "boolean"
                 },
diff --git a/frontend/src/queries/schema/schema-general.ts b/frontend/src/queries/schema/schema-general.ts
index 79cb798d4e9d..b0130884b09b 100644
--- a/frontend/src/queries/schema/schema-general.ts
+++ b/frontend/src/queries/schema/schema-general.ts
@@ -5870,6 +5870,14 @@ export interface SourceConfig {
      * @default false
      */
     featured?: boolean
+
+    /**
+     * Whether the source-creation wizard should expose the per-column projection picker.
+     * Mirrors `SQLSource.supports_column_selection` so the wizard doesn't show a picker
+     * for drivers that ignore `enabled_columns` at sync time.
+     * @default false
+     */
+    supportsColumnSelection?: boolean
 }
 
 export const externalDataSources = [
diff --git a/posthog/schema.py b/posthog/schema.py
index 3382b420b33e..26e41804ddb1 100644
--- a/posthog/schema.py
+++ b/posthog/schema.py
@@ -26936,6 +26936,15 @@ class SourceConfig(BaseModel):
         default=[],
         description="Tables to suggest enabling, with optional tooltip explaining why",
     )
+    supportsColumnSelection: bool | None = Field(
+        default=False,
+        description=(
+            "Whether the source-creation wizard should expose the per-column projection"
+            " picker. Mirrors `SQLSource.supports_column_selection` so the wizard"
+            " doesn't show a picker for drivers that ignore `enabled_columns` at sync"
+            " time."
+        ),
+    )
     unreleasedSource: bool | None = None
     webhookFields: (
         list[
diff --git a/posthog/temporal/data_imports/sources/bigquery/bigquery.py b/posthog/temporal/data_imports/sources/bigquery/bigquery.py
index cae749a0088c..fe0891c2bc75 100644
--- a/posthog/temporal/data_imports/sources/bigquery/bigquery.py
+++ b/posthog/temporal/data_imports/sources/bigquery/bigquery.py
@@ -39,8 +39,11 @@
 from posthog.temporal.data_imports.pipelines.pipeline.typings import SourceInputs, SourceResponse
 from posthog.temporal.data_imports.pipelines.pipeline.utils import DEFAULT_PARTITION_TARGET_SIZE_IN_BYTES
 from posthog.temporal.data_imports.sources.common.http import DEFAULT_RETRY, TrackedHTTPAdapter
+from posthog.temporal.data_imports.sources.common.sql import compute_projected_columns
+from posthog.temporal.data_imports.sources.common.sql.identifiers import BacktickIdentifierQuoter
 from posthog.temporal.data_imports.sources.common.sql.implementation import SQLSourceImplementation
 from posthog.temporal.data_imports.sources.common.sql.incremental import IncrementalFieldFilter
+from posthog.temporal.data_imports.sources.common.sql.projection import format_projected_select_clause
 from posthog.temporal.data_imports.sources.generated_configs import BigQuerySourceConfig
 
 from products.data_warehouse.backend.types import IncrementalFieldType, PartitionSettings
@@ -369,13 +372,30 @@ def _get_rows_to_sync(
         return 0
 
 
+_BQ_QUOTER = BacktickIdentifierQuoter()
+
+
+def _bq_select_clause(
+    enabled_columns: list[str] | None,
+    primary_keys: list[str] | None,
+    incremental_field: str | None,
+) -> str:
+    """BigQuery SELECT-list with backtick quoting and identifier allowlist."""
+    projected = compute_projected_columns(enabled_columns, primary_keys, incremental_field)
+    return format_projected_select_clause(projected, _BQ_QUOTER)
+
+
 def _get_query(
     should_use_incremental_field: bool,
     db_incremental_field_last_value: typing.Any,
     bq_table: bigquery.Table,
     incremental_field: str | None = None,
     incremental_field_type: IncrementalFieldType | None = None,
+    enabled_columns: list[str] | None = None,
+    primary_keys: list[str] | None = None,
 ) -> str:
+    select_clause = _bq_select_clause(enabled_columns, primary_keys, incremental_field)
+
     if should_use_incremental_field:
         if incremental_field is None or incremental_field_type is None:
             raise ValueError("incremental_field and incremental_field_type can't be None")
@@ -390,12 +410,12 @@ def _get_query(
 
         operator = incremental_type_to_operator(incremental_field_type)
         return f"""
-            SELECT * FROM `{bq_table.dataset_id}`.`{bq_table.table_id}`
+            SELECT {select_clause} FROM `{bq_table.dataset_id}`.`{bq_table.table_id}`
             WHERE `{incremental_field}` {operator} {last_value}
             ORDER BY `{incremental_field}` ASC
             """
 
-    return f"SELECT * FROM `{bq_table.dataset_id}`.`{bq_table.table_id}`"
+    return f"SELECT {select_clause} FROM `{bq_table.dataset_id}`.`{bq_table.table_id}`"
 
 
 class BigQueryImplementation(SQLSourceImplementation[BigQuerySourceConfig, bigquery.Client, Any]):
@@ -659,6 +679,7 @@ def _build_source_response(
         db_incremental_field_last_value = (
             inputs.db_incremental_field_last_value if should_use_incremental_field else None
         )
+        enabled_columns = inputs.enabled_columns
         logger = inputs.logger
 
         project_id = config.key_file.project_id
@@ -705,6 +726,11 @@ def get_rows(max_table_size: int) -> collections.abc.Iterator[pa.Table]:
             ) as bq_client:
                 bq_table = bq_client.get_table(fully_qualified_table_name)
 
+                # Query path projects into the temp table; direct-storage path projects via
+                # `selected_fields` on the read session.
+                projected_columns = compute_projected_columns(enabled_columns, primary_keys, incremental_field)
+                storage_selected_fields: list[str] | None = None
+
                 if should_use_incremental_field:
                     # This is only done because incremental syncs require progress tracking.
                     # This requirement means we need to enforce an order, as otherwise
@@ -720,6 +746,8 @@ def get_rows(max_table_size: int) -> collections.abc.Iterator[pa.Table]:
                         bq_table,
                         incremental_field,
                         incremental_field_type,
+                        enabled_columns=enabled_columns,
+                        primary_keys=primary_keys,
                     )
 
                     destination_table = bigquery.Table(bq_destination_table_id)
@@ -741,6 +769,8 @@ def get_rows(max_table_size: int) -> collections.abc.Iterator[pa.Table]:
                         bq_table,
                         incremental_field,
                         incremental_field_type,
+                        enabled_columns=enabled_columns,
+                        primary_keys=primary_keys,
                     )
 
                     destination_table = bigquery.Table(bq_destination_table_id)
@@ -750,16 +780,21 @@ def get_rows(max_table_size: int) -> collections.abc.Iterator[pa.Table]:
 
                     bq_table = bq_client.get_table(destination_table)
 
+                else:
+                    if projected_columns is not None:
+                        storage_selected_fields = projected_columns
+
                 requested_session = bigquery_storage.ReadSession(
                     table=bq_table.to_bqstorage(),
                     data_format=bigquery_storage.DataFormat.ARROW,
                     read_options=bigquery_storage.ReadSession.TableReadOptions(
+                        selected_fields=storage_selected_fields or [],
                         arrow_serialization_options=bigquery_storage.ArrowSerializationOptions(
                             # LZ4 offers a good trade-off of low resource usage for compression, so
                             # as an initial value without further testing it should do fine. That being said,
                             # TODO: Evaluate if ZSTD is a better alternative for our use case.
                             buffer_compression=bigquery_storage.ArrowSerializationOptions.CompressionCodec.LZ4_FRAME
-                        )
+                        ),
                     ),
                 )
                 with bigquery_storage_read_client(
diff --git a/posthog/temporal/data_imports/sources/bigquery/tests/test_source.py b/posthog/temporal/data_imports/sources/bigquery/tests/test_source.py
index f23d4ec8c592..fa58dc9ae895 100644
--- a/posthog/temporal/data_imports/sources/bigquery/tests/test_source.py
+++ b/posthog/temporal/data_imports/sources/bigquery/tests/test_source.py
@@ -5,8 +5,13 @@
 from dateutil import parser
 
 from posthog.temporal.data_imports.pipelines.pipeline.typings import SourceInputs
-from posthog.temporal.data_imports.sources.bigquery.bigquery import BigQueryImplementation
+from posthog.temporal.data_imports.sources.bigquery.bigquery import (
+    BigQueryImplementation,
+    _bq_select_clause,
+    _get_query,
+)
 from posthog.temporal.data_imports.sources.bigquery.source import BigQuerySource
+from posthog.temporal.data_imports.sources.common.sql.identifiers import InvalidIdentifierError
 from posthog.temporal.data_imports.sources.generated_configs import (
     BigQueryDatasetProjectConfig,
     BigQueryKeyFileConfig,
@@ -14,6 +19,8 @@
     BigQueryTemporaryDatasetConfig,
 )
 
+from products.data_warehouse.backend.types import IncrementalFieldType
+
 
 def _make_inputs(**overrides) -> SourceInputs:
     defaults: dict = {
@@ -125,3 +132,60 @@ def test_bigquery_build_pipeline_resolves_dataset_routing(
     assert mock_build.call_args.kwargs["bq_destination_table_id"] == expected_table_id
 
     assert mock_delete.call_args.kwargs["table_id"] == expected_table_id
+
+
+@pytest.mark.parametrize(
+    "enabled_columns,primary_keys,incremental_field,expected",
+    [
+        (None, ["id"], None, "*"),
+        (["email"], ["id"], None, "`email`, `id`"),
+        (["email"], ["id"], "created_at", "`email`, `id`, `created_at`"),
+        ([], None, None, "*"),
+        ([], ["id"], None, "`id`"),
+    ],
+)
+def test_bigquery_select_clause(enabled_columns, primary_keys, incremental_field, expected):
+    assert _bq_select_clause(enabled_columns, primary_keys, incremental_field) == expected
+
+
+def test_bigquery_get_query_projects_enabled_columns():
+    bq_table = mock.MagicMock(dataset_id="ds", table_id="t")
+    query = _get_query(
+        should_use_incremental_field=False,
+        db_incremental_field_last_value=None,
+        bq_table=bq_table,
+        enabled_columns=["email"],
+        primary_keys=["id"],
+    )
+    assert "SELECT `email`, `id` FROM" in query
+
+
+def test_bigquery_get_query_keeps_incremental_field_in_projection():
+    bq_table = mock.MagicMock(dataset_id="ds", table_id="t")
+    query = _get_query(
+        should_use_incremental_field=True,
+        db_incremental_field_last_value=42,
+        bq_table=bq_table,
+        incremental_field="updated_at",
+        incremental_field_type=IncrementalFieldType.Integer,
+        enabled_columns=["email"],
+        primary_keys=["id"],
+    )
+    assert "SELECT `email`, `id`, `updated_at` FROM" in query
+    assert "WHERE `updated_at` > 42" in query
+
+
+@pytest.mark.parametrize(
+    "malicious_column",
+    [
+        "x` FROM `other.private` --",
+        "id; DROP TABLE customers",
+        "email`, `secret",
+        "name with space",
+        "col\x00null",
+    ],
+)
+def test_bigquery_select_clause_rejects_injection_attempts(malicious_column):
+    """`enabled_columns` flows from user config — must be allowlisted before backtick quoting."""
+    with pytest.raises(InvalidIdentifierError):
+        _bq_select_clause([malicious_column], primary_keys=None, incremental_field=None)
diff --git a/posthog/temporal/data_imports/sources/common/base.py b/posthog/temporal/data_imports/sources/common/base.py
index 27ad85d3cb3c..d714aea50188 100644
--- a/posthog/temporal/data_imports/sources/common/base.py
+++ b/posthog/temporal/data_imports/sources/common/base.py
@@ -51,6 +51,10 @@ class _BaseSource(ABC, Generic[ConfigType]):
     source_for_pipeline - use SimpleSource or ResumableSource instead.
     """
 
+    # Default `False` for every source; `SQLSource` flips to `True` (subclasses opt out
+    # via their own override if a driver genuinely can't project columns).
+    supports_column_selection: bool = False
+
     @property
     @abstractmethod
     def source_type(self) -> ExternalDataSourceType:
diff --git a/posthog/temporal/data_imports/sources/common/sql/base.py b/posthog/temporal/data_imports/sources/common/sql/base.py
index 1a3b1bb8cf66..1033504a8db4 100644
--- a/posthog/temporal/data_imports/sources/common/sql/base.py
+++ b/posthog/temporal/data_imports/sources/common/sql/base.py
@@ -53,7 +53,7 @@ class SQLSource(SimpleSource[ConfigType], Generic[ConfigType]):
     wiring around it.
     """
 
-    supports_column_selection: bool = False
+    supports_column_selection: bool = True
 
     @property
     @abstractmethod
diff --git a/posthog/temporal/data_imports/sources/mssql/mssql.py b/posthog/temporal/data_imports/sources/mssql/mssql.py
index 3fea30f2f316..8c96e3427bd3 100644
--- a/posthog/temporal/data_imports/sources/mssql/mssql.py
+++ b/posthog/temporal/data_imports/sources/mssql/mssql.py
@@ -33,7 +33,14 @@
     table_from_iterator,
 )
 from posthog.temporal.data_imports.sources.common.mixins import open_ssh_tunnel
-from posthog.temporal.data_imports.sources.common.sql import BracketIdentifierQuoter, Column, Table
+from posthog.temporal.data_imports.sources.common.sql import (
+    BracketIdentifierQuoter,
+    Column,
+    Table,
+    compute_projected_columns,
+    format_projected_select_clause,
+    project_arrow_columns,
+)
 from posthog.temporal.data_imports.sources.common.sql.implementation import SQLSourceImplementation, TableStats
 from posthog.temporal.data_imports.sources.common.sql.incremental import IncrementalFieldFilter
 from posthog.temporal.data_imports.sources.generated_configs import MSSQLSourceConfig
@@ -73,13 +80,17 @@ def _build_query(
     incremental_field_type: IncrementalFieldType | None,
     db_incremental_field_last_value: Any | None,
     add_limit: bool = False,
+    enabled_columns: list[str] | None = None,
+    primary_keys: list[str] | None = None,
 ) -> tuple[str, dict[str, Any]]:
     # Every identifier interpolated below is validated by the bracket
     # quoter — bad input (`;`, `]`, whitespace, etc.) raises before any
     # SQL is built. Parameter values still flow through pymssql binding.
     qualified_table = _IDENTIFIER_QUOTER.quote_qualified(schema, table_name)
     top = "TOP 100 " if add_limit else ""
-    base_query = f"SELECT {top}* FROM {qualified_table}"
+    projected = compute_projected_columns(enabled_columns, primary_keys, incremental_field)
+    select_clause = format_projected_select_clause(projected, _IDENTIFIER_QUOTER)
+    base_query = f"SELECT {top}{select_clause} FROM {qualified_table}"
 
     if not should_use_incremental_field:
         return base_query, {}
@@ -561,9 +572,22 @@ def build_pipeline(self, config: MSSQLSourceConfig, inputs: SourceInputs) -> Sou
         incremental_field = inputs.incremental_field
         incremental_field_type = inputs.incremental_field_type
         db_incremental_field_last_value = inputs.db_incremental_field_last_value
+        enabled_columns = inputs.enabled_columns
 
         with self.connect(config) as connection:
             with connection.cursor() as cursor:
+                primary_keys = self.get_primary_keys_for_table(cursor, schema, table_name)
+                full_table = self.get_table_metadata(cursor, schema, table_name)
+
+                # Resolve PKs before projection so SELECT and Arrow schema agree.
+                if primary_keys is None and "id" in full_table:
+                    primary_keys = ["id"]
+
+                projected = compute_projected_columns(enabled_columns, primary_keys, incremental_field)
+                table = project_arrow_columns(full_table, projected)
+                arrow_schema = table.to_arrow_schema()
+                logger.debug(f"Source schema: {arrow_schema}")
+
                 inner_query, inner_query_args = _build_query(
                     schema,
                     table_name,
@@ -571,12 +595,10 @@ def build_pipeline(self, config: MSSQLSourceConfig, inputs: SourceInputs) -> Sou
                     incremental_field,
                     incremental_field_type,
                     db_incremental_field_last_value,
+                    enabled_columns=enabled_columns,
+                    primary_keys=primary_keys,
                 )
 
-                primary_keys = self.get_primary_keys_for_table(cursor, schema, table_name)
-                table = self.get_table_metadata(cursor, schema, table_name)
-                arrow_schema = table.to_arrow_schema()
-                logger.debug(f"Source schema: {arrow_schema}")
                 rows_to_sync = self.get_rows_to_sync(cursor, inner_query, inner_query_args, logger)
                 chunk_size = self.get_chunk_size(cursor, schema, table_name, inner_query, inner_query_args, logger)
                 partition_settings = (
@@ -585,10 +607,6 @@ def build_pipeline(self, config: MSSQLSourceConfig, inputs: SourceInputs) -> Sou
                     else None
                 )
 
-                # Fallback on checking for an `id` field on the table
-                if primary_keys is None and "id" in table:
-                    primary_keys = ["id"]
-
         def get_rows() -> Iterator[Any]:
             with self.connect(config) as streaming_connection:
                 with streaming_connection.cursor() as cursor:
@@ -599,6 +617,8 @@ def get_rows() -> Iterator[Any]:
                         incremental_field,
                         incremental_field_type,
                         db_incremental_field_last_value,
+                        enabled_columns=enabled_columns,
+                        primary_keys=primary_keys,
                     )
                     if incremental_field:
                         query = f"{query} ORDER BY {_IDENTIFIER_QUOTER.quote(incremental_field)} ASC"
diff --git a/posthog/temporal/data_imports/sources/mssql/tests/test_mssql.py b/posthog/temporal/data_imports/sources/mssql/tests/test_mssql.py
index f68322bd1df1..745539b206f9 100644
--- a/posthog/temporal/data_imports/sources/mssql/tests/test_mssql.py
+++ b/posthog/temporal/data_imports/sources/mssql/tests/test_mssql.py
@@ -165,6 +165,45 @@ def test_rejects_unsafe_incremental_field(self):
             )
 
 
+class TestBuildQueryEnabledColumns:
+    @pytest.mark.parametrize(
+        "enabled_columns,primary_keys,expected_select",
+        [
+            (None, ["id"], "SELECT * FROM"),
+            (["email"], ["id"], "SELECT [email], [id] FROM"),
+            ([], None, "SELECT * FROM"),
+            ([], ["id"], "SELECT [id] FROM"),
+        ],
+    )
+    def test_full_refresh_projection(self, enabled_columns, primary_keys, expected_select):
+        query, _ = _build_query(
+            schema="dbo",
+            table_name="users",
+            should_use_incremental_field=False,
+            incremental_field=None,
+            incremental_field_type=None,
+            db_incremental_field_last_value=None,
+            enabled_columns=enabled_columns,
+            primary_keys=primary_keys,
+        )
+        assert query.startswith(expected_select)
+
+    def test_incremental_projection_retains_incremental_field(self):
+        query, args = _build_query(
+            schema="dbo",
+            table_name="users",
+            should_use_incremental_field=True,
+            incremental_field="created_at",
+            incremental_field_type=IncrementalFieldType.DateTime,
+            db_incremental_field_last_value="2025-01-01",
+            enabled_columns=["email"],
+            primary_keys=["id"],
+        )
+        assert query.startswith("SELECT [email], [id], [created_at] FROM")
+        assert "WHERE [created_at] > %(incremental_value)s" in query
+        assert args == {"incremental_value": "2025-01-01"}
+
+
 # ---------------------------------------------------------------------------
 # Per-cursor metadata queries
 # ---------------------------------------------------------------------------
diff --git a/posthog/temporal/data_imports/sources/mysql/mysql.py b/posthog/temporal/data_imports/sources/mysql/mysql.py
index 62b2043e9c91..32dacf6eb4f2 100644
--- a/posthog/temporal/data_imports/sources/mysql/mysql.py
+++ b/posthog/temporal/data_imports/sources/mysql/mysql.py
@@ -48,6 +48,8 @@
     InvalidIdentifierError,
     SelectQueryBuilder,
     Table,
+    compute_projected_columns,
+    project_arrow_columns,
 )
 from posthog.temporal.data_imports.sources.common.sql.implementation import SQLSourceImplementation, TableStats
 from posthog.temporal.data_imports.sources.common.sql.incremental import IncrementalFieldFilter
@@ -158,6 +160,8 @@ def _build_query(
     incremental_field_type: IncrementalFieldType | None,
     db_incremental_field_last_value: Any | None,
     force_index_name: str | None = None,
+    enabled_columns: list[str] | None = None,
+    primary_keys: list[str] | None = None,
 ) -> tuple[str, dict[str, Any]]:
     hint: str | None = None
     if force_index_name is not None:
@@ -169,6 +173,8 @@ def _build_query(
             schema=schema,
             table_name=table_name,
             extra_table_hint=hint,
+            enabled_columns=enabled_columns,
+            primary_keys=primary_keys,
         )
         params = result.params if isinstance(result.params, dict) else {}
         return result.sql, params
@@ -183,6 +189,8 @@ def _build_query(
         incremental_field_type=incremental_field_type,
         incremental_last_value=db_incremental_field_last_value,
         extra_table_hint=hint,
+        enabled_columns=enabled_columns,
+        primary_keys=primary_keys,
     )
     params = result.params if isinstance(result.params, dict) else {}
     return result.sql, params
@@ -722,9 +730,22 @@ def build_pipeline(self, config: MySQLSourceConfig, inputs: SourceInputs) -> Sou
         incremental_field = inputs.incremental_field
         incremental_field_type = inputs.incremental_field_type
         db_incremental_field_last_value = inputs.db_incremental_field_last_value
+        enabled_columns = inputs.enabled_columns
 
         with self.connect(config) as connection:
             with connection.cursor() as cursor:
+                primary_keys = self.get_primary_keys_for_table(cursor, schema, table_name)
+                full_table = self.get_table_metadata(cursor, schema, table_name)
+
+                # Resolve PKs before the projection so probe/sample queries match the streaming SELECT.
+                if primary_keys is None and "id" in full_table:
+                    primary_keys = ["id"]
+
+                projected = compute_projected_columns(enabled_columns, primary_keys, incremental_field)
+                table = project_arrow_columns(full_table, projected)
+                arrow_schema = table.to_arrow_schema()

Leaderboard for this case

hard · 0 of 9 bots fully caught it (0% solve rate) · 0% avg coverage.

#BotResultCaughtCoverageFindingsFalse alarms
1 Claude Code (Opus 5.5) miss 0/1 0.0% 2 0
2 cursor miss 0/1 0.0% 1 0
3 Drape (Deep Review) miss 0/1 0.0% 5 0
4 Drape (Review) miss 0/1 0.0% 2 0
5 greptile miss 0/1 0.0% 2 0
6 Mjolnir (Opus 5.5) miss 0/1 0.0% 4 0
7 Mjolnir (Sonnet 5.5) miss 0/1 0.0% 2 0
8 qodo miss 0/1 0.0% 3 0
9 Cubic miss 0/1 0.0% 3 1

Bot results

Claude Code (Opus 5.5) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✗ —
valid · impact medium
Before the change, the duplicate check ran only inside `if primary_keys is None and "id" in table`. The new condition is also true when Redshift reports a real `["id"]` primary key, so the full-table GROUP BY now runs on tables where it never ran before, under the 10-minute statement timeout. [grounded: diff] [consensus 3/3: valid=3]
valid · impact medium
`_validate_identifier` in identifiers.py accepts only alphanumerics plus `._-$@`, so it rejects names containing spaces. Before this change, Snowflake, BigQuery and MSSQL ignored `enabled_columns` and ran `SELECT *`. Now any non-None projection that includes such a column name raises InvalidIdentifierError and the sync fails. [grounded: diff] [consensus 3/3: valid=3]
raw comments (2)
posthog/temporal/data_imports/sources/redshift/redshift.py:852
The duplicate-PK check used to run only when the code fell back to `['id']`. The new condition `primary_keys == ['id'] and 'id' in full_table` cannot tell a fallback apart from a real primary key named `id`. So any Redshift table with a real `id` primary key now runs `has_duplicate_primary_keys`, a full-table `GROUP BY ... HAVING COUNT(*) > 1`. On large tables this costs a lot, and it runs under the 10-minute `statement_timeout`. A `QueryCanceled` is re-raised, so syncs that worked before can now fail, and duplicates may be flagged on tables that were previously trusted. Keep a separate `fell_back_to_id` flag instead of comparing values.
posthog/temporal/data_imports/sources/snowflake/snowflake.py:576
Turning on `supports_column_selection` for every SQLSource sends each selected column, primary key and incremental field through `format_projected_select_clause`. That runs the strict identifier allowlist, which allows only alphanumerics and `._-$@`. Snowflake quoted identifiers can legitimately contain spaces and other characters, and so can BigQuery flexible column names (`_bq_select_clause`) and MSSQL bracketed names. Before this change these tables synced with `SELECT *`. Once a user picks columns on such a table, `_build_query` raises `InvalidIdentifierError` and the whole sync fails, including when only the PK or incremental field has a space. Either use escaping-based quoting (as Redshift does with `sql.Identifier`) or block column selection for tables with names that fail the check.
Cubic · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✗ —
valid · impact medium
The projection only adds the PKs BigQuery detected. pipeline_v3 then replaces those with the user's schema.primary_key_columns (`if schema.primary_key_columns: self._resource.primary_keys = schema.primary_key_columns`), so a configured PK that isn't in enabled_columns gets dropped from the projection and the merge has no column to key on. [grounded: diff] [consensus 5/5: valid=5]
valid · impact low
BracketIdentifierQuoter rejects whitespace (the existing tests reject 'dbo with space'). Before this change MSSQL used SELECT *. Now any enabled column whose name has whitespace, which MSSQL allows for delimited identifiers, raises before the sync starts. [grounded: diff] [consensus 5/5: valid=5]
invalid
The caller already passes incremental_field=None when incremental syncing is off, so a full-refresh Snowflake sync never receives a stale cursor field to add to the projection. [invalid: contradiction] [consensus 5/5: invalid=5]
raw comments (4)
posthog/temporal/data_imports/sources/bigquery/bigquery.py:731
<!-- cubic:v=46f66c51-710c-4216-a0c9-5e1ad7fd7807 --> <!-- cubic:review-run=783592f8-dec7-589b-a5e5-df0769baf085 --> <!-- metadata:{"confidence":9} --> P1: When a schema's configured primary keys differ from BigQuery's detected keys, this projection can omit a configured key that the pipeline later uses for merging. Pass the effective configured primary keys into the source projection, or otherwise retain them before creating the read session. <details> <summary>Prompt for AI agents</summary> ```text Check if this issue is valid — if so, understand the root cause and fix it. At posthog/temporal/data_imports/sources/bigquery/bigquery.py, line 731: <comment>When a schema's configured primary keys differ from BigQuery's detected keys, this projection can omit a configured key that the pipeline later uses for merging. Pass the effective configured primary keys into the source projection, or otherwise retain them before creating the read session.</comment> <file context> @@ -705,6 +726,11 @@ def get_rows(max_table_size: int) -> collections.abc.Iterator[pa.Table]: + # Query path projects into the temp table; direct-storage path projects via + # `selected_fields` on the read session. + projected_columns = compute_projected_columns(enabled_columns, primary_keys, incremental_field) + storage_selected_fields: list[str] | None = None + </file context> ``` </details>
posthog/temporal/data_imports/sources/mssql/mssql.py:92
<!-- cubic:v=95e8f42d-1028-4072-8481-5beb00f0f0ad --> <!-- cubic:review-run=783592f8-dec7-589b-a5e5-df0769baf085 --> <!-- metadata:{"confidence":9} --> P2: When a selected MSSQL column has a valid delimited name containing whitespace, `_build_query` rejects it before the sync starts because `BracketIdentifierQuoter` disallows whitespace. Quote valid MSSQL identifiers consistently or support these column names in the quoter. <details> <summary>Prompt for AI agents</summary> ```text Check if this issue is valid — if so, understand the root cause and fix it. At posthog/temporal/data_imports/sources/mssql/mssql.py, line 92: <comment>When a selected MSSQL column has a valid delimited name containing whitespace, `_build_query` rejects it before the sync starts because `BracketIdentifierQuoter` disallows whitespace. Quote valid MSSQL identifiers consistently or support these column names in the quoter.</comment> <file context> @@ -73,13 +80,17 @@ def _build_query( top = "TOP 100 " if add_limit else "" - base_query = f"SELECT {top}* FROM {qualified_table}" + projected = compute_projected_columns(enabled_columns, primary_keys, incremental_field) + select_clause = format_projected_select_clause(projected, _IDENTIFIER_QUOTER) + base_query = f"SELECT {top}{select_clause} FROM {qualified_table}" </file context> ``` </details>
posthog/temporal/data_imports/sources/snowflake/snowflake.py:80
<!-- cubic:v=98bb95c1-d6e4-4f7e-92dc-2407d4a3cab0 --> <!-- cubic:review-run=783592f8-dec7-589b-a5e5-df0769baf085 --> <!-- metadata:{"confidence":8} --> P2: When a full-refresh schema still has a non-null `incremental_field`, this call adds it to the projection even though incremental syncing is disabled. Only retain the cursor field when `should_use_incremental_field` is true, otherwise column selection can sync an unrequested field or reference a stale column. <details> <summary>Prompt for AI agents</summary> ```text Check if this issue is valid — if so, understand the root cause and fix it. At posthog/temporal/data_imports/sources/snowflake/snowflake.py, line 80: <comment>When a full-refresh schema still has a non-null `incremental_field`, this call adds it to the projection even though incremental syncing is disabled. Only retain the cursor field when `should_use_incremental_field` is true, otherwise column selection can sync an unrequested field or reference a stale column.</comment> <file context> @@ -66,9 +74,14 @@ def _build_query( + enabled_columns: list[str] | None = None, + primary_keys: list[str] | None = None, ) -> tuple[str, tuple[Any, ...]]: + projected = compute_projected_columns(enabled_columns, primary_keys, incremental_field) + select_clause = format_projected_select_clause(projected, _SNOWFLAKE_IDENTIFIER_QUOTER) + </file context> ``` </details> ```suggestion projected = compute_projected_columns( enabled_columns, primary_keys, incremental_field if should_use_incremental_field else None ) ```
PR-level
<!-- cubic:review-summary:start --> **3 issues found** across 23 files <!-- cubic:review-summary:end --> <details> <summary>Prompt for AI agents (unresolved issues)</summary> ```text Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately. <file name="posthog/temporal/data_imports/sources/bigquery/bigquery.py"> <violation number="1" location="posthog/temporal/data_imports/sources/bigquery/bigquery.py:731"> P1: When a schema's configured primary keys differ from BigQuery's detected keys, this projection can omit a configured key that the pipeline later uses for merging. Pass the effective configured primary keys into the source projection, or otherwise retain them before creating the read session.</violation> </file> <file name="posthog/temporal/data_imports/sources/mssql/mssql.py"> <violation number="1" location="posthog/temporal/data_imports/sources/mssql/mssql.py:92"> P2: When a selected MSSQL column has a valid delimited name containing whitespace, `_build_query` rejects it before the sync starts because `BracketIdentifierQuoter` disallows whitespace. Quote valid MSSQL identifiers consistently or support these column names in the quoter.</violation> </file> <file name="posthog/temporal/data_imports/sources/snowflake/snowflake.py"> <violation number="1" location="posthog/temporal/data_imports/sources/snowflake/snowflake.py:80"> P2: When a full-refresh schema still has a non-null `incremental_field`, this call adds it to the projection even though incremental syncing is disabled. Only retain the cursor field when `should_use_incremental_field` is true, otherwise column selection can sync an unrequested field or reference a stale column.</violation> </file> ``` </details> <sub>Reply with feedback, questions, or to request a fix.<br /><br />[Re-trigger cubic](https://www.cubic.dev/action/re-review/pr/drape-io/crbench-cubic-2026-09/93/ai_pr_review_github_delivery_6b55c39c1b092c8367b061f74a769fc60168ef45e55ad5e510a3dcd1ee6362a8?returnTo=https%3A%2F%2Fgithub.com%2Fdrape-io%2Fcrbench-cubic-2026-09%2Fpull%2F93)</sub> <!-- cubic:review-post:ai_pr_review_github_delivery_6b55c39c1b092c8367b061f74a769fc60168ef45e55ad5e510a3dcd1ee6362a8:18634a4a873d410f528154c368e399898ba0938c:783592f8-dec7-589b-a5e5-df0769baf085 -->
cursor · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✗ —
valid · impact medium
Before the change, the duplicate check ran only inside the `primary_keys is None` fallback branch. Now it runs whenever the resolved PK is ["id"], so it also runs for tables whose real primary key is `id`. That adds a full-table GROUP BY under the 10-minute statement timeout. `has_duplicate_primary_keys` re-raises QueryCanceled, and the caller turns it into a QueryTimeoutException for incremental syncs. [grounded: diff] [consensus 5/5: valid=5]
raw comments (2)
posthog/temporal/data_imports/sources/redshift/redshift.py:853
### Redshift duplicate check runs too often **Medium Severity** <!-- DESCRIPTION START --> The duplicate-primary-key scan now runs whenever resolved keys are `["id"]`, not only after the no-PK fallback. Tables with a real `id` primary key now run a full-table `GROUP BY` that can hit the 10-minute statement timeout and fail incremental Redshift syncs. <!-- DESCRIPTION END --> <!-- BUGBOT_BUG_ID: d6d36f33-dd6a-4ed5-9565-5cd8170fb6a5 --> <!-- LOCATIONS START posthog/temporal/data_imports/sources/redshift/redshift.py#L851-L857 LOCATIONS END --> <div><a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9DVVJTT1IiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OmZiYjljY2VlLWUxMmEtNGY3NC04Yzc5LTU1ZTRlZThlOWNmNiIsImVuY3J5cHRpb25LZXkiOiJzWmF3NXpwMXNFck4zMlRVUGJ1cktXRVNUQnQzdWw0TDF0dUcxOFZyOG5nIiwiYnJhbmNoIjoidDItcG9zdGhvZy02MDE1NSIsInJlcG9Pd25lciI6ImRyYXBlLWlvIiwicmVwb05hbWUiOiJjcmJlbmNoLWN1cnNvci0yMDI2LTA5IiwicHJvdmlkZXIiOiJnaXRodWIifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/agents?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9XRUIiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OmZiYjljY2VlLWUxMmEtNGY3NC04Yzc5LTU1ZTRlZThlOWNmNiIsImVuY3J5cHRpb25LZXkiOiJzWmF3NXpwMXNFck4zMlRVUGJ1cktXRVNUQnQzdWw0TDF0dUcxOFZyOG5nIiwiYnJhbmNoIjoidDItcG9zdGhvZy02MDE1NSIsInJlcG9Pd25lciI6ImRyYXBlLWlvIiwicmVwb05hbWUiOiJjcmJlbmNoLWN1cnNvci0yMDI2LTA5IiwicHJOdW1iZXIiOjk0LCJjb21taXRTaGEiOiI5MzA2MWM3NDgwMDcxMzdhNzAxN2M5YWZmNzEyNmU4NzRlYmQ1YWM3IiwicHJvdmlkZXIiOiJnaXRodWIifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-web-light.png"><img alt="Fix in Web" width="99" height="28" src="https://cursor.com/assets/images/fix-in-web-dark.png"></picture></a></div> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 93061c748007137a7017c9aff7126e874ebd5ac7. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
PR-level
<!-- BUGBOT_REVIEW --> Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue. <!-- BUGBOT_FIX_ALL --> <a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9BTExfSU5fQ1VSU09SIiwiZGF0YSI6eyJyZWRpc0tleSI6ImJ1Z2JvdC1tdWx0aTpmNjVkYjQ5YS1mODZkLTQxZDctYTNkZi0yNWFmYjgzODIwZjEiLCJlbmNyeXB0aW9uS2V5IjoiVlV1ajRQN1M3a252LVhfSzVSVnRIWkwzcGIxSlMtc3dkcC1XUGNKQlF3SSIsImJyYW5jaCI6InQyLXBvc3Rob2ctNjAxNTUiLCJyZXBvT3duZXIiOiJkcmFwZS1pbyIsInJlcG9OYW1lIjoiY3JiZW5jaC1jdXJzb3ItMjAyNi0wOSIsInByb3ZpZGVyIjoiZ2l0aHViIn19" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix All in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a> <!-- /BUGBOT_FIX_ALL --> <!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_BEGIN --> <sup>❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the [Cursor dashboard](https://www.cursor.com/dashboard/bugbot).</sup> <!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_END --> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 93061c748007137a7017c9aff7126e874ebd5ac7. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
Drape (Deep Review) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✗ —
valid · impact medium
The new projection sends user-selected column names through `_validate_identifier`, which rejects whitespace and similar characters. The added test even asserts that "name with space" raises. Before this change, SELECT * handled those columns, so syncs with legitimately-named columns (BigQuery flexible names, MSSQL/Snowflake quoted names) now fail once columns are customized. [grounded: diff] [consensus 3/3: valid=3]
valid · impact medium
Previously the duplicate check ran only inside the fallback branch (primary_keys was None). Now it also runs when the declared PK is exactly ['id'], which changes the cost and the has_duplicate_primary_keys outcome for those tables. [grounded: diff] [consensus 3/3: valid=3]
valid · impact medium
MySQL's query builder now runs format_projected_select_clause with the backtick quoter, which allowlist-rejects spaces and parentheses. Column names like `order date` were fine under SELECT * but now raise once a subset is selected. [grounded: diff] [consensus 3/3: valid=3]
valid · impact medium
Snowflake gets primary_keys only from SHOW PRIMARY KEYS and has no 'id' fallback. During create, however, discovery's 'id' fallback is stored as primary_key_columns, and the pipeline overrides PKs with it. If the user deselects 'id', the projection omits a column the pipeline still merges on. [grounded: diff] [consensus 3/3: valid=3]
valid · impact medium
The projection keeps only driver-detected PKs, while the pipeline overrides them with schema.primary_key_columns. The serializer only checks that enabled_columns exist and does not force user PKs into the selection, so a deselected user PK goes missing from the synced data. [grounded: tree posthog/temporal/data_imports/pipelines/pipeline/pipeline.py] [consensus 3/3: valid=3]
raw comments (5)
posthog/temporal/data_imports/sources/bigquery/bigquery.py:385
**BigQuery incremental/view syncs reject some column names** On the query path (incremental syncs and views), `_bq_select_clause` sends each enabled column through the backtick quoter's allowlist. That allowlist rejects names BigQuery accepts with flexible column names (spaces, etc.), so the sync raises once the user customizes columns. Escape backticks in column names instead of allowlist-validating them. The same mistake is at `posthog/temporal/data_imports/sources/snowflake/snowflake.py:81` and `posthog/temporal/data_imports/sources/mssql/mssql.py:92`. ``` posthog/temporal/data_imports/sources/bigquery/bigquery.py:378 def _bq_select_clause( posthog/temporal/data_imports/sources/bigquery/bigquery.py:379 enabled_columns: list[str] | None, posthog/temporal/data_imports/sources/bigquery/bigquery.py:380 primary_keys: list[str] | None, posthog/temporal/data_imports/sources/bigquery/bigquery.py:381 incremental_field: str | None, posthog/temporal/data_imports/sources/common/sql/identifiers.py:37 def _validate_identifier(identifier: str) -> None: posthog/temporal/data_imports/sources/common/sql/identifiers.py:38 """Allowlist check shared by every concrete quoter. posthog/temporal/data_imports/sources/common/sql/identifiers.py:40 Accepts: alphanumeric plus `.`, `_`, `-`, `$`, `@`. An identifier that posthog/temporal/data_imports/sources/common/sql/identifiers.py:41 is nothing but digits is also allowed (matches today's MySQL behavior posthog/temporal/data_imports/sources/common/sql/identifiers.py:42 where `"851"` is quoted verbatim). Anything else — whitespace, a posthog/temporal/data_imports/sources/common/sql/identifiers.py:43 semicolon, a quote character, a NUL byte — raises. ```
posthog/temporal/data_imports/sources/redshift/redshift.py:852
**Redshift duplicate-key scan runs on real PKs** `primary_keys == ["id"]` also matches tables whose declared PK is `id`, so the costly duplicate check now runs on every sync of those tables and can newly flag duplicates. Set a fallback flag when falling back to `id` and gate the check on it. ``` posthog/temporal/data_imports/sources/redshift/redshift.py:794 # Resolve PKs before projection so SELECT and Arrow schema agree. posthog/temporal/data_imports/sources/redshift/redshift.py:795 if primary_keys is None and "id" in full_table: posthog/temporal/data_imports/sources/redshift/redshift.py:796 logger.debug("Falling back to ['id'] for primary keys...") posthog/temporal/data_imports/sources/redshift/redshift.py:797 primary_keys = ["id"] posthog/temporal/data_imports/sources/redshift/redshift.py:812 (base) # Fallback on checking for an `id` field on the table posthog/temporal/data_imports/sources/redshift/redshift.py:813 (base) if primary_keys is None and "id" in table: posthog/temporal/data_imports/sources/redshift/redshift.py:814 (base) logger.debug("Falling back to ['id'] for primary keys...") posthog/temporal/data_imports/sources/redshift/redshift.py:815 (base) primary_keys = ["id"] posthog/temporal/data_imports/sources/redshift/redshift.py:816 (base) logger.debug("Checking duplicate primary keys...") posthog/temporal/data_imports/sources/redshift/redshift.py:817 (base) duplicate_primary_keys = self.has_duplicate_primary_keys( ```
posthog/temporal/data_imports/sources/mysql/mysql.py:176
**MySQL syncs fail for column names with spaces** Passing `enabled_columns` into the query builder sends every selected column through the backtick allowlist. Columns such as `order date` or `price(usd)` now raise, so the table stops syncing as soon as the user picks a subset. Quote column names by escaping backticks instead of allowlisting characters. ``` posthog/temporal/data_imports/sources/mysql/mysql.py:163 enabled_columns: list[str] | None = None, posthog/temporal/data_imports/sources/mysql/mysql.py:164 primary_keys: list[str] | None = None, posthog/temporal/data_imports/sources/mysql/mysql.py:165 ) -> tuple[str, dict[str, Any]]: posthog/temporal/data_imports/sources/mysql/mysql.py:166 hint: str | None = None posthog/temporal/data_imports/sources/common/sql/identifiers.py:37 def _validate_identifier(identifier: str) -> None: posthog/temporal/data_imports/sources/common/sql/identifiers.py:38 """Allowlist check shared by every concrete quoter. posthog/temporal/data_imports/sources/common/sql/identifiers.py:40 Accepts: alphanumeric plus `.`, `_`, `-`, `$`, `@`. An identifier that posthog/temporal/data_imports/sources/common/sql/identifiers.py:41 is nothing but digits is also allowed (matches today's MySQL behavior posthog/temporal/data_imports/sources/common/sql/identifiers.py:42 where `"851"` is quoted verbatim). Anything else — whitespace, a posthog/temporal/data_imports/sources/common/sql/identifiers.py:43 semicolon, a quote character, a NUL byte — raises. ```
posthog/temporal/data_imports/sources/snowflake/snowflake.py:415
**Snowflake projection drops the stored primary key** Without a declared PK, `primary_keys` is None here, so the stored `id` fallback isn't projected. If the user deselects `id`, the pipeline still merges on `id` and the column is missing. Also project the schema's `primary_key_columns` and the `id` fallback. ``` posthog/temporal/data_imports/sources/snowflake/snowflake.py:415 primary_keys = self.get_primary_keys_for_table(cursor, database, schema, table_name) posthog/temporal/data_imports/sources/snowflake/snowflake.py:416 inner_query, inner_query_params = _build_query( posthog/temporal/data_imports/sources/snowflake/snowflake.py:417 database, posthog/temporal/data_imports/sources/snowflake/snowflake.py:418 schema, posthog/temporal/data_imports/sources/snowflake/snowflake.py:354 cursor.execute("SHOW PRIMARY KEYS IN IDENTIFIER(%s)", (f"{database}.{schema}.{table_name}",)) posthog/temporal/data_imports/sources/snowflake/snowflake.py:356 column_index = next((i for i, row in enumerate(cursor.description) if row.name == "column_name"), -1) posthog/temporal/data_imports/sources/snowflake/snowflake.py:358 if column_index == -1: posthog/temporal/data_imports/sources/snowflake/snowflake.py:359 raise ValueError("column_name not found in Snowflake cursor description") posthog/temporal/data_imports/sources/snowflake/snowflake.py:361 keys = [row[column_index] for row in cursor] posthog/temporal/data_imports/sources/snowflake/snowflake.py:363 return keys if len(keys) > 0 else None ```
posthog/temporal/data_imports/sources/mysql/mysql.py:744
**User-chosen primary keys not kept in projection** Only driver-detected PKs are projected. A user-set `primary_key_columns` left out of the picker is dropped from the SELECT, but the pipeline still merges on it. MSSQL, Redshift and BigQuery do the same. Pass the schema's configured primary key columns into the projection. ``` posthog/temporal/data_imports/sources/mysql/mysql.py:737 primary_keys = self.get_primary_keys_for_table(cursor, schema, table_name) posthog/temporal/data_imports/sources/mysql/mysql.py:738 full_table = self.get_table_metadata(cursor, schema, table_name) posthog/temporal/data_imports/sources/mysql/mysql.py:740 # Resolve PKs before the projection so probe/sample queries match the streaming SELECT. posthog/temporal/data_imports/sources/mysql/mysql.py:741 if primary_keys is None and "id" in full_table: posthog/temporal/data_imports/pipelines/pipeline/pipeline.py:154 # Allow user-specified primary keys to override auto-detected ones posthog/temporal/data_imports/pipelines/pipeline/pipeline.py:155 if schema.primary_key_columns: posthog/temporal/data_imports/pipelines/pipeline/pipeline.py:156 self._resource.primary_keys = schema.primary_key_columns posthog/temporal/data_imports/sources/common/sql/projection.py:23 def compute_projected_columns( posthog/temporal/data_imports/sources/common/sql/projection.py:24 enabled_columns: list[str] | None, posthog/temporal/data_imports/sources/common/sql/projection.py:25 primary_keys: list[str] | None = None, ```
Drape (Review) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✗ —
valid · impact medium
Previously the query was always SELECT *. Now, whenever enabled_columns is set, each projected column goes through the allowlist quoter. _validate_identifier rejects whitespace and similar characters, so a sync with a selected column like `First Name` raises InvalidIdentifierError. [grounded: diff] [consensus 3/3: valid=3]
valid · impact low
The old code ran has_duplicate_primary_keys only inside the `primary_keys is None` fallback branch. The new condition is also true when the declared PK is id, which adds a full-table GROUP BY scan for those tables. [grounded: diff] [consensus 3/3: valid=3]
raw comments (2)
posthog/temporal/data_imports/sources/mssql/mssql.py:92
**Column picker breaks sync for names with spaces** With selected columns, the SELECT list uses the strict quoter, which rejects anything but alphanumerics and `._-$@`. A column like `First Name` raises `InvalidIdentifierError`, where `SELECT *` worked before. MySQL, BigQuery and Snowflake share this. Escape quote characters per dialect instead of rejecting, or fall back to `SELECT *`. ``` posthog/temporal/data_imports/sources/mssql/mssql.py:91 projected = compute_projected_columns(enabled_columns, primary_keys, incremental_field) posthog/temporal/data_imports/sources/mssql/mssql.py:92 select_clause = format_projected_select_clause(projected, _IDENTIFIER_QUOTER) posthog/temporal/data_imports/sources/mssql/mssql.py:93 base_query = f"SELECT {top}{select_clause} FROM {qualified_table}" posthog/temporal/data_imports/sources/bigquery/bigquery.py:375 _BQ_QUOTER = BacktickIdentifierQuoter() posthog/temporal/data_imports/sources/common/sql/identifiers.py:37 def _validate_identifier(identifier: str) -> None: posthog/temporal/data_imports/sources/common/sql/identifiers.py:38 """Allowlist check shared by every concrete quoter. posthog/temporal/data_imports/sources/common/sql/identifiers.py:40 Accepts: alphanumeric plus `.`, `_`, `-`, `$`, `@`. An identifier that posthog/temporal/data_imports/sources/common/sql/identifiers.py:41 is nothing but digits is also allowed (matches today's MySQL behavior posthog/temporal/data_imports/sources/common/sql/identifiers.py:42 where `"851"` is quoted verbatim). Anything else — whitespace, a posthog/temporal/data_imports/sources/common/sql/identifiers.py:43 semicolon, a quote character, a NUL byte — raises. ```
posthog/temporal/data_imports/sources/redshift/redshift.py:852
**Redshift duplicate-key check runs on real id PK** The duplicate-PK check now runs whenever `primary_keys == ["id"]`, not only when the code fell back to `id`. A table whose real, declared PK is `id` now gets the extra full-table duplicate scan. Track a `used_id_fallback` flag when primary_keys is set to ['id'] and gate the duplicate check on it. ``` posthog/temporal/data_imports/sources/redshift/redshift.py:851 duplicate_primary_keys = False posthog/temporal/data_imports/sources/redshift/redshift.py:852 if primary_keys == ["id"] and "id" in full_table: posthog/temporal/data_imports/sources/redshift/redshift.py:853 # Only check dupes when we fell back to the `id` PK above. posthog/temporal/data_imports/sources/redshift/redshift.py:854 logger.debug("Checking duplicate primary keys...") posthog/temporal/data_imports/sources/redshift/redshift.py:855 duplicate_primary_keys = self.has_duplicate_primary_keys( posthog/temporal/data_imports/sources/redshift/redshift.py:856 cursor, schema, table_name, primary_keys, logger posthog/temporal/data_imports/sources/redshift/redshift.py:857 ) ```
greptile · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✗ —
valid · impact low
`compute_projected_columns` returns None when the ordered list is empty (`if not ordered: return None`), so an empty `enabled_columns` with no PK and no incremental field becomes `SELECT *` (or an empty `selected_fields`, which also means all fields) in every source this PR wires up. The fallback is deliberate (it avoids emitting `SELECT FROM`), but an explicit empty selection still imports every column. This PR is what exposes the picker and projection for these sources. [grounded: diff] [consensus 3/3: valid=3]
valid · impact medium
`_validate_identifier` in identifiers.py raises `InvalidIdentifierError` for any character that is not alphanumeric or one of `._-$@`, so a valid quoted column such as `First Name` fails. Before this change these sources used `SELECT *` and never quoted column names. Now any non-null `enabled_columns` (including the primary keys that are always added) goes through the quoter, so the sync fails. [grounded: diff] [consensus 3/3: valid=3]
raw comments (3)
posthog/temporal/data_imports/sources/bigquery/bigquery.py:732
<a href="#"><img alt="P1" src="https://greptile-static-assets.s3.amazonaws.com/badges/p1.svg?v=9" align="top"></a> **Empty selection imports everything** When a user unchecks every column on a full-refresh table without a primary key or incremental field, `compute_projected_columns([], None, None)` returns `None`. This BigQuery path therefore requests every field instead of none. The same fallback is used by the new MSSQL, MySQL, Redshift, and Snowflake projection paths. Because the picker permits and persists an empty list, an explicit no-column selection unexpectedly imports all source columns, including data the user intended to exclude.
posthog/temporal/data_imports/sources/snowflake/snowflake.py:81
<a href="#"><img alt="P1" src="https://greptile-static-assets.s3.amazonaws.com/badges/p1.svg?v=9" align="top"></a> **Valid identifiers reject syncs** When a selected column has a valid quoted database name such as `First Name`, the shared identifier allowlist rejects its whitespace instead of escaping it, so this Snowflake query raises `InvalidIdentifierError` before execution. The newly enabled BigQuery, MSSQL, and MySQL pickers use the same validation. These databases support such quoted identifiers, so selecting a legitimate column can make an otherwise valid sync fail.
PR-level
<h3>Greptile Summary</h3> This PR extends per-column selection across SQL warehouse sources, exposes source capabilities through wizard and source APIs, persists and refreshes schema metadata, adds CDC creation validation, and updates the schema configuration UI. - Adds projected queries for BigQuery, MSSQL, MySQL, Redshift, and Snowflake while retaining primary-key and incremental fields. - Enables column-selection controls based on backend source capabilities and adds schema-refresh affordances. - Moves CDC setup after primary-key validation and rejects inconsistent CDC payloads. - Two projection edge cases remain: an empty selection can import all columns, and valid identifiers containing spaces or quotes are rejected. <h3>Confidence Score: 3/5</h3> The PR is not safe to merge until empty selections stop expanding to all columns and valid quoted database column names can be projected reliably. The new projection paths have two blocking failures: selecting no columns on a table without retained fields imports the full table, and selecting legitimate identifiers containing spaces or quotes aborts supported SQL-source syncs. **Files Needing Attention:** posthog/temporal/data_imports/sources/bigquery/bigquery.py, posthog/temporal/data_imports/sources/mssql/mssql.py, posthog/temporal/data_imports/sources/mysql/mysql.py, posthog/temporal/data_imports/sources/redshift/redshift.py, posthog/temporal/data_imports/sources/snowflake/snowflake.py <details><summary><h3>Important Files Changed</h3></summary> | Filename | Overview | |----------|----------| | posthog/temporal/data_imports/sources/bigquery/bigquery.py | Adds query and Storage API projection, but an empty selection without required fields expands to a full-table read. | | posthog/temporal/data_imports/sources/snowflake/snowflake.py | Adds projected SELECT clauses, but valid Snowflake identifiers outside the shared allowlist make selected-column syncs fail. | | posthog/temporal/data_imports/sources/mssql/mssql.py | Projects SQL and Arrow metadata consistently, while sharing the empty-selection and strict-identifier failures. | | posthog/temporal/data_imports/sources/mysql/mysql.py | Extends the query builder and Arrow schema to selected columns, while sharing the empty-selection and strict-identifier failures. | | posthog/temporal/data_imports/sources/redshift/redshift.py | Adds composable projected queries and matching Arrow metadata; it shares the empty-selection fallback but safely quotes broader identifiers. | | products/data_warehouse/backend/api/external_data_source.py | Exposes capabilities, persists SQL metadata, reconciles refreshed schemas, and validates CDC configuration before remote setup. | | products/data_warehouse/frontend/shared/components/forms/SchemaForm.tsx | Gates wizard column controls on connector capability and integrates the shared selection picker. | | products/data_warehouse/frontend/scenes/SchemaScene/SchemaScene.tsx | Shows the existing-source columns section according to the serialized backend capability. | </details> <details open><summary><h3>Flowchart</h3></summary> ```mermaid %%{init: {'theme': 'neutral'}}%% flowchart TD A[Wizard source capability] --> B[Show column picker] B --> C[Persist enabled_columns] C --> D[Build SourceInputs] D --> E[compute_projected_columns] E --> F[Retain selected columns] E --> G[Append primary keys] E --> H[Append incremental field] E --> I[SQL source query] I --> J[BigQuery] I --> K[MSSQL / MySQL] I --> L[Redshift] I --> M[Snowflake] J --> N[Projected Arrow data] K --> N L --> N M --> N ``` </details> <!-- greptile_other_comments_section --> <sub>Reviews (1): Last reviewed commit: ["Update frontend/src/queries/schema.json ..."](https://github.com/drape-io/crbench-greptile-2026-09/commit/509622939ba1a661dbc1da8575196644b7660e9a) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=64284769)</sub>
Mjolnir (Opus 5.5) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✗ —
valid · impact medium
format_projected_select_clause calls quoter.quote, which runs _validate_identifier (allowlist: alphanumerics plus ._-$@). Column names containing spaces therefore raise InvalidIdentifierError. Before this change those tables synced with SELECT *, and the change both turns on the picker and wires up the projection. [grounded: diff] [consensus 3/3: valid=3]
valid · impact medium
The old code ran the duplicate check only inside `if primary_keys is None and "id" in table`. The new condition is also true when the table declares `id` as its primary key, so the duplicate scan now runs in cases it did not run before. [grounded: diff] [consensus 3/3: valid=3]
valid · impact medium
This is the same issue as finding 1: the duplicate check now also runs when `id` is the declared primary key, not only when it is the fallback. This adds a GROUP BY scan and changes the duplicate-key flag for those tables. [grounded: diff] [consensus 3/3: valid=3]
valid · impact medium
The MSSQL, BigQuery and Snowflake projections quote each column through the allowlist validator, so non-allowlisted column names make query building raise once columns are selected. Before the change these tables synced with SELECT *. [grounded: diff] [consensus 3/3: valid=3]
raw comments (4)
posthog/temporal/data_imports/sources/snowflake/snowflake.py:80
[P2] Column projection rejects column names with spaces or other non-allowlisted characters, so syncs fail. (same pattern at mssql/mssql.py:91-92 and bigquery/bigquery.py:384-385). `format_projected_select_clause` quotes each projected column through `_BaseQuoter.quote`, which calls `_validate_identifier`. That check only allows alphanumerics plus `._-$@`, so a column name such as `Order Date`, `Amount (USD)` or `col#1` raises `InvalidIdentifierError`. Before this change, Snowflake passed the table through `IDENTIFIER(%s)` and BigQuery used hand-written backticks, so these columns synced fine with `SELECT *`. Postgres and Redshift avoid the problem by using `psycopg.sql.Identifier`. Now that `SQLSource.supports_column_selection` defaults to `True`, the picker appears for these sources. As soon as a user picks columns on a table where any enabled column, primary key or incremental column has such a name, every sync of that table errors out. Space-containing column names are common in MSSQL, Snowflake and BigQuery, so this is a realistic case. For these dialects, projected column names need proper escaping (doubling the delimiter character) rather than the strict allowlist
posthog/temporal/data_imports/sources/redshift/redshift.py:852
[P2] Redshift now runs the duplicate-primary-key scan on tables whose declared primary key is `id`. Before this change, the `has_duplicate_primary_keys` GROUP BY/HAVING scan only ran when the code fell back to `["id"]` because no primary key was declared. The new condition `primary_keys == ["id"] and "id" in full_table` is also true when `get_primary_keys_for_table` returns a declared `["id"]` key, which contradicts the inline comment. On large tables with a declared `id` key, every sync now runs an extra full-table aggregation under the 10-minute `statement_timeout`. If that times out on a non-incremental sync, the `QueryCanceled` is re-raised and the sync fails where it used to succeed. Separately, Redshift doesn't enforce primary keys, so a duplicate result now changes `has_duplicate_primary_keys` for those tables. A boolean flag recording whether the fallback actually happened would keep the old behavior
PR-level
[P2] Redshift now runs the duplicate-key check for real `id` primary keys, not just the fallback posthog/temporal/data_imports/sources/redshift/redshift.py:851-857 The `id` fallback was moved earlier, and the duplicate check is now gated on `primary_keys == ["id"] and "id" in full_table`. That condition is also true when Redshift itself reports `id` as the declared primary key, so `has_duplicate_primary_keys` now runs for those tables too. Before this change it only ran when `get_primary_keys_for_table` returned `None`. The inline comment says the check should only run for the fallback, so this looks unintended. It has two effects. First, every sync of such a table runs an extra full-table `GROUP BY id HAVING COUNT(*) › 1`. That query falls under the 10-minute statement timeout, and on large incremental tables a cancellation surfaces as `QueryTimeoutException`. Second, Redshift does not enforce primary keys, so a table with a declared `id` key can contain duplicates. Incremental syncs of those tables used to proceed; now `validate_incremental_sync` will raise `DuplicatePrimaryKeysException`. Tracking whether the fallback was actually used (for example, with a flag set where `primary_keys` is assigned `["id"]`) would restore the old behavior.
PR-level
[P2] Selecting columns breaks syncs on MSSQL/MySQL/Snowflake/BigQuery tables with column names outside the allowlist posthog/temporal/data_imports/sources/mssql/mssql.py:204-206 `format_projected_select_clause` sends every projected column through `IdentifierQuoter.quote`. That method calls `_validate_identifier`, which only allows alphanumerics plus `._-$@`. When `enabled_columns` is set, the column list includes every selected column, not only the one the user unchecked. So if any selected column has a space or another legal-but-unlisted character (for example `[Order Date]` in MSSQL, a quoted identifier in Snowflake, or a flexible column name in BigQuery), building the query raises `InvalidIdentifierError` and the whole table sync fails. With `SELECT *` (`enabled_columns=None`) the same table synced fine. This PR makes `supports_column_selection` default to `True` for every `SQLSource`, so the picker is now offered for these sources, and a user who deselects a single column can turn a working sync into a failing one. The same path is used by `_bq_select_clause` (bigquery.py), the Snowflake `_build_query`, and MySQL's `SelectQueryBuilder.select_all`. Redshift is not affected because it uses `psycopg.sql.Identifier`.
Mjolnir (Sonnet 5.5) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✗ —
valid · impact medium
Before the change, the duplicate check ran only inside the `primary_keys is None` fallback block. The new condition also matches a real `id` PK reported by Redshift, so those tables now get an extra scan and their duplicate-key result can change. [grounded: diff] [consensus 3/3: valid=3]
valid · impact medium
This is the same issue as finding 0. The PK is now resolved before this check, so the check can no longer tell a fallback `id` from a real `id` primary key. The new behavior contradicts the comment next to it. [grounded: diff] [consensus 3/3: valid=3]
raw comments (2)
posthog/temporal/data_imports/sources/redshift/redshift.py:850
[P3] Redshift duplicate-PK check now also runs for real single-column `id` primary keys The comment says it only checks duplicates when the `id` fallback was used. The new condition `primary_keys == ["id"] and "id" in full_table` is also true when Redshift reports a genuine primary key named `id`, because `primary_keys` was already resolved. Before this change the check ran only when `primary_keys is None`. Every sync of a Redshift table whose declared PK is `id` now runs the extra `has_duplicate_primary_keys` query, which can be a full-table scan. Redshift doesn't enforce PKs, so a positive result could also change merge behavior for those tables. Tracking whether the fallback was applied (a separate boolean) would keep the old behavior
posthog/temporal/data_imports/sources/redshift/redshift.py:852
[P3] Redshift duplicate-PK check now runs for real `id` primary keys The old code ran `has_duplicate_primary_keys` only when `get_primary_keys_for_table` returned `None` and the `id` fallback was used. The new condition `primary_keys == ["id"] and "id" in full_table` is also true when the table has a real single-column primary key named `id`. Those tables now get the extra duplicate-key scan, and `has_duplicate_primary_keys` can now be set for them, which changes how the pipeline merges them. The comment says this only happens on the fallback, but the code no longer matches it. Resolving the fallback before projection needs a separate flag, such as `used_id_fallback`, to keep the old behavior.
qodo · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 high ✗ —
valid · impact low
compute_projected_columns returns None when enabled_columns is [] and there are no PKs or incremental field, and format_projected_select_clause renders None as '*'. The new MySQL/MSSQL/Redshift tests lock this in. The change switches on the column picker for every SQL source, and the schema serializer accepts an empty list, so a user who deselects every column on a table with no key still has all columns synced. The tests show this is a deliberate fallback, but they don't refute the user-facing outcome. [grounded: diff] [consensus 3/4: valid=3, invalid=1]
valid · impact medium
The new projection paths take persisted enabled_columns straight from inputs and put them into the SELECT list without intersecting them with the freshly fetched metadata. project_arrow_columns trims only the Arrow schema, not the SQL. Until reconcile_schema_metadata prunes the list on refresh_schemas, a column dropped upstream makes the source query fail. Before this change these sources used SELECT *, which could not hit this failure. [grounded: diff] [consensus 3/3: valid=3]
valid · impact medium
The old code ran has_duplicate_primary_keys only inside the fallback branch (primary_keys was None). The new condition also matches a genuinely discovered ['id'] PK, so those tables now run an extra GROUP BY scan. has_duplicate_primary_keys re-raises QueryCanceled, so a timeout during that scan can abort the build. [grounded: diff] [consensus 3/3: valid=3]
raw comments (5)
posthog/temporal/data_imports/sources/common/sql/base.py:56
<img src="https://img.shields.io/badge/High-634FD1?style=flat-square" height="20px" alt="Action required"> 1\. Empty selections still sync every column <code>🐞 Bug</code> <code>≡ Correctness</code> <pre> <b><i>compute_projected_columns</i></b> converts an empty selection without a primary key or incremental field to <b><i>None</i></b>, which every new query path renders as <b><i>SELECT *</i></b>. When a user deselects every column from a full-refresh table lacking a detected key, all columns—including those explicitly disabled—continue to be ingested. </pre> <details> <summary><strong>Agent Prompt</strong></summary> ``` ## Issue description Column selection is now enabled for every SQL source, but an empty selection on a table without retained key or incremental columns becomes `SELECT *`, causing explicitly disabled columns to be synced. ## Fix Focus Areas - posthog/temporal/data_imports/sources/common/sql/base.py[56-56] - products/data_warehouse/frontend/scenes/SchemaScene/ConfigurationTab.tsx[444-453] - products/data_warehouse/backend/api/external_data_source.py[1229-1237] ## Recommended Fix Validate column selections in both create/update APIs and prevent saving an empty selection when the table has no primary-key or incremental columns. Keep `None` as the only representation for syncing all columns, and add tests proving that `[]` can never silently become `SELECT *`. ``` <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> </details>
posthog/temporal/data_imports/sources/mysql/mysql.py:746
<img src="https://img.shields.io/badge/Medium-634FD1?style=flat-square" height="20px" alt="Remediation recommended"> 2\. Removed columns break scheduled syncs <code>🐞 Bug</code> <code>☼ Reliability</code> <pre> The new query paths pass persisted <b><i>enabled_columns</i></b> directly to <b><i>compute_projected_columns</i></b> without intersecting them with current source metadata. If an upstream column is removed after discovery but before metadata reconciliation, each scheduled sync continues selecting the nonexistent identifier and fails at the source database or BigQuery Storage API. </pre> <details> <summary><strong>Agent Prompt</strong></summary> ``` ## Issue description Persisted enabled-column names are interpolated into source queries even when they no longer exist, so schema drift can break scheduled synchronization before reconciliation removes stale selections. ## Fix Focus Areas - posthog/temporal/data_imports/sources/mysql/mysql.py[744-746] - posthog/temporal/data_imports/sources/mssql/mssql.py[586-588] - posthog/temporal/data_imports/sources/redshift/redshift.py[799-801] - posthog/temporal/data_imports/sources/bigquery/bigquery.py[729-733] - posthog/temporal/data_imports/sources/snowflake/snowflake.py[80-82] ## Recommended Fix Before constructing each projection, intersect enabled columns with freshly fetched table metadata while retaining valid primary-key and incremental fields. Persist or log removed names, and return a clear validation error rather than falling back to `SELECT *` when no valid projected columns remain. ``` <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> </details>
posthog/temporal/data_imports/sources/redshift/redshift.py:854
<img src="https://img.shields.io/badge/Medium-634FD1?style=flat-square" height="20px" alt="Remediation recommended"> 3\. Valid redshift keys trigger full scans <code>🐞 Bug</code> <code>➹ Performance</code> <pre> The new duplicate check treats every discovered single-column <b><i>id</i></b> primary key as though it came from the fallback because it tests only <b><i>primary_keys == [&quot;id&quot;]</i></b>. Redshift tables with a genuine <b><i>id</i></b> key now run an unnecessary grouped duplicate scan during every pipeline build, which can consume the ten-minute statement timeout and abort synchronization. </pre> <details> <summary><strong>Agent Prompt</strong></summary> ``` ## Issue description Redshift duplicate checking is intended only for an inferred `id` key, but the condition also matches genuine primary keys named `id` and performs an expensive table-wide grouped scan. ## Fix Focus Areas - posthog/temporal/data_imports/sources/redshift/redshift.py[794-797] - posthog/temporal/data_imports/sources/redshift/redshift.py[852-857] ## Recommended Fix Record an explicit boolean when the `id` fallback is applied and guard `has_duplicate_primary_keys` with that boolean. Add tests distinguishing a discovered `["id"]` primary key from a fallback inferred from table columns. ``` <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> </details>
PR-level
<h3>PR Summary by Qodo</h3> Expand SQL column selection and validate CDC source creation <code>✨ Enhancement</code> <code>🐞 Bug fix</code> <code>🧪 Tests</code> <code>🕐 40+ Minutes</code> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <details> <summary>AI Description</summary> <dl> <dd> <br/> ><pre> >• Enable projected column syncs across all supported SQL data import connectors. >• Expose connector capabilities and schema metadata to frontend column pickers. >• Reject invalid CDC configurations before creating replication resources. ></pre> </dd> </dl> </details> <details> <summary>Diagram</summary> <dl> <dd> <br/> ```mermaid graph TD F["Frontend Wizard"] --> A["Warehouse API"] --> S["SQL Sources"] --> P["Projection Rules"] --> Q["Driver Queries"] --> D[("Source Databases")] S --> M[("Schema Metadata")] --> A ``` </dd> </dl> </details> <details> <summary>High-Level Assessment</summary> <dl> <dd> <br/> >The following are alternative approaches to this PR: <details> <summary><b>1. Explicit per-connector capability opt-in</b></summary> <dl> <dd> - ➕ Prevents future SQL drivers from advertising unsupported behavior automatically. - ➕ Allows connector-by-connector rollout and rollback. - ➖ Duplicates capability declarations across every SQL source. - ➖ Can hide working functionality when new projection implementations forget to opt in. </dd> </dl> </details> <details> <summary><b>2. Filter columns after fetching rows</b></summary> <dl> <dd> - ➕ Avoids modifying driver-specific query builders. - ➕ Provides one projection point in the Arrow processing layer. - ➖ Transfers and scans every source column, eliminating the main efficiency benefit. - ➖ Risks mismatches between cursor output, Arrow schemas, and configured projections. </dd> </dl> </details> >**Recommendation:** Keep the PR&#x27;s shared SQL capability and projection helpers, with driver-native identifier quoting and query-time projection. All existing SQL drivers receive explicit query support, while primary keys and incremental cursors remain available for correctness; validating CDC before replication side effects is also the safest ordering. Consider requiring future SQL connectors to demonstrate projection tests before inheriting the default capability. </dd> </dl> </details> <details> <summary> Files changed (23) <code> +828 / -113 </code> </summary> <dl> <dd> <br/> <details> <summary>Enhancement (10) <code> +227 / -66 </code></summary> <dl> <dd> <details> <summary>bigquery.py<code>Project selected columns in BigQuery reads</code> <code>+38/-3</code></summary> <br/> >Project selected columns in BigQuery reads > ><pre> >• Applies safely backtick-quoted projections to incremental query paths and &#x27;selected_fields&#x27; to direct Storage API sessions. Primary keys and incremental fields are retained automatically. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-b49c850a60a412e3aedb2b4a8343a7a17d894baadf1c2548935aae0d92d6da37'>posthog/temporal/data_imports/sources/bigquery/bigquery.py</a> </details> <details> <summary>base.py<code>Define a default source column-selection capability</code> <code>+4/-0</code></summary> <br/> >Define a default source column-selection capability > ><pre> >• Adds a false &#x27;supports_column_selection&#x27; default to the common source base so non-SQL connectors remain unsupported unless explicitly enabled. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-81cb3aac59a0e4877226ab6ee6927d60421abdc05c2c5bb67fbea32be794ce9a'>posthog/temporal/data_imports/sources/common/base.py</a> </details> <details> <summary>base.py<code>Enable column selection for SQL sources by default</code> <code>+1/-1</code></summary> <br/> >Enable column selection for SQL sources by default > ><pre> >• Changes the shared SQL source capability default to true, allowing all implemented SQL connectors to advertise column selection. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-ebc14bcd3b325c3701c0d1358df22d0a7aa863f002957464ab25487dfe70feae'>posthog/temporal/data_imports/sources/common/sql/base.py</a> </details> <details> <summary>mssql.py<code>Apply column projections to MSSQL pipelines</code> <code>+30/-10</code></summary> <br/> >Apply column projections to MSSQL pipelines > ><pre> >• Builds bracket-quoted projected SELECT lists and filters Arrow metadata to match returned columns. Primary-key fallback now occurs before query and schema construction. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-74913746b52a51a5a211b2738a3bec518e79034a6395137ce06f7ee75877050a'>posthog/temporal/data_imports/sources/mssql/mssql.py</a> </details> <details> <summary>mysql.py<code>Apply column projections to MySQL pipelines</code> <code>+25/-8</code></summary> <br/> >Apply column projections to MySQL pipelines > ><pre> >• Threads enabled columns and primary keys through full, incremental, probe, and streaming queries. Projects Arrow metadata consistently after resolving fallback primary keys. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-7e850befda815dbe5df535bcc53a3d7d0378e2bdaccd45756499a3734d312858'>posthog/temporal/data_imports/sources/mysql/mysql.py</a> </details> <details> <summary>redshift.py<code>Apply column projections to Redshift pipelines</code> <code>+72/-32</code></summary> <br/> >Apply column projections to Redshift pipelines > ><pre> >• Builds projections with &#x27;psycopg.sql.Identifier&#x27; for sampled, full, and incremental queries. Resolves primary keys before projecting Arrow metadata and constructing queries. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-a437d4e1e73eeeb69322827ddc073a31eda0d8071a1b0e00fea1108ed6a366a9'>posthog/temporal/data_imports/sources/redshift/redshift.py</a> </details> <details> <summary>snowflake.py<code>Apply column projections to Snowflake pipelines</code> <code>+28/-7</code></summary> <br/> >Apply column projections to Snowflake pipelines > ><pre> >• Generates safely ANSI-quoted SELECT lists and passes selected columns and detected primary keys through row-count and streaming queries. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-3cdf1d61467915f6b5bfa4b3ea4549ee61b22a09357b1a014504ef88fbcfb4e9'>posthog/temporal/data_imports/sources/snowflake/snowflake.py</a> </details> <details> <summary>public_source_configs.py<code>Publish connector column-selection capabilities</code> <code>+5/-1</code></summary> <br/> >Publish connector column-selection capabilities > ><pre> >• Adds each source implementation&#x27;s &#x27;supports_column_selection&#x27; value to public source configuration responses. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-0c711aceebb3a9406a75beae21f181b8803605de6d044db0edfa5b13fa08904b'>products/data_warehouse/backend/api/public_source_configs.py</a> </details> <details> <summary>ConfigurationTab.tsx<code>Allow schema refresh when columns are undiscovered</code> <code>+22/-3</code></summary> <br/> >Allow schema refresh when columns are undiscovered > ><pre> >• Replaces the unavailable message with an empty state and a permission-aware action to pull schemas, including refresh loading feedback. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-7a6484a9296a41a06d62ef9cd56ad65c72658af35147a4e0b4ef679ac586cc97'>products/data_warehouse/frontend/scenes/SchemaScene/ConfigurationTab.tsx</a> </details> <details> <summary>SchemaForm.tsx<code>Gate wizard column controls by connector capability</code> <code>+2/-1</code></summary> <br/> >Gate wizard column controls by connector capability > ><pre> >• Shows per-column synchronization controls only outside direct-query mode and when the selected connector advertises support. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-3d62ed0786e85ef109cbc3fea6b31acba5385fdbb885c1d50bd8dc41ba3667ca'>products/data_warehouse/frontend/shared/components/forms/SchemaForm.tsx</a> </details> </dd> </dl> </details> <details> <summary>Bug fix (1) <code> +1 / -1 </code></summary> <dl> <dd> <details> <summary>SchemaScene.tsx<code>Show columns section using connector capability</code> <code>+1/-1</code></summary> <br/> >Show columns section using connector capability > ><pre> >• Displays schema column configuration whenever the source supports column selection, even before any columns have been discovered. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-3f297f3d7643c0f0f61016450e32a96994fa99c0ddb026af429f033bbf07309a'>products/data_warehouse/frontend/scenes/SchemaScene/SchemaScene.tsx</a> </details> </dd> </dl> </details> <details> <summary>Refactor (1) <code> +0 / -2 </code></summary> <dl> <dd> <details> <summary>source.py<code>Inherit Postgres column-selection capability</code> <code>+0/-2</code></summary> <br/> >Inherit Postgres column-selection capability > ><pre> >• Removes the redundant Postgres-specific capability declaration now that &#x27;SQLSource&#x27; enables column selection by default. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-16f60e4bc87e5df37503c4b8996adb60616d146d2b46bbadcc0fd01bb841486b'>posthog/temporal/data_imports/sources/postgres/source.py</a> </details> </dd> </dl> </details> <details> <summary>Tests (7) <code> +476 / -1 </code></summary> <dl> <dd> <details> <summary>test_source.py<code>Test BigQuery projections and identifier safety</code> <code>+65/-1</code></summary> <br/> >Test BigQuery projections and identifier safety > ><pre> >• Covers projection ordering, fallback behavior, incremental cursor retention, generated queries, and rejection of malicious column identifiers. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-5d24fdb7021da188215cce9ccb207375a2f3902720475d87d066ee577bb659f3'>posthog/temporal/data_imports/sources/bigquery/tests/test_source.py</a> </details> <details> <summary>test_mssql.py<code>Test MSSQL selected-column queries</code> <code>+39/-0</code></summary> <br/> >Test MSSQL selected-column queries > ><pre> >• Verifies full and empty selections, primary-key retention, and incremental-field retention with bound incremental values. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-ad1fd3b7c7fbe3d097ac5bcffe719a2201672e5206f01c5f7be47120b143fc72'>posthog/temporal/data_imports/sources/mssql/tests/test_mssql.py</a> </details> <details> <summary>test_mysql.py<code>Test MySQL selected-column queries</code> <code>+43/-0</code></summary> <br/> >Test MySQL selected-column queries > ><pre> >• Covers wildcard fallback, selected columns, mandatory primary keys, and incremental cursor retention. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-d6a66db94edbc117d0a81a5471d34b4bd314477b323bd8f0045af4af94bbcc8b'>posthog/temporal/data_imports/sources/mysql/tests/test_mysql.py</a> </details> <details> <summary>test_redshift.py<code>Test Redshift selected-column queries</code> <code>+43/-0</code></summary> <br/> >Test Redshift selected-column queries > ><pre> >• Verifies wildcard behavior, quoted projections, primary-key retention, and incremental-field inclusion. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-646e947313f4b66804c336b123328e3b5abe9a3010d2afaf40b83a80b9126e40'>posthog/temporal/data_imports/sources/redshift/tests/test_redshift.py</a> </details> <details> <summary>test_snowflake.py<code>Test Snowflake selected-column queries</code> <code>+43/-0</code></summary> <br/> >Test Snowflake selected-column queries > ><pre> >• Covers wildcard selection, quoted subsets, retained primary keys and incremental fields, and unchanged parameter binding. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-be3ecf515c5bb3b7350b949ee5496326a011c52b9ee4033b8f2a988ef141671d'>posthog/temporal/data_imports/sources/snowflake/tests/test_snowflake.py</a> </details> <details> <summary>test_external_data_schema.py<code>Test SQL source column metadata and capabilities</code> <code>+93/-0</code></summary> <br/> >Test SQL source column metadata and capabilities > ><pre> >• Parameterized API tests confirm available columns work across all supported SQL sources, missing metadata returns an empty list, and non-SQL sources remain unsupported. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-4d2d5d623fb559682e4765ae8517efa5faec32f0ce5e2c996ae839e1221d563b'>products/data_warehouse/backend/api/test/test_external_data_schema.py</a> </details> <details> <summary>test_external_data_source.py<code>Test CDC source-creation rejection paths</code> <code>+150/-0</code></summary> <br/> >Test CDC source-creation rejection paths > ><pre> >• Ensures CDC schemas are rejected when source-level CDC is disabled or a selected table lacks a primary key. Confirms failed validation removes source records and avoids replication side effects. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-91fd2d1149cfbceb7708caef6d9d4d93bfc7a09992808d49ce6098acd3ff4aaa'>products/data_warehouse/backend/api/test/test_external_data_source.py</a> </details> </dd> </dl> </details> <details> <summary>Other (4) <code> +124 / -43 </code></summary> <dl> <dd> <details> <summary>schema.json<code>Add column-selection capability to generated JSON schema</code> <code>+5/-0</code></summary> <br/> >Add column-selection capability to generated JSON schema > ><pre> >• Adds the optional &#x27;supportsColumnSelection&#x27; source configuration property with a false default and API-facing documentation. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-8f817b85ae0a280ff57e0d0df33adcb9670acc0a55e78bb92a4c197babc2158e'>frontend/src/queries/schema.json</a> </details> <details> <summary>schema-general.ts<code>Expose column-selection capability in frontend types</code> <code>+8/-0</code></summary> <br/> >Expose column-selection capability in frontend types > ><pre> >• Extends &#x27;SourceConfig&#x27; with the generated optional &#x27;supportsColumnSelection&#x27; boolean used by the source wizard. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-cda5dd489794f26592a54d50e02265377df53e48ac46b102acaac6d8cc27eed6'>frontend/src/queries/schema/schema-general.ts</a> </details> <details> <summary>schema.py<code>Add column-selection capability to backend schema model</code> <code>+9/-0</code></summary> <br/> >Add column-selection capability to backend schema model > ><pre> >• Adds &#x27;supportsColumnSelection&#x27; to the generated Pydantic &#x27;SourceConfig&#x27; model with a false default. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-ce21cf622cc2afb01211fa45b21e9ad52d5aa578e9c18ba26e54c5a4b60f351e'>posthog/schema.py</a> </details> <details> <summary>external_data_source.py<code>Expose SQL capabilities, metadata, and safer CDC validation</code> <code>+102/-43</code></summary> <br/> >Expose SQL capabilities, metadata, and safer CDC validation > ><pre> >• Publishes column-selection support, persists and reconciles schema metadata for supported SQL sources, and generalizes metadata handling beyond Postgres. Source creation now rejects disabled CDC or CDC tables without primary keys before creating slots and publications. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-27fa7cf4c41297f5c8a4fd7fade882c2d629dcacd2c8cb3f7040d3361c1230bd'>products/data_warehouse/backend/api/external_data_source.py</a> </details> </dd> </dl> </details> </dd> </dl> </details>
PR-level
<h3>Code Review by Qodo</h3> <code>🐞 Bugs (3)</code> <code>📘 Rule violations (0)</code> <code>📜 Skill insights (0)</code> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <br/> <img src="https://img.shields.io/badge/High-634FD1?style=flat-square" height="20px" alt="Action required"> <details> <summary> 1. Empty selections still sync every column <code>🐞 Bug</code> <code>≡ Correctness</code></summary> <br/> > <details open> ><summary>Description</summary> ><br/> > ><pre> ><b><i>compute_projected_columns</i></b> converts an empty selection without a primary key or incremental field >to <b><i>None</i></b>, which every new query path renders as <b><i>SELECT *</i></b>. When a user deselects every column from >a full-refresh table lacking a detected key, all columns—including those explicitly >disabled—continue to be ingested. ></pre> ></details> > <details> ><summary>Code</summary> ><br/> > ><code>[posthog/temporal/data_imports/sources/common/sql/base.py[56]](https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-ebc14bcd3b325c3701c0d1358df22d0a7aa863f002957464ab25487dfe70feaeR56-R56)</code> > >```diff >+ supports_column_selection: bool = True >``` ></details> > <details> ><summary>Evidence</summary> ><br/> > ><pre> >The PR enables the picker for all SQL sources, while the shared projection helper returns <b><i>None</i></b> >when an empty selection has no retained fields and query formatters interpret <b><i>None</i></b> as <b><i>*</i></b>. The >picker permits all non-retained columns to be unchecked and saves the resulting selection without >enforcing at least one retained column. ></pre> > > <code>[posthog/temporal/data_imports/sources/common/sql/base.py[53-56]](https://github.com/drape-io/crbench-qodo-2026-09/blob/2a93965e322c2eaa8c291cc45ca94c395a30f520/posthog/temporal/data_imports/sources/common/sql/base.py/#L53-L56)</code> > <code>[posthog/temporal/data_imports/sources/common/sql/projection.py[23-48]](https://github.com/drape-io/crbench-qodo-2026-09/blob/2a93965e322c2eaa8c291cc45ca94c395a30f520/posthog/temporal/data_imports/sources/common/sql/projection.py/#L23-L48)</code> > <code>[posthog/temporal/data_imports/sources/common/sql/projection.py[51-58]](https://github.com/drape-io/crbench-qodo-2026-09/blob/2a93965e322c2eaa8c291cc45ca94c395a30f520/posthog/temporal/data_imports/sources/common/sql/projection.py/#L51-L58)</code> > <code>[products/data_warehouse/frontend/scenes/SourceScene/tabs/ColumnSelectionModal.tsx[160-166]](https://github.com/drape-io/crbench-qodo-2026-09/blob/2a93965e322c2eaa8c291cc45ca94c395a30f520/products/data_warehouse/frontend/scenes/SourceScene/tabs/ColumnSelectionModal.tsx/#L160-L166)</code> > <code>[products/data_warehouse/frontend/scenes/SourceScene/tabs/ColumnSelectionModal.tsx[209-215]](https://github.com/drape-io/crbench-qodo-2026-09/blob/2a93965e322c2eaa8c291cc45ca94c395a30f520/products/data_warehouse/frontend/scenes/SourceScene/tabs/ColumnSelectionModal.tsx/#L209-L215)</code> ></details> > <details> ><summary>Agent prompt</summary> ><br/> > >``` >The issue below was found during a code review. Follow the provided context and guidance below and implement a solution > >## Issue description >Column selection is now enabled for every SQL source, but an empty selection on a table without retained key or incremental columns becomes `SELECT *`, causing explicitly disabled columns to be synced. > >## Fix Focus Areas >- posthog/temporal/data_imports/sources/common/sql/base.py[56-56] >- products/data_warehouse/frontend/scenes/SchemaScene/ConfigurationTab.tsx[444-453] >- products/data_warehouse/backend/api/external_data_source.py[1229-1237] > >## Recommended Fix >Validate column selections in both create/update APIs and prevent saving an empty selection when the table has no primary-key or incremental columns. Keep `None` as the only representation for syncing all columns, and add tests proving that `[]` can never silently become `SELECT *`. >``` > <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> ></details> <hr/> </details> <br/> <img src="https://img.shields.io/badge/Medium-634FD1?style=flat-square" height="20px" alt="Remediation recommended"> <details> <summary> 2. Removed columns break scheduled syncs <code>🐞 Bug</code> <code>☼ Reliability</code></summary> <br/> > <details open> ><summary>Description</summary> ><br/> > ><pre> >The new query paths pass persisted <b><i>enabled_columns</i></b> directly to <b><i>compute_projected_columns</i></b> without >intersecting them with current source metadata. If an upstream column is removed after discovery but >before metadata reconciliation, each scheduled sync continues selecting the nonexistent identifier >and fails at the source database or BigQuery Storage API. ></pre> ></details> > <details> ><summary>Code</summary> ><br/> > ><code>[posthog/temporal/data_imports/sources/mysql/mysql.py[R744-746]](https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-7e850befda815dbe5df535bcc53a3d7d0378e2bdaccd45756499a3734d312858R744-R746)</code> > >```diff >+ projected = compute_projected_columns(enabled_columns, primary_keys, incremental_field) >+ table = project_arrow_columns(full_table, projected) >+ arrow_schema = table.to_arrow_schema() >``` ></details> > <details> ><summary>Evidence</summary> ><br/> > ><pre> >The query helper preserves every supplied enabled-column name, whereas only the separate refresh >reconciliation path prunes missing names. MySQL, MSSQL, Redshift, Snowflake, and BigQuery feed the >unpruned list into their new query or storage projection paths, so a schema change between refresh >and sync reaches the database as an invalid field reference. ></pre> > > <code>[posthog/temporal/data_imports/sources/common/sql/projection.py[23-48]](https://github.com/drape-io/crbench-qodo-2026-09/blob/2a93965e322c2eaa8c291cc45ca94c395a30f520/posthog/temporal/data_imports/sources/common/sql/projection.py/#L23-L48)</code> > <code>[posthog/temporal/data_imports/sources/common/sql/base.py[182-197]](https://github.com/drape-io/crbench-qodo-2026-09/blob/2a93965e322c2eaa8c291cc45ca94c395a30f520/posthog/temporal/data_imports/sources/common/sql/base.py/#L182-L197)</code> > <code>[posthog/temporal/data_imports/sources/mysql/mysql.py[744-758]](https://github.com/drape-io/crbench-qodo-2026-09/blob/2a93965e322c2eaa8c291cc45ca94c395a30f520/posthog/temporal/data_imports/sources/mysql/mysql.py/#L744-L758)</code> > <code>[posthog/temporal/data_imports/sources/mssql/mssql.py[586-599]](https://github.com/drape-io/crbench-qodo-2026-09/blob/2a93965e322c2eaa8c291cc45ca94c395a30f520/posthog/temporal/data_imports/sources/mssql/mssql.py/#L586-L599)</code> > <code>[posthog/temporal/data_imports/sources/redshift/redshift.py[799-814]](https://github.com/drape-io/crbench-qodo-2026-09/blob/2a93965e322c2eaa8c291cc45ca94c395a30f520/posthog/temporal/data_imports/sources/redshift/redshift.py/#L799-L814)</code> > <code>[posthog/temporal/data_imports/sources/bigquery/bigquery.py[783-798]](https://github.com/drape-io/crbench-qodo-2026-09/blob/2a93965e322c2eaa8c291cc45ca94c395a30f520/posthog/temporal/data_imports/sources/bigquery/bigquery.py/#L783-L798)</code> > <code>[posthog/temporal/data_imports/sources/snowflake/snowflake.py[80-95]](https://github.com/drape-io/crbench-qodo-2026-09/blob/2a93965e322c2eaa8c291cc45ca94c395a30f520/posthog/temporal/data_imports/sources/snowflake/snowflake.py/#L80-L95)</code> ></details> > <details> ><summary>Agent prompt</summary> ><br/> > >``` >The issue below was found during a code review. Follow the provided context and guidance below and implement a solution > >## Issue description >Persisted enabled-column names are interpolated into source queries even when they no longer exist, so schema drift can break scheduled synchronization before reconciliation removes stale selections. > >## Fix Focus Areas >- posthog/temporal/data_imports/sources/mysql/mysql.py[744-746] >- posthog/temporal/data_imports/sources/mssql/mssql.py[586-588] >- posthog/temporal/data_imports/sources/redshift/redshift.py[799-801] >- posthog/temporal/data_imports/sources/bigquery/bigquery.py[729-733] >- posthog/temporal/data_imports/sources/snowflake/snowflake.py[80-82] > >## Recommended Fix >Before constructing each projection, intersect enabled columns with freshly fetched table metadata while retaining valid primary-key and incremental fields. Persist or log removed names, and return a clear validation error rather than falling back to `SELECT *` when no valid projected columns remain. >``` > <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> ></details> <hr/> </details> <details> <summary> 3. Valid Redshift keys trigger full scans <code>🐞 Bug</code> <code>➹ Performance</code></summary> <br/> > <details open> ><summary>Description</summary> ><br/> > ><pre> >The new duplicate check treats every discovered single-column <b><i>id</i></b> primary key as though it came >from the fallback because it tests only <b><i>primary_keys == [&quot;id&quot;]</i></b>. Redshift tables with a genuine ><b><i>id</i></b> key now run an unnecessary grouped duplicate scan during every pipeline build, which can >consume the ten-minute statement timeout and abort synchronization. ></pre> ></details> > <details> ><summary>Code</summary> ><br/> > ><code>[posthog/temporal/data_imports/sources/redshift/redshift.py[R852-854]](https://github.com/drape-io/crbench-qodo-2026-09/pull/64/files#diff-a437d4e1e73eeeb69322827ddc073a31eda0d8071a1b0e00fea1108ed6a366a9R852-R854)</code> > >```diff >+ if primary_keys == ["id"] and "id" in full_table: >+ # Only check dupes when we fell back to the `id` PK above. > logger.debug("Checking duplicate primary keys...") >``` ></details> > <details> ><summary>Evidence</summary> ><br/> > ><pre> >The fallback and a database-discovered primary key both produce the identical list <b><i>[&quot;id&quot;]</i></b>, so the >changed condition cannot distinguish them despite its comment. The called method executes `GROUP BY >... HAVING COUNT(*) &gt; 1` over the source table, and pipeline setup applies a ten-minute statement >timeout that propagates cancellation. ></pre> > > <code>[posthog/temporal/data_imports/sources/redshift/redshift.py[785-797]](https://github.com/drape-io/crbench-qodo-2026-09/blob/2a93965e322c2eaa8c291cc45ca94c395a30f520/posthog/temporal/data_imports/sources/redshift/redshift.py/#L785-L797)</code> > <code>[posthog/temporal/data_imports/sources/redshift/redshift.py[851-857]](https://github.com/drape-io/crbench-qodo-2026-09/blob/2a93965e322c2eaa8c291cc45ca94c395a30f520/posthog/temporal/data_imports/sources/redshift/redshift.py/#L851-L857)</code> > <code>[posthog/temporal/data_imports/sources/redshift/redshift.py[540-571]](https://github.com/drape-io/crbench-qodo-2026-09/blob/2a93965e322c2eaa8c291cc45ca94c395a30f520/posthog/temporal/data_imports/sources/redshift/redshift.py/#L540-L571)</code> > <code>[posthog/temporal/data_imports/sources/redshift/redshift.py[858-863]](https://github.com/drape-io/crbench-qodo-2026-09/blob/2a93965e322c2eaa8c291cc45ca94c395a30f520/posthog/temporal/data_imports/sources/redshift/redshift.py/#L858-L863)</code> ></details> > <details> ><summary>Agent prompt</summary> ><br/> > >``` >The issue below was found during a code review. Follow the provided context and guidance below and implement a solution > >## Issue description >Redshift duplicate checking is intended only for an inferred `id` key, but the condition also matches genuine primary keys named `id` and performs an expensive table-wide grouped scan. > >## Fix Focus Areas >- posthog/temporal/data_imports/sources/redshift/redshift.py[794-797] >- posthog/temporal/data_imports/sources/redshift/redshift.py[852-857] > >## Recommended Fix >Record an explicit boolean when the `id` fallback is applied and guard `has_duplicate_primary_keys` with that boolean. Add tests distinguishing a discovered `["id"]` primary key from a fallback inferred from table columns. >``` > <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> ></details> <hr/> </details> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <!-- qodo-context:start --> <details><summary><strong>Context sources</strong></summary> <div>Review mode: <code>🧠 Deep</code>: This broad, behavior-changing PR spans multiple SQL ingestion drivers, CDC/API validation paths, schema metadata, and frontend flows, with many independent logic sites where subtle regressions could be missed in one pass.</div> <!-- qodo-context:end --> </details> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <!-- qodo-daily-tip:start --> <details> <summary><strong>Tip of the day</strong></summary> <br/> <pre>💡 Did you know, you can reply &#x27;qodo&#x27; on any finding to push back, ask questions, or dig deeper</pre> <a href="https://docs.qodo.ai/tips-and-tricks">More tips ↗</a> | <a href="https://app.qodo.ai/configurations?tab=display-preferences">Customize Qodo ↗</a> | <a href="https://docs.qodo.ai">Qodo docs ↗</a> </details> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <!-- qodo-daily-tip:end --> <!-- https://github.com/drape-io/crbench-qodo-2026-09/commit/2a93965e322c2eaa8c291cc45ca94c395a30f520 --> <a href="https://www.qodo.ai"><img src="https://www.qodo.ai/wp-content/uploads/2025/03/qodo-logo.svg" width="80" alt="Qodo Logo"></a>