crbench//

← all cases

Serve LISTs with exact ResourceVersion and continuations from the cache

Tier 2 hard go · cross-file · real merged PR from kubernetes/kubernetes #130423

Ground-truth defects

d1 · correctness · medium
The cacher delegator's `GetList` adds an unconditional `if errors.IsResourceExpired(err) { return c.storage.GetList(...) }` fallback to etcd. When the cache snapshot is nil the cacher returns `ResourceExpired`, so consistent paginated LISTs silently fall back to etcd instead of being served from cache, defeating the feature and adding etcd load. The fix gates that fallback behind the cache-snapshot feature flag.
d2 · correctness · high
Limited consistent LISTs bypass the watch cache and hit etcd. For a consistent read the delegator sets `opts.ResourceVersion` to the current etcd RV, so in `watchCache.list` the new "legacy exact match" branch (`Limit > 0 && len(opts.ResourceVersion) > 0 && opts.ResourceVersion != "0"`) now matches and routes the request to `listExactRV`. With `ListFromCacheSnapshot` off (its default) the snapshot store is nil, so `listExactRV` returns `ResourceExpired` and the delegator falls back to etcd. The net effect is that in the default config every consistent LIST that carries a limit (e.g. `GET /pods?limit=500`) now bypasses the cache and queries etcd directly; even with snapshots enabled it drops the index-based prefiltering that `listLatestRV` would apply.
d3 · correctness · high
Continue requests are served from a cache snapshot without a freshness wait, so a lagging cache returns a stale, incomplete page. A continue request carries no explicit ResourceVersion (forbidden), so `Cacher.GetList` computes `listRV = 0` and `WaitUntilFreshAndList` does not block; the serve path then calls `listExactRV(key, continueKey, continueRV)` -> `snapshots.GetLessOrEqual(continueRV)`, which returns the greatest snapshot <= continueRV even when the cache has not reached continueRV, and labels the response `ResourceVersion = continueRV`. In an HA cluster a continue token minted by a fresh apiserver A and routed to an apiserver B whose watch cache lags continueRV yields an older snapshot's page mislabeled at continueRV, silently dropping objects created in (snapshotRV, continueRV].

The change under review

diff (truncated — full diff)
diff --git a/staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go b/staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go
index ef7f58fec9840..5d18d83133e51 100644
--- a/staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go
+++ b/staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go
@@ -1328,15 +1328,52 @@ func newErrWatcher(err error) *errWatcher {
 }
 
 func (c *Cacher) ShouldDelegateExactRV(resourceVersion string, recursive bool) (delegator.Result, error) {
-	return delegator.CacheWithoutSnapshots{}.ShouldDelegateExactRV(resourceVersion, recursive)
+	// Not Recursive is not supported unitl exact RV is implemented for WaitUntilFreshAndGet.
+	if !recursive || c.watchCache.snapshots == nil {
+		return delegator.Result{ShouldDelegate: true}, nil
+	}
+	listRV, err := c.versioner.ParseResourceVersion(resourceVersion)
+	if err != nil {
+		return delegator.Result{}, err
+	}
+	return c.shouldDelegateExactRV(listRV)
 }
 
 func (c *Cacher) ShouldDelegateContinue(continueToken string, recursive bool) (delegator.Result, error) {
-	return delegator.CacheWithoutSnapshots{}.ShouldDelegateContinue(continueToken, recursive)
+	// Not Recursive is not supported unitl exact RV is implemented for WaitUntilFreshAndGet.
+	if !recursive || c.watchCache.snapshots == nil {
+		return delegator.Result{ShouldDelegate: true}, nil
+	}
+	_, continueRV, err := storage.DecodeContinue(continueToken, c.resourcePrefix)
+	if err != nil {
+		return delegator.Result{}, err
+	}
+	if continueRV > 0 {
+		return c.shouldDelegateExactRV(uint64(continueRV))
+	} else {
+		// Continue with negative RV is a consistent read.
+		return c.ShouldDelegateConsistentRead()
+	}
+}
+
+func (c *Cacher) shouldDelegateExactRV(rv uint64) (delegator.Result, error) {
+	// Exact requests on future revision require support for consistent read, but are not a consistent read by themselves.
+	if c.watchCache.notFresh(rv) {
+		return delegator.Result{
+			ShouldDelegate: !delegator.ConsistentReadSupported(),
+		}, nil
+	}
+	_, canServe := c.watchCache.snapshots.GetLessOrEqual(rv)
+	return delegator.Result{
+		ShouldDelegate: !canServe,
+	}, nil
 }
 
 func (c *Cacher) ShouldDelegateConsistentRead() (delegator.Result, error) {
-	return delegator.CacheWithoutSnapshots{}.ShouldDelegateConsistentRead()
+	return delegator.Result{
+		ConsistentRead: true,
+		ShouldDelegate: !delegator.ConsistentReadSupported(),
+	}, nil
 }
 
 // Implements watch.Interface.
diff --git a/staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_test.go b/staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_test.go
index 9a6538446f537..fe8ca100823b9 100644
--- a/staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_test.go
+++ b/staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_test.go
@@ -179,20 +179,30 @@ func TestListPaging(t *testing.T) {
 func TestList(t *testing.T) {
 	for _, consistentRead := range []bool{true, false} {
 		t.Run(fmt.Sprintf("ConsistentListFromCache=%v", consistentRead), func(t *testing.T) {
-			featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ConsistentListFromCache, consistentRead)
-			ctx, cacher, server, terminate := testSetupWithEtcdServer(t)
-			t.Cleanup(terminate)
-			storagetesting.RunTestList(ctx, t, cacher, increaseRV(server.V3Client.Client), true)
+			for _, listFromCacheSnapshot := range []bool{true, false} {
+				t.Run(fmt.Sprintf("ListFromCacheSnapsthot=%v", listFromCacheSnapshot), func(t *testing.T) {
+					featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ListFromCacheSnapshot, listFromCacheSnapshot)
+					featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ConsistentListFromCache, consistentRead)
+					ctx, cacher, server, terminate := testSetupWithEtcdServer(t)
+					t.Cleanup(terminate)
+					storagetesting.RunTestList(ctx, t, cacher, increaseRV(server.V3Client.Client), true)
+				})
+			}
 		})
 	}
 }
 func TestConsistentList(t *testing.T) {
 	for _, consistentRead := range []bool{true, false} {
 		t.Run(fmt.Sprintf("ConsistentListFromCache=%v", consistentRead), func(t *testing.T) {
-			featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ConsistentListFromCache, consistentRead)
-			ctx, cacher, server, terminate := testSetupWithEtcdServer(t)
-			t.Cleanup(terminate)
-			storagetesting.RunTestConsistentList(ctx, t, cacher, increaseRV(server.V3Client.Client), true, consistentRead)
+			for _, listFromCacheSnapshot := range []bool{true, false} {
+				t.Run(fmt.Sprintf("ListFromCacheSnapsthot=%v", listFromCacheSnapshot), func(t *testing.T) {
+					featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ListFromCacheSnapshot, listFromCacheSnapshot)
+					featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ConsistentListFromCache, consistentRead)
+					ctx, cacher, server, terminate := testSetupWithEtcdServer(t)
+					t.Cleanup(terminate)
+					storagetesting.RunTestConsistentList(ctx, t, cacher, increaseRV(server.V3Client.Client), true, consistentRead, listFromCacheSnapshot)
+				})
+			}
 		})
 	}
 }
@@ -200,10 +210,15 @@ func TestConsistentList(t *testing.T) {
 func TestGetListNonRecursive(t *testing.T) {
 	for _, consistentRead := range []bool{true, false} {
 		t.Run(fmt.Sprintf("ConsistentListFromCache=%v", consistentRead), func(t *testing.T) {
-			featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ConsistentListFromCache, consistentRead)
-			ctx, cacher, server, terminate := testSetupWithEtcdServer(t)
-			t.Cleanup(terminate)
-			storagetesting.RunTestGetListNonRecursive(ctx, t, increaseRV(server.V3Client.Client), cacher)
+			for _, listFromCacheSnapshot := range []bool{true, false} {
+				t.Run(fmt.Sprintf("ListFromCacheSnapsthot=%v", listFromCacheSnapshot), func(t *testing.T) {
+					featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ListFromCacheSnapshot, listFromCacheSnapshot)
+					featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ConsistentListFromCache, consistentRead)
+					ctx, cacher, server, terminate := testSetupWithEtcdServer(t)
+					t.Cleanup(terminate)
+					storagetesting.RunTestGetListNonRecursive(ctx, t, increaseRV(server.V3Client.Client), cacher)
+				})
+			}
 		})
 	}
 }
diff --git a/staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_whitebox_test.go b/staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_whitebox_test.go
index ca53fde6a0572..6d6d5f8d1a07d 100644
--- a/staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_whitebox_test.go
+++ b/staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_whitebox_test.go
@@ -344,7 +344,7 @@ func TestShouldDelegateList(t *testing.T) {
 		}
 	}
 
-	runTestCases := func(t *testing.T, testcases map[opts]bool, overrides ...map[opts]bool) {
+	runTestCases := func(t *testing.T, snapshotAvailable bool, testcases map[opts]bool, overrides ...map[opts]bool) {
 		for opt, expectBypass := range testCases {
 			for _, override := range overrides {
 				if bypass, ok := override[opt]; ok {
@@ -362,6 +362,9 @@ func TestShouldDelegateList(t *testing.T) {
 				t.Fatalf("Couldn't create cacher: %v", err)
 			}
 			defer cacher.Stop()
+			if snapshotAvailable {
+				cacher.watchCache.snapshots.Add(uint64(mustAtoi(oldRV)), fakeOrderedLister{})
+			}
 			result, err := shouldDelegateList(toStorageOpts(opt), cacher)
 			if err != nil {
 				t.Fatal(err)
@@ -371,19 +374,46 @@ func TestShouldDelegateList(t *testing.T) {
 			}
 		}
 	}
-	consistentListFromCacheOverrides := map[opts]bool{}
-	for _, recursive := range []bool{true, false} {
-		consistentListFromCacheOverrides[opts{Recursive: recursive}] = false
-		consistentListFromCacheOverrides[opts{Limit: 100, Recursive: recursive}] = false
-	}
+
+	// Exacts and continue on current cache RV.
+	listFromSnapshotEnabledOverrides := map[opts]bool{}
+	listFromSnapshotEnabledOverrides[opts{Recursive: true, ResourceVersion: cacheRV, Limit: 100}] = false
+	listFromSnapshotEnabledOverrides[opts{Recursive: true, ResourceVersion: cacheRV, ResourceVersionMatch: metav1.ResourceVersionMatchExact}] = false
+	listFromSnapshotEnabledOverrides[opts{Recursive: true, ResourceVersion: cacheRV, Limit: 100, ResourceVersionMatch: metav1.ResourceVersionMatchExact}] = false
+	listFromSnapshotEnabledOverrides[opts{Recursive: true, ResourceVersion: "0", Limit: 100, Continue: continueOnCacheRV}] = false
+	listFromSnapshotEnabledOverrides[opts{Recursive: true, ResourceVersion: "0", Continue: continueOnCacheRV}] = false
+	listFromSnapshotEnabledOverrides[opts{Recursive: true, Limit: 100, Continue: continueOnCacheRV}] = false
+	listFromSnapshotEnabledOverrides[opts{Recursive: true, Continue: continueOnCacheRV}] = false
+
+	// Exacts and continue RV with a snapshot.
+	snapshotAvailableOverrides := map[opts]bool{}
+	snapshotAvailableOverrides[opts{Recursive: true, Continue: continueOnOldRV}] = false
+	snapshotAvailableOverrides[opts{Recursive: true, Limit: 100, Continue: continueOnOldRV}] = false
+	snapshotAvailableOverrides[opts{Recursive: true, ResourceVersion: "0", Continue: continueOnOldRV}] = false
+	snapshotAvailableOverrides[opts{Recursive: true, ResourceVersion: "0", Limit: 100, Continue: continueOnOldRV}] = false
+	snapshotAvailableOverrides[opts{Recursive: true, ResourceVersion: oldRV, Limit: 100}] = false
+	snapshotAvailableOverrides[opts{Recursive: true, ResourceVersion: oldRV, ResourceVersionMatch: metav1.ResourceVersionMatchExact}] = false
+	snapshotAvailableOverrides[opts{Recursive: true, ResourceVersion: oldRV, ResourceVersionMatch: metav1.ResourceVersionMatchExact, Limit: 100}] = false
 
 	t.Run("ConsistentListFromCache=false", func(t *testing.T) {
 		featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ConsistentListFromCache, false)
-		runTestCases(t, testCases)
+		t.Run("ListFromCacheSnapshot=false", func(t *testing.T) {
+			featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ListFromCacheSnapshot, false)
+			runTestCases(t, false, testCases)
+		})
+
+		t.Run("ListFromCacheSnapshot=true", func(t *testing.T) {
+			featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ListFromCacheSnapshot, true)
+			t.Run("SnapshotAvailable=false", func(t *testing.T) {
+				runTestCases(t, false, testCases, listFromSnapshotEnabledOverrides)
+			})
+			t.Run("SnapshotAvailable=true", func(t *testing.T) {
+				runTestCases(t, true, testCases, listFromSnapshotEnabledOverrides, snapshotAvailableOverrides)
+			})
+		})
 	})
 	t.Run("ConsistentListFromCache=true", func(t *testing.T) {
 		featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ConsistentListFromCache, true)
-
 		// TODO(p0lyn0mial): the following tests assume that etcdfeature.DefaultFeatureSupportChecker.Supports(storage.RequestWatchProgress)
 		// evaluates to true. Otherwise the cache will be bypassed and the test will fail.
 		//
@@ -391,7 +421,41 @@ func TestShouldDelegateList(t *testing.T) {
 		// However in CI all test are run and there must be a test(s) that properly
 		// initialize the storage layer so that the mentioned method evaluates to true
 		forceRequestWatchProgressSupport(t)
-		runTestCases(t, testCases, consistentListFromCacheOverrides)
+
+		consistentListFromCacheOverrides := map[opts]bool{}
+		for _, recursive := range []bool{true, false} {
+			consistentListFromCacheOverrides[opts{Recursive: recursive}] = false
+			consistentListFromCacheOverrides[opts{Limit: 100, Recursive: recursive}] = false
+		}
+
+		t.Run("ListFromCacheSnapshot=false", func(t *testing.T) {
+			featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ListFromCacheSnapshot, false)
+			runTestCases(t, false, testCases, consistentListFromCacheOverrides)
+		})
+		t.Run("ListFromCacheSnapshot=true", func(t *testing.T) {
+			featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ListFromCacheSnapshot, true)
+
+			consistentReadWithSnapshotOverrides := map[opts]bool{}
+			// Continues with negative RV are same as consistent read.
+			consistentReadWithSnapshotOverrides[opts{Recursive: true, Limit: 0, Continue: continueOnNegativeRV}] = false
+			consistentReadWithSnapshotOverrides[opts{Recursive: true, Limit: 100, Continue: continueOnNegativeRV}] = false
+			consistentReadWithSnapshotOverrides[opts{Recursive: true, ResourceVersion: "0", Limit: 0, Continue: continueOnNegativeRV}] = false
+			consistentReadWithSnapshotOverrides[opts{Recursive: true, ResourceVersion: "0", Limit: 100, Continue: continueOnNegativeRV}] = false
+			// Exact on RV not yet observed by cache
+			consistentReadWithSnapshotOverrides[opts{Recursive: true, ResourceVersion: etcdRV, Limit: 100}] = false
+			consistentReadWithSnapshotOverrides[opts{Recursive: true, ResourceVersion: etcdRV, ResourceVersionMatch: metav1.ResourceVersionMatchExact}] = false
+			consistentReadWithSnapshotOverrides[opts{Recursive: true, ResourceVersion: etcdRV, ResourceVersionMatch: metav1.ResourceVersionMatchExact, Limit: 100}] = false
+			consistentReadWithSnapshotOverrides[opts{Recursive: true, Continue: continueOnEtcdRV}] = false
+			consistentReadWithSnapshotOverrides[opts{Recursive: true, ResourceVersion: "0", Continue: continueOnEtcdRV}] = false
+			consistentReadWithSnapshotOverrides[opts{Recursive: true, Continue: continueOnEtcdRV, Limit: 100}] = false
+			consistentReadWithSnapshotOverrides[opts{Recursive: true, ResourceVersion: "0", Continue: continueOnEtcdRV, Limit: 100}] = false
+			t.Run("SnapshotAvailable=false", func(t *testing.T) {
+				runTestCases(t, false, testCases, listFromSnapshotEnabledOverrides, consistentListFromCacheOverrides, consistentReadWithSnapshotOverrides)
+			})
+			t.Run("SnapshotAvailable=true", func(t *testing.T) {
+				runTestCases(t, true, testCases, listFromSnapshotEnabledOverrides, consistentListFromCacheOverrides, consistentReadWithSnapshotOverrides, snapshotAvailableOverrides)
+			})
+		})
 	})
 }
 
@@ -568,6 +632,91 @@ apiserver_watch_cache_consistent_read_total{fallback="true", resource="pods", su
 	}
 }
 
+func TestMatchExactResourceVersionFallback(t *testing.T) {
+	tcs := []struct {
+		name               string
+		snapshotsAvailable []bool
+
+		expectStoreRequests    int
+		expectSnapshotRequests int
+	}{
+		{
+			name:                   "Disabled",
+			snapshotsAvailable:     []bool{false, false},
+			expectStoreRequests:    2,
+			expectSnapshotRequests: 1,
+		},
+		{
+			name:                   "Enabled",
+			snapshotsAvailable:     []bool{true, true},
+			expectStoreRequests:    1,
+			expectSnapshotRequests: 2,
+		},
+		{
+			name:                   "Fallback",
+			snapshotsAvailable:     []bool{true, false},
+			expectSnapshotRequests: 2,
+			expectStoreRequests:    2,
+		},
+	}
+	for _, tc := range tcs {
+		t.Run(tc.name, func(t *testing.T) {
+			backingStorage := &dummyStorage{}
+			expectStoreRequests := 0
+			backingStorage.getListFn = func(_ context.Context, key string, opts storage.ListOptions, listObj runtime.Object) error {
+				expectStoreRequests++
+				podList := listObj.(*example.PodList)
+				switch opts.ResourceVersionMatch {
+				case "":
+					podList.ResourceVersion = "42"
+				case metav1.ResourceVersionMatchExact:
+					podList.ResourceVersion = opts.ResourceVersion
+				}
+				return nil
+			}
+			cacher, _, err := newTestCacherWithoutSyncing(backingStorage, clock.RealClock{})
+			if err != nil {
+				t.Fatalf("Couldn't create cacher: %v", err)
+			}
+			defer cacher.Stop()
+			snapshotRequestCount := 0
+			cacher.watchCache.RWMutex.Lock()
+			cacher.watchCache.snapshots = &fakeSnapshotter{
+				getLessOrEqual: func(rv uint64) (orderedLister, bool) {
+					snapshotAvailable := tc.snapshotsAvailable[snapshotRequestCount]
+					snapshotRequestCount++
+					if snapshotAvailable {
+						return fakeOrderedLister{}, true
+					} else {
+						return nil, false
+					}
+				},
+			}
+			cacher.watchCache.RWMutex.Unlock()
+			if err := cacher.ready.wait(context.Background()); err != nil {
+				t.Fatalf("unexpected error waiting for the cache to be ready")
+			}
+			delegator := NewCacheDelegator(cacher, backingStorage)
+
+			result := &example.PodList{}
+			err = delegator.GetList(context.TODO(), "pods/ns", storage.ListOptions{ResourceVersion: "20", ResourceVersionMatch: metav1.ResourceVersionMatchExact, Recursive: true}, result)
+			if err != nil {
+				t.Fatalf("Unexpected error: %v", err)
+			}
+			if result.ResourceVersion != "20" {
+				t.Fatalf("Unexpected List response RV, got: %q, want: %d", result.ResourceVersion, 20)
+			}
+			if expectStoreRequests != tc.expectStoreRequests {
+				t.Fatalf("Unexpected number of requests to storage, got: %d, want: %d", expectStoreRequests, tc.expectStoreRequests)
+			}
+			if snapshotRequestCount != tc.expectSnapshotRequests {
+				t.Fatalf("Unexpected number of requests to snapshots, got: %d, want: %d", snapshotRequestCount, tc.expectSnapshotRequests)
+			}
+
+		})
+	}
+}
+
 func TestGetListNonRecursiveCacheBypass(t *testing.T) {
 	featuregatetesting.SetFeatureGateDuringTest(t, utilfeature.DefaultFeatureGate, features.ConsistentListFromCache, false)
 	backingStorage := &dummyStorage{}
diff --git a/staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go b/staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go
index 0b968eef5cf03..501190dcc104c 100644
--- a/staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go
+++ b/staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go
@@ -218,6 +218,9 @@ func (c *CacheDelegator) GetList(ctx context.Context, key string, opts storage.L
 	success := "true"
 	fallback := "false"
 	if err != nil {
+		if errors.IsResourceExpired(err) {
+			return c.storage.GetList(ctx, key, opts, listObj)
+		}
 		if result.ConsistentRead {
 			if storage.IsTooLargeResourceVersion(err) {
 				fallback = "true"
diff --git a/staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go b/staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go
index 908081efb958d..967a60c9b1d68 100644
--- a/staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go
+++ b/staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go
@@ -25,6 +25,7 @@ import (
 	"time"
 
 	"k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/fields"
 	"k8s.io/apimachinery/pkg/labels"
 	"k8s.io/apimachinery/pkg/runtime"
@@ -508,11 +509,61 @@ func (w *watchCache) WaitUntilFreshAndList(ctx context.Context, resourceVersion
 	if err != nil {
 		return listResp{}, "", err
 	}
+	return w.list(ctx, resourceVersion, key, opts)
+}
+
+// NOTICE: Structure follows the shouldDelegateList function in
+// staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go
+func (w *watchCache) list(ctx context.Context, resourceVersion uint64, key string, opts storage.ListOptions) (resp listResp, index string, err error) {
+	switch opts.ResourceVersionMatch {
+	case metav1.ResourceVersionMatchExact:
+		return w.listExactRV(key, "", resourceVersion)
+	case metav1.ResourceVersionMatchNotOlderThan:
+	case "":
+		// Continue
+		if len(opts.Predicate.Continue) > 0 {
+			continueKey, continueRV, err := storage.DecodeContinue(opts.Predicate.Continue, key)
+			if err != nil {
+				return listResp{}, "", errors.NewBadRequest(fmt.Sprintf("invalid continue token: %v", err))
+			}
+			if continueRV > 0 {
+				return w.listExactRV(key, continueKey, uint64(continueRV))
+			} else {
+				// Continue with negative RV is a consistent read - already handled via waitUntilFreshAndBlock.
+				// Don't pass matchValues as they don't support continueKey
+				return w.listLatestRV(key, continueKey, nil)
+			}
+		}
+		// Legacy exact match
+		if opts.Predicate.Limit > 0 && len(opts.ResourceVersion) > 0 && opts.ResourceVersion != "0" {
+			return w.listExactRV(key, "", resourceVersion)
+		}
+		// Consistent Read - already handled via waitUntilFreshAndBlock
+	}
+	return w.listLatestRV(key, "", opts.Predicate.MatcherIndex(ctx))
+}
+
+func (w *watchCache) listExactRV(key, continueKey string, resourceVersion uint64) (resp listResp, index string, err error) {
+	if w.snapshots == nil {
+		return listResp{}, "", errors.NewResourceExpired(fmt.Sprintf("too old resource version: %d", resourceVersion))
+	}
+	store, ok := w.snapshots.GetLessOrEqual(resourceVersion)
+	if !ok {
+		return listResp{}, "", errors.NewResourceExpired(fmt.Sprintf("too old resource version: %d", resourceVersion))
+	}
+	items := store.ListPrefix(key, continueKey)
+	return listResp{
+		Items:           items,
+		ResourceVersion: resourceVersion,
+	}, "", nil
+}
+
+func (w *watchCache) listLatestRV(key, continueKey string, matchValues []storage.MatchValue) (resp listResp, index string, err error) {
 	// This isn't the place where we do "final filtering" - only some "prefiltering" is happening here. So the only
 	// requirement here is to NOT miss anything that should be returned. We can return as many non-matching items as we
 	// want - they will be filtered out later. The fact that we return less things is only further performance improvement.
 	// TODO: if multiple indexes match, return the one with the fewest items, so as to do as much filtering as possible.
-	for _, matchValue := range opts.Predicate.MatcherIndex(ctx) {
+	for _, matchValue := range matchValues {
 		if result, err := w.store.ByIndex(matchValue.IndexName, matchValue.Value); err == nil {
 			result, err = filterPrefixAndOrder(key, result)
 			return listResp{
@@ -522,7 +573,7 @@ func (w *watchCache) WaitUntilFreshAndList(ctx context.Context, resourceVersion
 		}
 	}
 	if store, ok := w.store.(orderedLister); ok {
-		result := store.ListPrefix(key, "")
+		result := store.ListPrefix(key, continueKey)
 		return listResp{
 			Items:           result,
 			ResourceVersion: w.resourceVersion,
diff --git a/staging/src/k8s.io/apiserver/pkg/storage/etcd3/store_test.go b/staging/src/k8s.io/apiserver/pkg/storage/etcd3/store_test.go
index ee74825241328..4a4ef819c9d4a 100644
--- a/staging/src/k8s.io/apiserver/pkg/storage/etcd3/store_test.go
+++ b/staging/src/k8s.io/apiserver/pkg/storage/etcd3/store_test.go
@@ -254,7 +254,7 @@ func TestList(t *testing.T) {
 
 func TestConsistentList(t *testing.T) {
 	ctx, store, client := testSetup(t)
-	storagetesting.RunTestConsistentList(ctx, t, store, increaseRV(client.Client), false, true)
+	storagetesting.RunTestConsistentList(ctx, t, store, increaseRV(client.Client), false, true, false)
 }
 
 func checkStorageCallsInvariants(transformer *storagetesting.PrefixTransformer, recorder *clientRecorder) storagetesting.CallsValidation {
diff --git a/staging/src/k8s.io/apiserver/pkg/storage/testing/store_tests.go b/staging/src/k8s.io/apiserver/pkg/storage/testing/store_tests.go
index 668d0b629fba4..284f6f8e9f501 100644
--- a/staging/src/k8s.io/apiserver/pkg/storage/testing/store_tests.go
+++ b/staging/src/k8s.io/apiserver/pkg/storage/testing/store_tests.go
@@ -1658,7 +1658,7 @@ func ExpectContinueMatches(t *testing.T, expect, got string) {
 	t.Errorf("expected continue token: %s, got: %s", expectDecoded, gotDecoded)
 }
 
-func RunTestConsistentList(ctx context.Context, t *testing.T, store storage.Interface, increaseRV IncreaseRVFunc, cacheEnabled, consistentReadsSupported bool) {
+func RunTestConsistentList(ctx context.Context, t *testing.T, store storage.Interface, increaseRV IncreaseRVFunc, cacheEnabled, consistentReadsSupported, listFromCacheSnapshot bool) {
 	outPod := &example.Pod{}
 	inPod := &example.Pod{ObjectMeta: metav1.ObjectMeta{Namespace: "default", Name: "foo"}}
 	err := store.Create(ctx, computePodKey(inPod), inPod, outPod, 0)
@@ -1701,8 +1701,10 @@ func RunTestConsistentList(ctx context.Context, t *testing.T, store storage.Inte
 			name:          "List with negative continue RV returns consistent RV",
 			continueToken: encodeContinueOrDie("/pods/a", -1),
 			validateResponseRV: func(t *testing.T, rv int) {
-				// TODO: Update cacheSyncRV after continue is served from cache.
 				assert.Equal(t, consistentRV, rv)
+				if listFromCacheSnapshot {
+					cacheSyncRV = rv
+				}
 			},
 		},
 		{
@@ -1728,6 +1730,7 @@ func RunTestConsistentList(ctx context.Context, t *testing.T, store storage.Inte
 		t.Run(tc.name, func(t *testing.T) {
 			out := &example.PodList{}
 			opts := storage.ListOptions{
+				Recursive:       true,
 				ResourceVersion: tc.requestRV,
 				Predicate: storage.SelectionPredicate{
 					Label:    labels.Everything(),
diff --git a/test/integration/apiserver/apiserver_test.go b/test/integration/apiserver/apiserver_test.go
index 1c50153b38022..b8d7e66607830 100644
--- a/test/integration/apiserver/apiserver_test.go
+++ b/test/integration/apiserver/apiserver_test.go
@@ -62,6 +62,7 @@ import (
 	"k8s.io/apimachinery/pkg/util/wait"
 	"k8s.io/apimachinery/pkg/watch"
 	"k8s.io/apiserver/pkg/endpoints/handlers"
+	"k8s.io/apiserver/pkg/features"
 	"k8s.io/apiserver/pkg/storage/storagebackend"
 	"k8s.io/apiserver/pkg/util/compatibility"
 	utilfeature "k8s.io/apiserver/pkg/util/feature"
@@ -386,12 +387,17 @@ var (
 // TestListOptions ensures that list works as expected for valid and invalid combinations of limit, continue,
 // resourceVersion and resourceVersionMatch.
 func TestListOptions(t *testing.T) {
-
-	for _, watchCacheEnabled := range []bool{true, false} {
-		t.Run(fmt.Sprintf("watchCacheEnabled=%t", watchCacheEnabled), func(t *testing.T) {
-			testListOptions(t, watchCacheEnabled)
-		})

Leaderboard for this case

hard · 1 of 9 bots fully caught it (11% solve rate) · 44% avg coverage.

#BotResultCaughtCoverageFindingsFalse alarms
1 Mjolnir (Opus 5.5) solved 3/3 100.0% 4 0
2 Drape (Deep Review) partial 2/3 80.0% 3 0
3 Drape (Review) partial 2/3 80.0% 3 0
4 qodo partial 2/3 80.0% 3 0
5 cursor partial 2/3 60.0% 3 0
6 Mjolnir (Sonnet 5.5) partial 1/3 40.0% 2 0
7 Claude Code (Opus 5.5) miss 0/3 0.0% 0 0
8 Cubic miss 0/3 0.0% 2 0
9 greptile miss 0/3 0.0% 0 0

Bot results

Claude Code (Opus 5.5) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 medium ✗ —
d2 high ✗ —
d3 high ✗ —
Cubic · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 medium ✗ —
d2 high ✗ —
d3 high ✗ —
valid · impact medium
With ListFromCacheSnapshot enabled, `snapshots` is non-nil whatever the store type. A continue token with a negative RV takes the ShouldDelegateConsistentRead path, which never checks that the store is an orderedLister. If BtreeWatchCache is disabled, listLatestRV falls through to `w.store.List()` and drops continueKey, so the next page starts again from the beginning and repeats items. This PR added the path that serves these continues from the cache. [grounded: tree staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go] [consensus 4/5: valid=4, invalid=1]
minor · impact low
This is about how robust the test is, and it depends on a future change to the number of lookups. With the current code the slices are the right size, and a panic would still fail the test. [grounded: diff-section staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_whitebox_test.go] [consensus 5/5: minor=5]
raw comments (3)
staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:534
<!-- cubic:v=18f55872-7a1b-4ed2-a4ac-aa94875b0a54 --> <!-- cubic:review-run=85c13b69-1dc4-56cc-adc4-92c0374343b0 --> <!-- metadata:{"confidence":9} --> P2: When a negative-RV continuation is served without an `orderedLister`, `listLatestRV` ignores `continueKey` and returns all matching objects from the beginning, causing paginated responses to repeat earlier items. Apply the continuation boundary in the unordered-store fallback or delegate this request instead. <details> <summary>Prompt for AI agents</summary> ```text Check if this issue is valid — if so, understand the root cause and fix it. At staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go, line 534: <comment>When a negative-RV continuation is served without an `orderedLister`, `listLatestRV` ignores `continueKey` and returns all matching objects from the beginning, causing paginated responses to repeat earlier items. Apply the continuation boundary in the unordered-store fallback or delegate this request instead.</comment> <file context> @@ -508,11 +509,61 @@ func (w *watchCache) WaitUntilFreshAndList(ctx context.Context, resourceVersion + } else { + // Continue with negative RV is a consistent read - already handled via waitUntilFreshAndBlock. + // Don't pass matchValues as they don't support continueKey + return w.listLatestRV(key, continueKey, nil) + } + } </file context> ``` </details>
staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_whitebox_test.go:686
<!-- cubic:v=d9f2f07c-e320-4be3-a34a-608e8a234e86 --> <!-- cubic:review-run=85c13b69-1dc4-56cc-adc4-92c0374343b0 --> <!-- metadata:{"confidence":8} --> P3: The fakeSnapshotter's getLessOrEqual closure indexes `tc.snapshotsAvailable` by the unbounded `snapshotRequestCount`. The slices are sized to exactly match the current call count, so any change in how many snapshot lookups the delegator performs causes an index-out-of-range panic rather than a clean assertion failure. Bound the index (return false when the counter exceeds the slice) so the test fails clearly instead of panicking. <details> <summary>Prompt for AI agents</summary> ```text Check if this issue is valid — if so, understand the root cause and fix it. At staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_whitebox_test.go, line 686: <comment>The fakeSnapshotter's getLessOrEqual closure indexes `tc.snapshotsAvailable` by the unbounded `snapshotRequestCount`. The slices are sized to exactly match the current call count, so any change in how many snapshot lookups the delegator performs causes an index-out-of-range panic rather than a clean assertion failure. Bound the index (return false when the counter exceeds the slice) so the test fails clearly instead of panicking.</comment> <file context> @@ -568,6 +632,91 @@ apiserver_watch_cache_consistent_read_total{fallback="true", resource="pods", su + cacher.watchCache.RWMutex.Lock() + cacher.watchCache.snapshots = &fakeSnapshotter{ + getLessOrEqual: func(rv uint64) (orderedLister, bool) { + snapshotAvailable := tc.snapshotsAvailable[snapshotRequestCount] + snapshotRequestCount++ + if snapshotAvailable { </file context> ``` </details> ```suggestion if snapshotRequestCount >= len(tc.snapshotsAvailable) { return nil, false } snapshotAvailable := tc.snapshotsAvailable[snapshotRequestCount] ```
PR-level
<!-- cubic:review-summary:start --> **2 issues found** across 8 files <!-- cubic:review-summary:end --> <details> <summary>Prompt for AI agents (unresolved issues)</summary> ```text Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately. <file name="staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go"> <violation number="1" location="staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:534"> P2: When a negative-RV continuation is served without an `orderedLister`, `listLatestRV` ignores `continueKey` and returns all matching objects from the beginning, causing paginated responses to repeat earlier items. Apply the continuation boundary in the unordered-store fallback or delegate this request instead.</violation> </file> <file name="staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_whitebox_test.go"> <violation number="1" location="staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_whitebox_test.go:686"> P3: The fakeSnapshotter's getLessOrEqual closure indexes `tc.snapshotsAvailable` by the unbounded `snapshotRequestCount`. The slices are sized to exactly match the current call count, so any change in how many snapshot lookups the delegator performs causes an index-out-of-range panic rather than a clean assertion failure. Bound the index (return false when the counter exceeds the slice) so the test fails clearly instead of panicking.</violation> </file> ``` </details> <sub>Reply with feedback, questions, or to request a fix.<br /><br />[Re-trigger cubic](https://www.cubic.dev/action/re-review/pr/drape-io/crbench-cubic-2026-09/49/ai_pr_review_github_delivery_f903332fddcdc93cfd87c6d4915be4dabd97f685c7a87de8cb1f3326b54f9a84?returnTo=https%3A%2F%2Fgithub.com%2Fdrape-io%2Fcrbench-cubic-2026-09%2Fpull%2F49)</sub> <!-- cubic:review-post:ai_pr_review_github_delivery_f903332fddcdc93cfd87c6d4915be4dabd97f685c7a87de8cb1f3326b54f9a84:3686a8e10d5b56d27b809f780144bf2a7e862c03:85c13b69-1dc4-56cc-adc4-92c0374343b0 -->
cursor · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 medium ✓ off by one
d2 high ✓ exact
d3 high ✗ —
valid · matches d1 · impact medium
valid · impact medium
Before this change, continue requests always went to etcd. Now a recursive continue with RV="0" is served from the cache, and `computeListLimit` returns 0 when `opts.ResourceVersion == "0"`. So the limit is dropped and the cache returns everything after `continueKey` instead of a single page. The test enumeration shows this combination passes validation. [grounded: tree staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_whitebox_test.go] [consensus 5/5: valid=5]
valid · impact medium
For an exact RV the cache has not seen yet, `shouldDelegateExactRV` keeps the request on the cache with `ConsistentRead` false. If `waitUntilFreshAndBlock` times out, it returns TooLargeResourceVersion. The delegator only falls back to etcd for ResourceExpired errors or when `result.ConsistentRead` is true, so this error goes straight to the client. Before this change, these requests were always delegated to etcd. [grounded: tree staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go] [consensus 5/5: valid=5]
raw comments (4)
staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:539
### Consistent paginated lists hit exact path **High Severity** <!-- DESCRIPTION START --> `list()` treats a consistent read with a `limit` as a legacy exact-RV list after `GetList` rewrites `ResourceVersion` to the current etcd RV. `shouldDelegateList` ran on the original empty RV, so the cache still accepts the request. With snapshots unset, `listExactRV` returns `ResourceExpired` and the request falls back to etcd, so paginated consistent lists no longer serve from cache even when `ListFromCacheSnapshot` is off. <!-- DESCRIPTION END --> <!-- BUGBOT_BUG_ID: 75be619d-d30e-4443-be7a-8e2312a758f6 --> <!-- LOCATIONS START staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go#L536-L539 staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go#L208-L222 LOCATIONS END --> <details> <summary>Additional Locations (1)</summary> - [`staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go#L208-L222`](https://github.com/drape-io/crbench-cursor-2026-09/blob/4d3743be80a25fffad4b09283d292581dd6abab3/staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go#L208-L222) </details> <div><a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9DVVJTT1IiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjdhMWMwZDA4LTE2NTQtNDZlOS05NWJiLTIwNzJhZjczNGNmZSIsImVuY3J5cHRpb25LZXkiOiJjVXRPLUg0bE5pRHd6d3FZYkFnVXl1VzJ3WElxa2hwSW16aXRwemlKLUgwIiwiYnJhbmNoIjoidDItazhzLTEzMDQyMyIsInJlcG9Pd25lciI6ImRyYXBlLWlvIiwicmVwb05hbWUiOiJjcmJlbmNoLWN1cnNvci0yMDI2LTA5IiwicHJvdmlkZXIiOiJnaXRodWIifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/agents?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9XRUIiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjdhMWMwZDA4LTE2NTQtNDZlOS05NWJiLTIwNzJhZjczNGNmZSIsImVuY3J5cHRpb25LZXkiOiJjVXRPLUg0bE5pRHd6d3FZYkFnVXl1VzJ3WElxa2hwSW16aXRwemlKLUgwIiwiYnJhbmNoIjoidDItazhzLTEzMDQyMyIsInJlcG9Pd25lciI6ImRyYXBlLWlvIiwicmVwb05hbWUiOiJjcmJlbmNoLWN1cnNvci0yMDI2LTA5IiwicHJOdW1iZXIiOjUwLCJjb21taXRTaGEiOiI0ZDM3NDNiZTgwYTI1ZmZmYWQ0YjA5MjgzZDI5MjU4MWRkNmFiYWIzIiwicHJvdmlkZXIiOiJnaXRodWIifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-web-light.png"><img alt="Fix in Web" width="99" height="28" src="https://cursor.com/assets/images/fix-in-web-dark.png"></picture></a></div> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 4d3743be80a25fffad4b09283d292581dd6abab3. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:1331
### Continue lists ignore limit at RV 0 **Medium Severity** <!-- DESCRIPTION START --> Continue lists with `resourceVersion` `"0"` are now served from cache, but `computeListLimit` still drops `limit` whenever RV is `"0"`. That combination is valid API input, so a paged continue can return the entire remaining set instead of one page. <!-- DESCRIPTION END --> <!-- BUGBOT_BUG_ID: 15a686df-53dc-43ae-87d4-5f5474688d6a --> <!-- LOCATIONS START staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go#L693-L698 staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go#L522-L530 LOCATIONS END --> <details> <summary>Additional Locations (1)</summary> - [`staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go#L522-L530`](https://github.com/drape-io/crbench-cursor-2026-09/blob/4d3743be80a25fffad4b09283d292581dd6abab3/staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go#L522-L530) </details> <div><a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9DVVJTT1IiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjdkOTYwMDI5LWI0NDQtNDdjNy04ZjZkLTdlMTRkNjkzZjczMCIsImVuY3J5cHRpb25LZXkiOiJDbmZueU1XOWJfQmxuV3hJTndOX044REUzQ04yQXNRcVNPMzJGTTlzWUFrIiwiYnJhbmNoIjoidDItazhzLTEzMDQyMyIsInJlcG9Pd25lciI6ImRyYXBlLWlvIiwicmVwb05hbWUiOiJjcmJlbmNoLWN1cnNvci0yMDI2LTA5IiwicHJvdmlkZXIiOiJnaXRodWIifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/agents?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9XRUIiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjdkOTYwMDI5LWI0NDQtNDdjNy04ZjZkLTdlMTRkNjkzZjczMCIsImVuY3J5cHRpb25LZXkiOiJDbmZueU1XOWJfQmxuV3hJTndOX044REUzQ04yQXNRcVNPMzJGTTlzWUFrIiwiYnJhbmNoIjoidDItazhzLTEzMDQyMyIsInJlcG9Pd25lciI6ImRyYXBlLWlvIiwicmVwb05hbWUiOiJjcmJlbmNoLWN1cnNvci0yMDI2LTA5IiwicHJOdW1iZXIiOjUwLCJjb21taXRTaGEiOiI0ZDM3NDNiZTgwYTI1ZmZmYWQ0YjA5MjgzZDI5MjU4MWRkNmFiYWIzIiwicHJvdmlkZXIiOiJnaXRodWIifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-web-light.png"><img alt="Fix in Web" width="99" height="28" src="https://cursor.com/assets/images/fix-in-web-dark.png"></picture></a></div> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 4d3743be80a25fffad4b09283d292581dd6abab3. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:1364
### Stale exact RV wait does not fall back **Medium Severity** <!-- DESCRIPTION START --> An exact list at an RV the cache has not observed yet is kept on cache when consistent reads are supported, but `ConsistentRead` is left false. If `waitUntilFreshAndBlock` times out, `TooLargeResourceVersion` is returned to the client. Unlike consistent reads, this path never falls back to etcd, which can still serve that historical revision. <!-- DESCRIPTION END --> <!-- BUGBOT_BUG_ID: 0595686c-d5db-4c2b-aa21-273c1c38bd81 --> <!-- LOCATIONS START staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go#L1359-L1364 staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go#L219-L236 LOCATIONS END --> <details> <summary>Additional Locations (1)</summary> - [`staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go#L219-L236`](https://github.com/drape-io/crbench-cursor-2026-09/blob/4d3743be80a25fffad4b09283d292581dd6abab3/staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go#L219-L236) </details> <div><a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9DVVJTT1IiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjRhMTY5MjMxLWMzZDMtNDY3NS04YmRiLTFhODA4MTRjZjNlZiIsImVuY3J5cHRpb25LZXkiOiJZMzZHdDVZUGN6ejBHU19mMHlIcDkyN0lnMUt0a0tHWnlpeF91c3pIbnkwIiwiYnJhbmNoIjoidDItazhzLTEzMDQyMyIsInJlcG9Pd25lciI6ImRyYXBlLWlvIiwicmVwb05hbWUiOiJjcmJlbmNoLWN1cnNvci0yMDI2LTA5IiwicHJvdmlkZXIiOiJnaXRodWIifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/agents?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9XRUIiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjRhMTY5MjMxLWMzZDMtNDY3NS04YmRiLTFhODA4MTRjZjNlZiIsImVuY3J5cHRpb25LZXkiOiJZMzZHdDVZUGN6ejBHU19mMHlIcDkyN0lnMUt0a0tHWnlpeF91c3pIbnkwIiwiYnJhbmNoIjoidDItazhzLTEzMDQyMyIsInJlcG9Pd25lciI6ImRyYXBlLWlvIiwicmVwb05hbWUiOiJjcmJlbmNoLWN1cnNvci0yMDI2LTA5IiwicHJOdW1iZXIiOjUwLCJjb21taXRTaGEiOiI0ZDM3NDNiZTgwYTI1ZmZmYWQ0YjA5MjgzZDI5MjU4MWRkNmFiYWIzIiwicHJvdmlkZXIiOiJnaXRodWIifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-web-light.png"><img alt="Fix in Web" width="99" height="28" src="https://cursor.com/assets/images/fix-in-web-dark.png"></picture></a></div> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 4d3743be80a25fffad4b09283d292581dd6abab3. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
PR-level
<!-- BUGBOT_REVIEW --> Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues. <!-- BUGBOT_FIX_ALL --> <a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9BTExfSU5fQ1VSU09SIiwiZGF0YSI6eyJyZWRpc0tleSI6ImJ1Z2JvdC1tdWx0aTpjYmVmNTgwMC0xZjcxLTRmZmItODQwMi02NjA4MTExYWEzYWUiLCJlbmNyeXB0aW9uS2V5IjoiRXpDVktuSVlzSEI2X3dDM0lDSHZWbGFOQUFPU1dzTk1NY28zQWlTU2ZaUSIsImJyYW5jaCI6InQyLWs4cy0xMzA0MjMiLCJyZXBvT3duZXIiOiJkcmFwZS1pbyIsInJlcG9OYW1lIjoiY3JiZW5jaC1jdXJzb3ItMjAyNi0wOSIsInByb3ZpZGVyIjoiZ2l0aHViIn19" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix All in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a> <!-- /BUGBOT_FIX_ALL --> <!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_BEGIN --> <sup>❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the [Cursor dashboard](https://www.cursor.com/dashboard/bugbot).</sup> <!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_END --> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 4d3743be80a25fffad4b09283d292581dd6abab3. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
Drape (Deep Review) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 medium ✗ —
d2 high ✓ exact
d3 high ✓ off by one
valid · impact low
A consistent read can return ResourceExpired. The delegator rewrites opts.ResourceVersion to the etcd RV. When a limit is set, watchCache.list then takes the legacy exact-match branch into listExactRV. That returns ResourceExpired if snapshots are nil or the RV is missing. The new early return then goes to etcd and skips the ConsistentReadTotal accounting below it. This path did not exist before the change. [grounded: tree staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go] [consensus 3/3: valid=3]
valid · matches d2 · impact high
valid · matches d3 · impact high
raw comments (3)
staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go:222
**Consistent-read metrics skipped on ResourceExpired fallback** The new ResourceExpired early return comes before the `ConsistentReadTotal` accounting. Consistent reads that fall back to etcd this way are never counted, as success or as fallback. When `result.ConsistentRead` is true, record the metric with fallback="true" before returning the etcd result. ``` staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go:221 if errors.IsResourceExpired(err) { staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go:222 return c.storage.GetList(ctx, key, opts, listObj) staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go:223 } staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go:224 if result.ConsistentRead { staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go:225 if storage.IsTooLargeResourceVersion(err) { staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go:226 fallback = "true" staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go:227 // Reset resourceVersion during fallback from consistent read. staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go:228 opts.ResourceVersion = "" staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go:229 err = c.storage.GetList(ctx, key, opts, listObj) staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go:230 } ```
staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:538
**Paginated consistent lists always fall back to etcd** The delegator writes the etcd RV into `opts.ResourceVersion`, so `list` treats a limited consistent list as a legacy exact match. With snapshots off (the default), `listExactRV` returns ResourceExpired and etcd serves it. For consistent reads without a continue token, also set `opts.ResourceVersionMatch` to NotOlderThan in the delegator. ``` staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:537 // Legacy exact match staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:538 if opts.Predicate.Limit > 0 && len(opts.ResourceVersion) > 0 && opts.ResourceVersion != "0" { staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:539 return w.listExactRV(key, "", resourceVersion) staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:540 } staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go:215 opts.ResourceVersion = strconv.FormatInt(int64(listRV), 10) staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:547 if w.snapshots == nil { staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:548 return listResp{}, "", errors.NewResourceExpired(fmt.Sprintf("too old resource version: %d", resourceVersion)) staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:549 } staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:190 if utilfeature.DefaultFeatureGate.Enabled(features.ListFromCacheSnapshot) { staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:191 wc.snapshots = newStoreSnapshotter() ```
staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:530
**Continue with future RV served from stale snapshot** If a continue token's RV is ahead of this cache (e.g. from another apiserver), nothing waits since listRV is 0, so `listExactRV` returns an older snapshot labeled with the token's RV. Wait for the cache to reach the continue RV before calling listExactRV, or delegate such continues to etcd. ``` staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:1360 // Exact requests on future revision require support for consistent read, but are not a consistent read by themselves. staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:1361 if c.watchCache.notFresh(rv) { staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:1362 return delegator.Result{ staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:1363 ShouldDelegate: !delegator.ConsistentReadSupported(), staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:472 for w.resourceVersion < resourceVersion { staging/src/k8s.io/apiserver/pkg/storage/cacher/store_btree.go:469 s.snapshots.DescendLessOrEqual(rvSnapshot{resourceVersion: rv}, func(rvs rvSnapshot) bool { staging/src/k8s.io/apiserver/pkg/storage/cacher/store_btree.go:470 result = &rvs staging/src/k8s.io/apiserver/pkg/storage/cacher/store_btree.go:471 return false staging/src/k8s.io/apiserver/pkg/storage/cacher/store_btree.go:472 }) staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:555 return listResp{ ```
Drape (Review) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 medium ✗ —
d2 high ✓ exact
d3 high ✓ off by one
valid · matches d3 · impact high
valid · matches d2 · impact high
valid · impact low
The estimator (checked in repo) still decides cache vs. storage with CacheWithoutSnapshots, so it treats exact-RV and continue lists as delegated and costs them at 2*limit. With this change, the cacher serves those requests from snapshots and scans the whole prefix (numStored). That leaves the sibling decision path out of sync even though the code says to keep the two in sync, and these requests get too few seats. [grounded: tree staging/src/k8s.io/apiserver/pkg/util/flowcontrol/request/list_work_estimator.go] [consensus 3/3: valid=3]
raw comments (3)
staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:530
**Continue on unobserved RV served from stale snapshot** For a continue token with an RV newer than the cache, `WaitUntilFreshAndList` waits on `opts.ResourceVersion` (empty or "0"), not the token's RV. `listExactRV` then returns an older snapshot labelled with the newer RV. Wait for the continue RV in `list`, or delegate when it is not yet observed. ``` staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:524 if len(opts.Predicate.Continue) > 0 { staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:525 continueKey, continueRV, err := storage.DecodeContinue(opts.Predicate.Continue, key) staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:526 if err != nil { staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:527 return listResp{}, "", errors.NewBadRequest(fmt.Sprintf("invalid continue token: %v", err)) staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:546 func (w *watchCache) listExactRV(key, continueKey string, resourceVersion uint64) (resp listResp, index string, err error) { staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:547 if w.snapshots == nil { staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:548 return listResp{}, "", errors.NewResourceExpired(fmt.Sprintf("too old resource version: %d", resourceVersion)) staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:549 } staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:550 store, ok := w.snapshots.GetLessOrEqual(resourceVersion) staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:551 if !ok { ```
staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:538
**Paginated consistent list always falls back to etcd** For a consistent read with a limit, the delegator sets opts.ResourceVersion to the etcd RV, so `list()` takes the legacy exact branch. With snapshots disabled (the default), `listExactRV` returns ResourceExpired and the request goes to etcd. Skip the legacy-exact branch in `list()` for requests the delegator turned into consistent reads. ``` staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:537 // Legacy exact match staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:538 if opts.Predicate.Limit > 0 && len(opts.ResourceVersion) > 0 && opts.ResourceVersion != "0" { staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:539 return w.listExactRV(key, "", resourceVersion) staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:540 } staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go:209 if result.ConsistentRead { staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go:210 listRV, err = c.storage.GetCurrentResourceVersion(ctx) staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go:211 if err != nil { staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go:212 return err staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go:213 } staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go:214 // Setting resource version for consistent read in cache based on current ResourceVersion in etcd. ```
staging/src/k8s.io/apiserver/pkg/util/flowcontrol/request/list_work_estimator.go:86
**Seat estimate ignores lists now served from snapshots** The estimator still uses `delegator.CacheWithoutSnapshots{}`, so exact-RV and continue lists are costed as etcd reads (about 2*limit). With snapshots on, the cacher serves them and scans the whole prefix, so they take too few seats. Pick the helper from the ListFromCacheSnapshot gate, or cost these requests as cache lists. ``` staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:1330 func (c *Cacher) ShouldDelegateExactRV(resourceVersion string, recursive bool) (delegator.Result, error) { staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:1331 // Not Recursive is not supported unitl exact RV is implemented for WaitUntilFreshAndGet. staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:1332 if !recursive || c.watchCache.snapshots == nil { staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:1333 return delegator.Result{ShouldDelegate: true}, nil staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator/interface.go:41 type CacheWithoutSnapshots struct{} staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator/interface.go:43 var _ Helper = CacheWithoutSnapshots{} staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator/interface.go:45 func (c CacheWithoutSnapshots) ShouldDelegateContinue(continueToken string, recursive bool) (Result, error) { staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator/interface.go:46 return Result{ staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator/interface.go:47 ShouldDelegate: true, staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator/interface.go:48 // Continue with negative RV is considered a consistent read, however token cannot be parsed without keyPrefix unavailable in staging/src/k8s.io/apiserver/pkg/util/flow_control/request/list_work_estimator.go. ```
greptile · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 medium ✗ —
d2 high ✗ —
d3 high ✗ —
raw comments (1)
PR-level
<h3>Greptile Summary</h3> This PR enables recursive exact-resource-version and continued list requests to be served from watch-cache snapshots when possible, while retaining backing-storage fallback when a suitable snapshot is unavailable. - Adds snapshot-aware delegation for exact, continued, and consistent list requests. - Selects historical snapshots for exact revisions and continuation tokens. - Falls back to backing storage when snapshot selection races with expiration. - Expands unit, storage, and integration coverage across the relevant feature gates. <h3>Confidence Score: 5/5</h3> The PR appears safe to merge; snapshot selection, freshness waiting, continuation handling, and storage fallback form a coherent list-serving path with expanded coverage. No actionable correctness or quality failures remain after checking snapshot revision semantics, continuation-token handling, destination mutation timing, freshness behavior, and fallback control flow. <details><summary><h3>Important Files Changed</h3></summary> | Filename | Overview | |----------|----------| | staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go | Adds snapshot-aware delegation decisions for exact revisions, continuation tokens, and consistent reads. | | staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go | Retries backing storage when a selected cache snapshot expires before the list is served. | | staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go | Splits latest and exact listing paths and serves historical exact lists from retained snapshots. | | staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_whitebox_test.go | Extends delegation and fallback coverage for available, unavailable, and disabled snapshots. | | staging/src/k8s.io/apiserver/pkg/storage/testing/store_tests.go | Updates consistent-list test expectations for snapshot-backed cache behavior. | | test/integration/apiserver/apiserver_test.go | Runs list-option integration coverage with snapshot listing both enabled and disabled. | </details> <details open><summary><h3>Flowchart</h3></summary> ```mermaid %%{init: {'theme': 'neutral'}}%% flowchart TD A[List request] --> B{Recursive and snapshots enabled?} B -- No --> E[Delegate to backing storage] B -- Yes --> C{Exact RV or positive continue RV?} C -- No --> D{Consistent read?} C -- Yes --> F{Cache fresh and snapshot available?} F -- No --> E F -- Yes --> G[Wait until requested RV is observed] G --> H[Read greatest snapshot at or below RV] H --> I{Snapshot still available?} I -- Yes --> J[Return filtered and paginated cached list] I -- No: ResourceExpired --> E D -- Yes --> K[Wait for watch progress and read latest cache] D -- No --> L[Read latest cache state] ``` </details> <!-- greptile_other_comments_section --> <sub>Reviews (1): Last reviewed commit: ["Update staging/src/k8s.io/apiserver/pkg/..."](https://github.com/drape-io/crbench-greptile-2026-09/commit/7d6c16b586543cfe99663c4dbcd2755a69fdf1ee) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=64284552)</sub>
Mjolnir (Opus 5.5) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 medium ✓ off by one
d2 high ✓ exact
d3 high ✓ exact
valid · matches d1 · impact medium
valid · impact high
The claim holds. When the continue RV is newer than the cache and consistent reads are supported, shouldDelegateExactRV returns ShouldDelegate=false and ConsistentRead=false. That means opts.ResourceVersion stays empty or 0, so Cacher.GetList parses listRV=0 and waitUntilFreshAndBlock(0) returns at once. listExactRV then calls GetLessOrEqual(continueRV), which returns an older snapshot, and still labels the response with ResourceVersion=continueRV. Pages can therefore come from different revisions. Exact-RV requests are not affected because their listRV is the requested RV, so they do wait. [grounded: diff] [consensus 3/3: valid=3]
valid · matches d2 · impact high
valid · matches d3 · impact high
raw comments (4)
staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:537
[P1] Paginated consistent reads are now treated as legacy exact reads and fall back to etcd · For a consistent read (`ResourceVersion: ""`, `Limit › 0`), `CacheDelegator.GetList` overwrites `opts.ResourceVersion` with the current etcd RV and leaves `ResourceVersionMatch` empty. Because of that, the new `list()` treats the request as a "Legacy exact match" and calls `listExactRV`. With `ListFromCacheSnapshot` disabled (the default, so `w.snapshots == nil`), `listExactRV` always returns ResourceExpired, and the new branch in `delegator.go` quietly sends the request to etcd. So when `ConsistentListFromCache` is on, every paginated consistent list stops being served from the watch cache, which is a large load regression on etcd. This fallback also returns before `ConsistentReadTotal` is recorded. With snapshots enabled, these requests skip the index-based prefiltering (`MatcherIndex`). The consistent-read path needs to be told apart from a client-supplied exact RV, for example by setting `ResourceVersionMatchNotOlderThan` when `opts.ResourceVersion` is rewritten and there is no continue token.
staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:1360
[P1] Continue tokens with an RV newer than the cache are served from an older snapshot without waiting · When the cache has not caught up to the continue RV (`notFresh(rv)`) and consistent reads are supported, `shouldDelegateExactRV` keeps the request in the cache and expects the cache to wait. For continue requests, though, `opts.ResourceVersion` is `""` or `"0"`, so `Cacher.GetList` parses `listRV = 0` and `waitUntilFreshAndBlock` returns at once. `listExactRV` then calls `snapshots.GetLessOrEqual(continueRV)`, which returns the newest snapshot. That snapshot is older than `continueRV`, but the response is still labelled with `continueRV`. The next page can therefore come from a different revision than the earlier pages, which breaks the consistency guarantee of paginated lists. This happens in HA setups when a continue token issued by one apiserver (or by etcd) reaches an apiserver whose watch cache is behind, which is the `continueOnEtcdRV` case the new tests mark as non-delegated. The continue RV needs to be used as the wait target, or these requests should be delegated.
staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:537
[P1] Paginated consistent lists now always fall through to etcd when snapshots are disabled For a consistent read (`ResourceVersion: ""`), `CacheDelegator.GetList` rewrites `opts.ResourceVersion` to the current etcd RV before calling the cacher. Once that has happened, a request with `Limit › 0` and no continue token matches the "Legacy exact match" branch in `watchCache.list` and goes to `listExactRV`. With `ListFromCacheSnapshot` disabled (the default, alpha), `w.snapshots` is nil, so this returns `ResourceExpired`, and the new `IsResourceExpired` branch in delegator.go sends the request to etcd. Before this change, these requests were served from the watch cache. So under default feature gates, every paginated consistent LIST (for example, kubectl's default `limit=500`) now reaches etcd, and the `ConsistentReadTotal` metric is skipped on that path. The delegator needs to mark the rewritten consistent read as not-older-than (as it does for non-continue requests), or `list` needs some other way to tell it apart from a real legacy exact-RV request.
staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:1362
[P1] Continue tokens newer than the cache are served from a stale snapshot When `continueRV` is newer than the cache (`notFresh(continueRV)` is true) and consistent reads are supported, `shouldDelegateExactRV` returns `ShouldDelegate: false`. The intent is for the cache to wait until it catches up. That does not happen for continue requests, because `ValidateListOptions` rejects any RV on them other than ""/"0", so `Cacher.GetList` gets `listRV == 0` and `waitUntilFreshAndBlock(0)` returns immediately. `listExactRV` then calls `snapshots.GetLessOrEqual(continueRV)`, which returns the cache's newest (older) snapshot, and the response is labelled `ResourceVersion: continueRV`. In HA setups this can happen when page 1 came from etcd or from another apiserver whose cache is fresher. The next page can then silently miss or include objects relative to `continueRV`, which breaks pagination consistency. Exact continue reads need to wait until the cache reaches `continueRV`, or be delegated to etcd when the cache is not yet fresh.
Mjolnir (Sonnet 5.5) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 medium ✗ —
d2 high ✓ exact
d3 high ✗ —
valid · matches d2 · impact high
valid · impact low
Whenever ListFromCacheSnapshot is on, snapshots is non-nil, and that does not depend on BtreeWatchCache. In that case ShouldDelegateContinue sends a negative-RV continue token to ShouldDelegateConsistentRead, so the cache serves it when consistent reads are supported. newStoreIndexer returns a plain cache.NewIndexer when BtreeWatchCache is off, so listLatestRV falls back to w.store.List() with filterPrefixAndOrder and never uses continueKey. Before this change, continue requests were always delegated to etcd, so the change introduced this path. [grounded: tree staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go] [consensus 3/5: valid=3, uncertain=1, invalid=1]
raw comments (2)
staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:538
[P1] Consistent LIST with a limit is misrouted to the exact-RV snapshot path For a consistent read (no resourceVersion, no continue), `CacheDelegator.GetList` overwrites `opts.ResourceVersion` with the current etcd RV before calling `Cacher.GetList`. `watchCache.list` then sees `ResourceVersionMatch == ""`, `Limit › 0` and a non-empty, non-"0" `ResourceVersion`. It takes the "Legacy exact match" branch and calls `listExactRV`, so the "Consistent Read" branch comment is unreachable whenever a limit is set. A paginated first page, e.g. `kubectl get --chunk-size=500`, is the common case. Two outcomes follow. With `ListFromCacheSnapshot` off (the default), `w.snapshots == nil`, so `listExactRV` returns ResourceExpired. The delegator's new `IsResourceExpired` fallback then sends the request to etcd with the rewritten exact RV. Every consistent LIST with a limit therefore pays a `GetCurrentResourceVersion` call, the cache wait and an etcd list, instead of being served from the cache as before. `ConsistentReadTotal` is also never recorded on that path. With snapshots on, these requests are served from a snapshot instead of the latest-RV path. The consistent-read case needs to be told apart from a client-supplied legacy exact RV. For example, pass an explicit flag, or check the original `ResourceVersion` before the delegator rewrites it.
staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:546
[P3] Continue token with a negative RV ignores the continue key when the store isn't ordered The new `ShouldDelegateContinue` serves a continue token with a negative RV from the cache whenever `snapshots != nil`, i.e. `ListFromCacheSnapshot` is on. `snapshots` does not depend on `BtreeWatchCache`, so the store can be the non-ordered indexer. In that case `listLatestRV` skips the `orderedLister` branch and falls back to `w.store.List()` with `filterPrefixAndOrder`, which ignores `continueKey`. The continuation page then restarts from the beginning of the prefix and returns duplicate items. This only occurs with `ListFromCacheSnapshot` enabled, `BtreeWatchCache` disabled, and a hand-crafted or legacy token with a negative RV. Apply `continueKey` in the fallback, or delegate when the store is not an `orderedLister`.
qodo · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 medium ✗ —
d2 high ✓ off by one
d3 high ✓ exact
valid · matches d3 · impact high
valid · matches d2 · impact high
valid · impact low
The finding holds. A consistent read with a Limit and an empty RV is classified as ConsistentRead, and the delegator then rewrites opts.ResourceVersion to the current etcd RV. In watchCache.list that request now matches the legacy exact match branch (Limit>0, RV non-empty and not 0) and calls listExactRV. With snapshots disabled, listExactRV returns ResourceExpired. The new early return then sends the request to storage and skips the ConsistentReadTotal update. Before this change the same request was served from listLatestRV and counted in the metric, so the change introduced the gap. [grounded: diff] [consensus 3/3: valid=3]
raw comments (5)
staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go:1352
<img src="https://img.shields.io/badge/High-634FD1?style=flat-square" height="20px" alt="Action required"> 1\. Future pages can return stale objects <code>🐞 Bug</code> <code>≡ Correctness</code> <pre> <b><i>ShouldDelegateContinue</i></b> keeps a future positive continuation revision in the cache path, but <b><i>Cacher.GetList</i></b> waits using the empty request resource version rather than the revision encoded in the token. When another API server issued the token at a revision this cache has not observed, <b><i>listExactRV</i></b> selects an older snapshot and labels its objects with the future revision. </pre> <details> <summary><strong>Agent Prompt</strong></summary> ``` ## Issue description Positive continuation revisions can remain on the cache path even when they are newer than the local watch cache, but the subsequent list waits on the empty request resource version and can return an older snapshot labeled with the future revision. ## Fix Focus Areas - staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go[1351-1352] - staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go[730-766] - staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go[499-530] ## Recommended Fix Decode positive continuation revisions before waiting and use the decoded revision as the freshness target for recursive continuation requests. Add an end-to-end test where the continuation token revision is newer than the local cache and verify that listing blocks until that revision is observed rather than using an older snapshot. ``` <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> </details>
staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go:539
<img src="https://img.shields.io/badge/Medium-634FD1?style=flat-square" height="20px" alt="Remediation recommended"> 2\. Paginated reads add a storage call <code>🐞 Bug</code> <code>➹ Performance</code> <pre> <b><i>watchCache.list</i></b> treats the internally injected resource version on a limited consistent list as a legacy exact request, so <b><i>listExactRV</i></b> fails immediately when the snapshot feature is disabled. With consistent cache reads enabled by default and snapshots disabled by default, each such request first obtains the current storage revision and enters the cache before falling back to another storage list. </pre> <details> <summary><strong>Agent Prompt</strong></summary> ``` ## Issue description Limited consistent lists are converted internally to positive-resource-version requests and then routed into the snapshot-only legacy exact path, causing an additional storage operation whenever snapshots are disabled. ## Fix Focus Areas - staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go[537-539] - staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go[209-222] - staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go[1372-1376] ## Recommended Fix Distinguish an internally established consistent-read revision from a client legacy exact request, or make the delegation decision account for a positive limit and snapshot availability. Route unsupported paginated consistent reads directly to storage instead of obtaining a revision, entering the cache, and relying on `ResourceExpired` fallback. ``` <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> </details>
staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go:223
<img src="https://img.shields.io/badge/Medium-634FD1?style=flat-square" height="20px" alt="Remediation recommended"> 3\. Paginated reads vanish from metrics <code>🐞 Bug</code> <code>◔ Observability</code> <pre> <b><i>CacheDelegator.GetList</i></b> returns immediately from the new <b><i>ResourceExpired</i></b> fallback before updating <b><i>ConsistentReadTotal</i></b>. Limited consistent reads reach this branch whenever snapshots are disabled, so their success, failure, and fallback activity is absent from the cache consistency metric. </pre> <details> <summary><strong>Agent Prompt</strong></summary> ``` ## Issue description The new resource-expired storage fallback returns before the existing consistent-read metric update, leaving paginated consistent reads uncounted when the snapshot path is unavailable. ## Fix Focus Areas - staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go[220-239] ## Recommended Fix Do not return directly from the resource-expired branch. Assign the storage result to `err`, mark the operation as a fallback when `result.ConsistentRead` is true, and pass through the existing success and metric-recording logic before returning. ``` <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> </details>
PR-level
<h3>PR Summary by Qodo</h3> Serve exact and paginated LIST requests from cache snapshots <code>✨ Enhancement</code> <code>🐞 Bug fix</code> <code>🧪 Tests</code> <code>🕐 40+ Minutes</code> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <details> <summary>AI Description</summary> <dl> <dd> <br/> ><pre> >• Serve exact and continued LIST requests from retained watch-cache snapshots. >• Route unsupported or unavailable snapshot reads to backing storage. >• Expand coverage across watch-cache and snapshot feature-gate combinations. ></pre> </dd> </dl> </details> <details> <summary>Diagram</summary> <dl> <dd> <br/> ```mermaid graph TD A["List request"] --> B{"Cache eligible?"} B -- No --> G["Backing storage"] --> H["List response"] B -- Yes --> C{"Revision mode?"} C -- "Exact or continue" --> D{"Snapshot found?"} D -- Yes --> E["Snapshot list"] --> H D -- No --> G C -- "Latest or consistent" --> F["Current cache"] --> H ``` </dd> </dl> </details> <details> <summary>High-Level Assessment</summary> <dl> <dd> <br/> >The following are alternative approaches to this PR: <details> <summary><b>1. Always delegate historical reads</b></summary> <dl> <dd> - ➕ Keeps exact-version and pagination behavior in backing storage - ➕ Avoids snapshot availability races in the cache path - ➖ Forfeits snapshot-backed LIST latency and storage load reductions - ➖ Does not advance the ListFromCacheSnapshot feature </dd> </dl> </details> <details> <summary><b>2. Reconstruct revisions from watch events</b></summary> <dl> <dd> - ➕ Could reduce the number of complete snapshots retained - ➕ Uses the existing watch history as the historical source - ➖ Adds replay complexity and request-time latency - ➖ Makes deletion, compaction, and pagination correctness harder to guarantee </dd> </dl> </details> >**Recommendation:** Use the PR&#x27;s retained-snapshot approach because the snapshot infrastructure already exists and provides efficient ordered historical reads. Keep backing-storage delegation and ResourceExpired fallback as correctness safeguards for unsupported requests, missing snapshots, and lookup races. </dd> </dl> </details> <details> <summary> Files changed (8) <code> +299 / -35 </code> </summary> <dl> <dd> <br/> <details> <summary>Enhancement (2) <code> +93 / -5 </code></summary> <dl> <dd> <details> <summary>cacher.go<code>Route eligible historical LIST requests through cache snapshots</code> <code>+40/-3</code></summary> <br/> >Route eligible historical LIST requests through cache snapshots > ><pre> >• Replaces unconditional delegation for exact, continued, and consistent LIST requests. Recursive requests can remain in the cache when snapshots or consistent-read support can satisfy the requested revision; unsupported cases still delegate. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/93/files#diff-14c4fb9f290753f50e0af4856d871ff41e2b520d747158cbff7bafc577fbb29e'>staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go</a> </details> <details> <summary>watch_cache.go<code>Serve exact and continued LISTs from retained snapshots</code> <code>+53/-2</code></summary> <br/> >Serve exact and continued LISTs from retained snapshots > ><pre> >• Separates historical and latest LIST paths, decodes continue tokens, and reads exact revisions from retained ordered snapshots. Continued latest reads now honor their continuation key while preserving index-based filtering for ordinary lists. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/93/files#diff-dc9b33f648eee119a44322015b3c50f9dbd1c45ac7790084579dc0a4cd075de2'>staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go</a> </details> </dd> </dl> </details> <details> <summary>Bug fix (1) <code> +3 / -0 </code></summary> <dl> <dd> <details> <summary>delegator.go<code>Fall back when a cache snapshot expires</code> <code>+3/-0</code></summary> <br/> >Fall back when a cache snapshot expires > ><pre> >• Retries LIST requests against backing storage when the cache returns ResourceExpired, covering snapshots that become unavailable after the initial routing decision. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/93/files#diff-ee13ee70f26706a93bd3b501e895b4ba003a402c15c145c4259b5f5d521f62c3'>staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go</a> </details> </dd> </dl> </details> <details> <summary>Tests (5) <code> +203 / -30 </code></summary> <dl> <dd> <details> <summary>cacher_test.go<code>Exercise LIST behavior with snapshots enabled and disabled</code> <code>+27/-12</code></summary> <br/> >Exercise LIST behavior with snapshots enabled and disabled > ><pre> >• Expands list, consistent-list, and non-recursive test coverage across both ConsistentListFromCache and ListFromCacheSnapshot feature-gate states. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/93/files#diff-d82df65cc0625a3f621333245e76fbc801b579eedd3d266fe36eeb99366b5680'>staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_test.go</a> </details> <details> <summary>cacher_whitebox_test.go<code>Validate snapshot delegation and fallback decisions</code> <code>+158/-9</code></summary> <br/> >Validate snapshot delegation and fallback decisions > ><pre> >• Adds snapshot-aware delegation matrices for exact revisions, continue tokens, future revisions, and consistent reads. Adds an end-to-end fallback test covering available, disabled, and disappearing snapshots. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/93/files#diff-aed72203cf711bc5f518b68da5f6131358a0a79139efcbfa675f991951712168'>staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher_whitebox_test.go</a> </details> <details> <summary>store_test.go<code>Adapt etcd consistent-list tests to snapshot parameter</code> <code>+1/-1</code></summary> <br/> >Adapt etcd consistent-list tests to snapshot parameter > ><pre> >• Updates the shared consistent-list test invocation to indicate that direct etcd storage does not use watch-cache snapshots. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/93/files#diff-917ca994006290fce1488cc9cf10fb8ab43c359f8c73b822b11ccb1bf6225c04'>staging/src/k8s.io/apiserver/pkg/storage/etcd3/store_test.go</a> </details> <details> <summary>store_tests.go<code>Make consistent-list tests snapshot-aware</code> <code>+5/-2</code></summary> <br/> >Make consistent-list tests snapshot-aware > ><pre> >• Adds a snapshot-mode parameter to shared consistent-list tests, marks requests recursive, and updates expected cache synchronization revisions for negative continuation tokens served from snapshots. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/93/files#diff-7d38ad6f2c7145b3ca250b13a5bc81ddae1f67da77823e573a59d3646ffe557d'>staging/src/k8s.io/apiserver/pkg/storage/testing/store_tests.go</a> </details> <details> <summary>apiserver_test.go<code>Test LIST options across snapshot feature states</code> <code>+12/-6</code></summary> <br/> >Test LIST options across snapshot feature states > ><pre> >• Runs watch-cache LIST option integration tests with ListFromCacheSnapshot both enabled and disabled, while retaining the non-watch-cache scenario. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/93/files#diff-c5e700b1b521973df191a7218a45a6aecaa4081af17a43f9bfbca2b9a287cf0e'>test/integration/apiserver/apiserver_test.go</a> </details> </dd> </dl> </details> </dd> </dl> </details>
PR-level
<h3>Code Review by Qodo</h3> <code>🐞 Bugs (3)</code> <code>📘 Rule violations (0)</code> <code>📜 Skill insights (0)</code> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <br/> <img src="https://img.shields.io/badge/High-634FD1?style=flat-square" height="20px" alt="Action required"> <details> <summary> 1. Future pages can return stale objects <code>🐞 Bug</code> <code>≡ Correctness</code></summary> <br/> > <details open> ><summary>Description</summary> ><br/> > ><pre> ><b><i>ShouldDelegateContinue</i></b> keeps a future positive continuation revision in the cache path, but ><b><i>Cacher.GetList</i></b> waits using the empty request resource version rather than the revision encoded in >the token. When another API server issued the token at a revision this cache has not observed, ><b><i>listExactRV</i></b> selects an older snapshot and labels its objects with the future revision. ></pre> ></details> > <details> ><summary>Code</summary> ><br/> > ><code>[staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go[R1351-1352]](https://github.com/drape-io/crbench-qodo-2026-09/pull/93/files#diff-14c4fb9f290753f50e0af4856d871ff41e2b520d747158cbff7bafc577fbb29eR1351-R1352)</code> > >```diff >+ if continueRV > 0 { >+ return c.shouldDelegateExactRV(uint64(continueRV)) >``` ></details> > <details> ><summary>Evidence</summary> ><br/> > ><pre> >The delegation decision explicitly retains future continuation revisions when consistent cache reads >are supported. The list path parses only <b><i>opts.ResourceVersion</i></b>, which is empty for a valid >continuation request, while snapshot lookup accepts any snapshot less than or equal to the future >token revision and reports that requested revision in the response. ></pre> > > <code>[staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go[1351-1368]](https://github.com/drape-io/crbench-qodo-2026-09/blob/e3b1319db0e83c3fb7462cdc2abf8396fbae099a/staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go/#L1351-L1368)</code> > <code>[staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go[726-766]](https://github.com/drape-io/crbench-qodo-2026-09/blob/e3b1319db0e83c3fb7462cdc2abf8396fbae099a/staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go/#L726-L766)</code> > <code>[staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go[499-530]](https://github.com/drape-io/crbench-qodo-2026-09/blob/e3b1319db0e83c3fb7462cdc2abf8396fbae099a/staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go/#L499-L530)</code> > <code>[staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go[546-558]](https://github.com/drape-io/crbench-qodo-2026-09/blob/e3b1319db0e83c3fb7462cdc2abf8396fbae099a/staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go/#L546-L558)</code> > <code>[staging/src/k8s.io/apiserver/pkg/storage/cacher/store_btree.go[464-476]](https://github.com/drape-io/crbench-qodo-2026-09/blob/e3b1319db0e83c3fb7462cdc2abf8396fbae099a/staging/src/k8s.io/apiserver/pkg/storage/cacher/store_btree.go/#L464-L476)</code> ></details> > <details> ><summary>Agent prompt</summary> ><br/> > >``` >The issue below was found during a code review. Follow the provided context and guidance below and implement a solution > >## Issue description >Positive continuation revisions can remain on the cache path even when they are newer than the local watch cache, but the subsequent list waits on the empty request resource version and can return an older snapshot labeled with the future revision. > >## Fix Focus Areas >- staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go[1351-1352] >- staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go[730-766] >- staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go[499-530] > >## Recommended Fix >Decode positive continuation revisions before waiting and use the decoded revision as the freshness target for recursive continuation requests. Add an end-to-end test where the continuation token revision is newer than the local cache and verify that listing blocks until that revision is observed rather than using an older snapshot. >``` > <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> ></details> <hr/> </details> <br/> <img src="https://img.shields.io/badge/Medium-634FD1?style=flat-square" height="20px" alt="Remediation recommended"> <details> <summary> 2. Paginated reads add a storage call <code>🐞 Bug</code> <code>➹ Performance</code></summary> <br/> > <details open> ><summary>Description</summary> ><br/> > ><pre> ><b><i>watchCache.list</i></b> treats the internally injected resource version on a limited consistent list as a >legacy exact request, so <b><i>listExactRV</i></b> fails immediately when the snapshot feature is disabled. With >consistent cache reads enabled by default and snapshots disabled by default, each such request first >obtains the current storage revision and enters the cache before falling back to another storage >list. ></pre> ></details> > <details> ><summary>Code</summary> ><br/> > ><code>[staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go[R537-539]](https://github.com/drape-io/crbench-qodo-2026-09/pull/93/files#diff-dc9b33f648eee119a44322015b3c50f9dbd1c45ac7790084579dc0a4cd075de2R537-R539)</code> > >```diff >+ // Legacy exact match >+ if opts.Predicate.Limit > 0 && len(opts.ResourceVersion) > 0 && opts.ResourceVersion != "0" { >+ return w.listExactRV(key, "", resourceVersion) >``` ></details> > <details> ><summary>Evidence</summary> ><br/> > ><pre> >The delegator injects a positive current revision for consistent reads before calling the cacher. >The new legacy-exact condition then invokes <b><i>listExactRV</i></b>, which returns <b><i>ResourceExpired</i></b> whenever >snapshots are disabled; repository feature definitions show consistent cache reads default on while >snapshots default off. ></pre> > > <code>[staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go[209-222]](https://github.com/drape-io/crbench-qodo-2026-09/blob/e3b1319db0e83c3fb7462cdc2abf8396fbae099a/staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go/#L209-L222)</code> > <code>[staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go[537-552]](https://github.com/drape-io/crbench-qodo-2026-09/blob/e3b1319db0e83c3fb7462cdc2abf8396fbae099a/staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go/#L537-L552)</code> > <code>[staging/src/k8s.io/apiserver/pkg/features/kube_features.go[315-318]](https://github.com/drape-io/crbench-qodo-2026-09/blob/e3b1319db0e83c3fb7462cdc2abf8396fbae099a/staging/src/k8s.io/apiserver/pkg/features/kube_features.go/#L315-L318)</code> > <code>[staging/src/k8s.io/apiserver/pkg/features/kube_features.go[331-333]](https://github.com/drape-io/crbench-qodo-2026-09/blob/e3b1319db0e83c3fb7462cdc2abf8396fbae099a/staging/src/k8s.io/apiserver/pkg/features/kube_features.go/#L331-L333)</code> ></details> > <details> ><summary>Agent prompt</summary> ><br/> > >``` >The issue below was found during a code review. Follow the provided context and guidance below and implement a solution > >## Issue description >Limited consistent lists are converted internally to positive-resource-version requests and then routed into the snapshot-only legacy exact path, causing an additional storage operation whenever snapshots are disabled. > >## Fix Focus Areas >- staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go[537-539] >- staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go[209-222] >- staging/src/k8s.io/apiserver/pkg/storage/cacher/cacher.go[1372-1376] > >## Recommended Fix >Distinguish an internally established consistent-read revision from a client legacy exact request, or make the delegation decision account for a positive limit and snapshot availability. Route unsupported paginated consistent reads directly to storage instead of obtaining a revision, entering the cache, and relying on `ResourceExpired` fallback. >``` > <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> ></details> <hr/> </details> <details> <summary> 3. Paginated reads vanish from metrics <code>🐞 Bug</code> <code>◔ Observability</code></summary> <br/> > <details open> ><summary>Description</summary> ><br/> > ><pre> ><b><i>CacheDelegator.GetList</i></b> returns immediately from the new <b><i>ResourceExpired</i></b> fallback before updating ><b><i>ConsistentReadTotal</i></b>. Limited consistent reads reach this branch whenever snapshots are disabled, >so their success, failure, and fallback activity is absent from the cache consistency metric. ></pre> ></details> > <details> ><summary>Code</summary> ><br/> > ><code>[staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go[R220-223]](https://github.com/drape-io/crbench-qodo-2026-09/pull/93/files#diff-ee13ee70f26706a93bd3b501e895b4ba003a402c15c145c4259b5f5d521f62c3R220-R223)</code> > >```diff > if err != nil { >+ if errors.IsResourceExpired(err) { >+ return c.storage.GetList(ctx, key, opts, listObj) >+ } >``` ></details> > <details> ><summary>Evidence</summary> ><br/> > ><pre> >The immediate return precedes both metric update sites. Limited consistent reads use the exact >snapshot path after their current revision is injected, and a missing snapshot produces the ><b><i>ResourceExpired</i></b> error that activates this early return. ></pre> > > <code>[staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go[209-239]](https://github.com/drape-io/crbench-qodo-2026-09/blob/e3b1319db0e83c3fb7462cdc2abf8396fbae099a/staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go/#L209-L239)</code> > <code>[staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go[537-552]](https://github.com/drape-io/crbench-qodo-2026-09/blob/e3b1319db0e83c3fb7462cdc2abf8396fbae099a/staging/src/k8s.io/apiserver/pkg/storage/cacher/watch_cache.go/#L537-L552)</code> > <code>[staging/src/k8s.io/apiserver/pkg/storage/cacher/metrics/metrics.go[171-178]](https://github.com/drape-io/crbench-qodo-2026-09/blob/e3b1319db0e83c3fb7462cdc2abf8396fbae099a/staging/src/k8s.io/apiserver/pkg/storage/cacher/metrics/metrics.go/#L171-L178)</code> ></details> > <details> ><summary>Agent prompt</summary> ><br/> > >``` >The issue below was found during a code review. Follow the provided context and guidance below and implement a solution > >## Issue description >The new resource-expired storage fallback returns before the existing consistent-read metric update, leaving paginated consistent reads uncounted when the snapshot path is unavailable. > >## Fix Focus Areas >- staging/src/k8s.io/apiserver/pkg/storage/cacher/delegator.go[220-239] > >## Recommended Fix >Do not return directly from the resource-expired branch. Assign the storage result to `err`, mark the operation as a fallback when `result.ConsistentRead` is true, and pass through the existing success and metric-recording logic before returning. >``` > <code>ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools</code> ></details> <hr/> </details> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <!-- qodo-context:start --> <details><summary><strong>Context sources</strong></summary> <div>Review mode: <code>⚖️ Balanced</code>: Downgraded extended -&gt; standard: change is below the extended eligibility bar (hunks 16/18, lines 334/200; both must reach the floor). Router rationale: This changes Kubernetes apiserver cache delegation, exact resource-version semantics, snapshot fallback, continuation handling, and integration behavior across multiple independent code paths, creating a dense set of subtle correctness risks.</div> <!-- qodo-context:end --> </details> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <!-- qodo-daily-tip:start --> <details> <summary><strong>Tip of the day</strong></summary> <br/> <pre>💡 Did you know, you can reply &#x27;qodo&#x27; on any finding to push back, ask questions, or dig deeper</pre> <a href="https://docs.qodo.ai/tips-and-tricks">More tips ↗</a> | <a href="https://app.qodo.ai/configurations?tab=display-preferences">Customize Qodo ↗</a> | <a href="https://docs.qodo.ai">Qodo docs ↗</a> </details> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <!-- qodo-daily-tip:end --> <!-- https://github.com/drape-io/crbench-qodo-2026-09/commit/e3b1319db0e83c3fb7462cdc2abf8396fbae099a --> <a href="https://www.qodo.ai"><img src="https://www.qodo.ai/wp-content/uploads/2025/03/qodo-logo.svg" width="80" alt="Qodo Logo"></a>