crbench//

← all cases

Dual Storage Architecture

Tier 2 medium go · local · real merged PR from grafana/grafana #90045

Ground-truth defects

d1 · other · low
The context is being created with d.Log instead of the log variable that was initialized with additional context values (name, kind, method). This means those values won't be propagated to the logging context.
d2 · other · low
Bug: calling recordLegacyDuration when storage operation fails should be recordStorageDuration.
d3 · other · medium
Inconsistency: using name instead of options.Kind for metrics recording differs from other methods.
d4 · correctness · high
All four write methods now hand the legacy write to a goroutine whose context descends from the request context (`context.WithTimeoutCause(ctx, ...)`). net/http cancels the request context as soon as the handler returns, and the goroutine starts microseconds before that, so the legacy write is aborted mid-flight on essentially every request and unified storage silently diverges from legacy storage.

The change under review

diff (truncated — full diff)
diff --git a/go.work.sum b/go.work.sum
index 2555950b69..542e4c1d14 100644
--- a/go.work.sum
+++ b/go.work.sum
@@ -402,6 +402,12 @@ github.com/grafana/e2e v0.1.1/go.mod h1:RpNLgae5VT+BUHvPE+/zSypmOXKwEu4t+tnEMS1A
 github.com/grafana/grafana-cloud-migration-snapshot v1.2.0 h1:FCUWASPPzGGbF2jTutR5i3rmoQdmnC4bypwJswdW3fI=
 github.com/grafana/grafana-cloud-migration-snapshot v1.2.0/go.mod h1:bd6Cm06EK0MzRO5ahUpbDz1SxNOKu+fzladbaRPHZPY=
 github.com/grafana/grafana/pkg/apimachinery v0.0.0-20240701135906-559738ce6ae1/go.mod h1:DkxMin+qOh1Fgkxfbt+CUfBqqsCQJMG9op8Os/irBPA=
+github.com/grafana/grafana-azure-sdk-go/v2 v2.1.0 h1:lajVqTWaE96MpbjZToj7EshvqgRWOfYNkD4MbIZizaY=
+github.com/grafana/grafana-azure-sdk-go/v2 v2.1.0/go.mod h1:aKlFPE36IDa8qccRg3KbgZX3MQ5xymS3RelT4j6kkVU=
+github.com/grafana/grafana-plugin-sdk-go v0.235.0/go.mod h1:6n9LbrjGL3xAATntYVNcIi90G9BVHRJjzHKz5FXVfWw=
+github.com/grafana/grafana/pkg/apimachinery v0.0.0-20240701135906-559738ce6ae1/go.mod h1:DkxMin+qOh1Fgkxfbt+CUfBqqsCQJMG9op8Os/irBPA=
+github.com/grafana/prometheus-alertmanager v0.25.1-0.20240422145632-c33c6b5b6e6b h1:HCbWyVL6vi7gxyO76gQksSPH203oBJ1MJ3JcG1OQlsg=
+github.com/grafana/prometheus-alertmanager v0.25.1-0.20240422145632-c33c6b5b6e6b/go.mod h1:01sXtHoRwI8W324IPAzuxDFOmALqYLCOhvSC2fUHWXc=
 github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7 h1:pdN6V1QBWetyv/0+wjACpqVH+eVULgEjkurDLq3goeM=
 github.com/grpc-ecosystem/grpc-opentracing v0.0.0-20180507213350-8e809c8a8645 h1:MJG/KsmcqMwFAkh8mTnAwhyKoB+sTAnY4CACC110tbU=
 github.com/grpc-ecosystem/grpc-opentracing v0.0.0-20180507213350-8e809c8a8645/go.mod h1:6iZfnjpejD4L/4DwD7NryNaJyCQdzwWwH2MWhCA90Kw=
@@ -793,15 +799,24 @@ go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.23.1 h1:ZqR
 go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.23.1/go.mod h1:D7ynngPWlGJrqyGSDOdscuv7uqttfCE3jcBvffDv9y4=
 go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.23.1 h1:q/Nj5/2TZRIt6PderQ9oU0M00fzoe8UZuINGw6ETGTw=
 go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.23.1/go.mod h1:DTE9yAu6r08jU3xa68GiSeI7oRcSEQ2RpKbbQGO+dWM=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.26.0/go.mod h1:z46paqbJ9l7c9fIPCXTqTGwhQZ5XoTIsfeFYWboizjs=
+go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.26.0/go.mod h1:wnJIG4fOqyynOnnQF/eQb4/16VlX2EJAHhHgqIqWfAo=
+go.opentelemetry.io/otel/exporters/prometheus v0.37.0 h1:NQc0epfL0xItsmGgSXgfbH2C1fq2VLXkZoDFsfRNHpc=
+go.opentelemetry.io/otel/exporters/prometheus v0.37.0/go.mod h1:hB8qWjsStK36t50/R0V2ULFb4u95X/Q6zupXLgvjTh8=
 go.opentelemetry.io/otel/exporters/prometheus v0.46.0 h1:I8WIFXR351FoLJYuloU4EgXbtNX2URfU/85pUPheIEQ=
 go.opentelemetry.io/otel/exporters/prometheus v0.46.0/go.mod h1:ztwVUHe5DTR/1v7PeuGRnU5Bbd4QKYwApWmuutKsJSs=
 go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.23.1 h1:C8r95vDR125t815KD+b1tI0Fbc1pFnwHTBxkbIZ6Szc=
 go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.23.1/go.mod h1:Qr0qomr64jentMtOjWMbtYeJMSuMSlsPEjmnRA2sWZ4=
 go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.24.0 h1:s0PHtIkN+3xrbDOpt2M8OTG92cWqUESvzh2MxiR5xY8=
 go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.24.0/go.mod h1:hZlFbDbRt++MMPCCfSJfmhkGIWnX1h3XjkfxZUjLrIA=
+go.opentelemetry.io/otel/metric v1.26.0/go.mod h1:SY+rHOI4cEawI9a7N1A4nIg/nTQXe1ccCNWYOJUrpX4=
+go.opentelemetry.io/otel/sdk v1.26.0/go.mod h1:0p8MXpqLeJ0pzcszQQN4F0S5FVjBLgypeGSngLsmirs=
+go.opentelemetry.io/otel/sdk/metric v0.39.0 h1:Kun8i1eYf48kHH83RucG93ffz0zGV1sh46FAScOTuDI=
+go.opentelemetry.io/otel/sdk/metric v0.39.0/go.mod h1:piDIRgjcK7u0HCL5pCA4e74qpK/jk3NiUoAHATVAmiI=
 go.opentelemetry.io/otel/sdk/metric v1.26.0 h1:cWSks5tfriHPdWFnl+qpX3P681aAYqlZHcAyHw5aU9Y=
 go.opentelemetry.io/otel/sdk/metric v1.26.0/go.mod h1:ClMFFknnThJCksebJwz7KIyEDHO+nTB6gK8obLy8RyE=
 go.opentelemetry.io/otel/trace v1.26.0/go.mod h1:4iDxvGDQuUkHve82hJJ8UqrwswHYsZuWCBllGV2U2y0=
+go.opentelemetry.io/proto/otlp v1.2.0/go.mod h1:gGpR8txAl5M03pDhMC79G6SdqNV26naRm/KDsgaHD8A=
 go.uber.org/automaxprocs v1.5.3 h1:kWazyxZUrS3Gs4qUpbwo5kEIMGe/DAvi5Z4tl2NW4j8=
 go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee h1:0mgffUl7nfd+FpvXMVz4IDEaUSmT1ysygQC7qYo7sG4=
 golang.org/x/arch v0.4.0 h1:A8WCeEWhLwPBKNbFi5Wv5UTCBx5zzubnXDlMOFAzFMc=
@@ -825,6 +840,7 @@ google.golang.org/genproto/googleapis/api v0.0.0-20240318140521-94a12d6c2237/go.
 google.golang.org/genproto/googleapis/api v0.0.0-20240604185151-ef581f913117/go.mod h1:OimBR/bc1wPO9iV4NC2bpyjy3VnAwZh5EBPQdtaE5oo=
 google.golang.org/genproto/googleapis/bytestream v0.0.0-20240325203815-454cdb8f5daa h1:wBkzraZsSqhj1M4L/nMrljUU6XasJkgHvUsq8oRGwF0=
 google.golang.org/genproto/googleapis/rpc v0.0.0-20240318140521-94a12d6c2237/go.mod h1:WtryC6hu0hhx87FDGxWCDptyssuo68sk10vYjF+T9fY=
+google.golang.org/genproto/googleapis/rpc v0.0.0-20240513163218-0867130af1f8/go.mod h1:I7Y+G38R2bu5j1aLzfFmQfTcU/WnFuqDwLZAbvKTKpM=
 google.golang.org/genproto/googleapis/rpc v0.0.0-20240604185151-ef581f913117/go.mod h1:EfXuqaE1J41VCDicxHzUDm+8rk+7ZdXzHV0IhO/I6s0=
 google.golang.org/genproto/googleapis/rpc v0.0.0-20240624140628-dc46fd24d27d/go.mod h1:Ue6ibwXGpU+dqIcODieyLOcgj7z8+IcskoNIgZxtrFY=
 google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.1.0 h1:M1YKkFIboKNieVO5DLUEVzQfGwJD30Nv2jfUgzb5UcE=
diff --git a/pkg/apiserver/rest/dualwriter_mode1_test.go b/pkg/apiserver/rest/dualwriter_mode1_test.go
index a9efdf2226..9e1fdff5ad 100644
--- a/pkg/apiserver/rest/dualwriter_mode1_test.go
+++ b/pkg/apiserver/rest/dualwriter_mode1_test.go
@@ -135,7 +135,6 @@ func TestMode1_Get(t *testing.T) {
 				tt.setupStorageFn(m, tt.input)
 			}
 
-			p := prometheus.NewRegistry()
 			dw := NewDualWriter(Mode1, ls, us, p)
 
 			obj, err := dw.Get(context.Background(), tt.input, &metav1.GetOptions{})
diff --git a/pkg/apiserver/rest/dualwriter_mode3.go b/pkg/apiserver/rest/dualwriter_mode3.go
index 12bf63865a..5d1c80df90 100644
--- a/pkg/apiserver/rest/dualwriter_mode3.go
+++ b/pkg/apiserver/rest/dualwriter_mode3.go
@@ -2,8 +2,9 @@ package rest
 
 import (
 	"context"
+	"errors"
+	"time"
 
-	apierrors "k8s.io/apimachinery/pkg/api/errors"
 	metainternalversion "k8s.io/apimachinery/pkg/apis/meta/internalversion"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/runtime"
@@ -21,7 +22,7 @@ type DualWriterMode3 struct {
 // newDualWriterMode3 returns a new DualWriter in mode 3.
 // Mode 3 represents writing to LegacyStorage and Storage and reading from Storage.
 func newDualWriterMode3(legacy LegacyStorage, storage Storage, dwm *dualWriterMetrics) *DualWriterMode3 {
-	return &DualWriterMode3{Legacy: legacy, Storage: storage, Log: klog.NewKlogr().WithName("DualWriterMode3"), dualWriterMetrics: dwm}
+	return &DualWriterMode3{Legacy: legacy, Storage: storage, Log: klog.NewKlogr().WithName("DualWriterMode3").WithValues("mode", mode3Str), dualWriterMetrics: dwm}
 }
 
 // Mode returns the mode of the dual writer.
@@ -29,106 +30,143 @@ func (d *DualWriterMode3) Mode() DualWriterMode {
 	return Mode3
 }
 
+const mode3Str = "3"
+
 // Create overrides the behavior of the generic DualWriter and writes to LegacyStorage and Storage.
 func (d *DualWriterMode3) Create(ctx context.Context, obj runtime.Object, createValidation rest.ValidateObjectFunc, options *metav1.CreateOptions) (runtime.Object, error) {
-	log := klog.FromContext(ctx)
+	var method = "create"
+	log := d.Log.WithValues("kind", options.Kind, "method", method)
+	ctx = klog.NewContext(ctx, log)
 
+	startStorage := time.Now()
 	created, err := d.Storage.Create(ctx, obj, createValidation, options)
 	if err != nil {
 		log.Error(err, "unable to create object in storage")
+		d.recordLegacyDuration(true, mode3Str, options.Kind, method, startStorage)
 		return created, err
 	}
+	d.recordStorageDuration(false, mode3Str, options.Kind, method, startStorage)
 
-	if _, err := d.Legacy.Create(ctx, obj, createValidation, options); err != nil {
-		log.WithValues("object", created).Error(err, "unable to create object in legacy storage")
-	}
-	return created, nil
+	go func() {
+		ctx, cancel := context.WithTimeoutCause(ctx, time.Second*10, errors.New("legacy create timeout"))
+		defer cancel()
+
+		startLegacy := time.Now()
+		_, errObjectSt := d.Legacy.Create(ctx, obj, createValidation, options)
+		d.recordLegacyDuration(errObjectSt != nil, mode3Str, options.Kind, method, startLegacy)
+	}()
+
+	return created, err
 }
 
 // Get overrides the behavior of the generic DualWriter and retrieves an object from Storage.
 func (d *DualWriterMode3) Get(ctx context.Context, name string, options *metav1.GetOptions) (runtime.Object, error) {
-	return d.Storage.Get(ctx, name, &metav1.GetOptions{})
+	var method = "get"
+	log := d.Log.WithValues("kind", options.Kind, "name", name, "method", method)
+	ctx = klog.NewContext(ctx, log)
+
+	startStorage := time.Now()
+	res, err := d.Storage.Get(ctx, name, options)
+	if err != nil {
+		log.Error(err, "unable to get object in storage")
+	}
+	d.recordStorageDuration(err != nil, mode3Str, options.Kind, method, startStorage)
+
+	return res, err
 }
 
-func (d *DualWriterMode3) Delete(ctx context.Context, name string, deleteValidation rest.ValidateObjectFunc, options *metav1.DeleteOptions) (runtime.Object, bool, error) {
-	log := d.Log.WithValues("name", name)
+// List overrides the behavior of the generic DualWriter and reads only from Unified Store.
+func (d *DualWriterMode3) List(ctx context.Context, options *metainternalversion.ListOptions) (runtime.Object, error) {
+	var method = "list"
+	log := d.Log.WithValues("kind", options.Kind, "resourceVersion", options.ResourceVersion, "method", method)
 	ctx = klog.NewContext(ctx, log)
 
-	deleted, async, err := d.Storage.Delete(ctx, name, deleteValidation, options)
+	startStorage := time.Now()
+	res, err := d.Storage.List(ctx, options)
 	if err != nil {
-		if !apierrors.IsNotFound(err) {
-			log.Error(err, "could not delete from unified store")
-			return deleted, async, err
-		}
+		log.Error(err, "unable to list object in storage")
 	}
+	d.recordStorageDuration(err != nil, mode3Str, options.Kind, method, startStorage)
 
-	_, _, errLS := d.Legacy.Delete(ctx, name, deleteValidation, options)
-	if errLS != nil {
-		if !apierrors.IsNotFound(errLS) {
-			log.WithValues("deleted", deleted).Error(errLS, "could not delete from legacy store")
-		}
+	return res, err
+}
+
+func (d *DualWriterMode3) Delete(ctx context.Context, name string, deleteValidation rest.ValidateObjectFunc, options *metav1.DeleteOptions) (runtime.Object, bool, error) {
+	var method = "delete"
+	log := d.Log.WithValues("name", name, "kind", options.Kind, "method", method)
+	ctx = klog.NewContext(ctx, d.Log)
+
+	startStorage := time.Now()
+	res, async, err := d.Storage.Delete(ctx, name, deleteValidation, options)
+	if err != nil {
+		log.Error(err, "unable to delete object in storage")
+		d.recordStorageDuration(true, mode3Str, options.Kind, method, startStorage)
+		return res, async, err
 	}
+	d.recordStorageDuration(false, mode3Str, name, method, startStorage)
 
-	return deleted, async, err
+	go func() {
+		startLegacy := time.Now()
+		ctx, cancel := context.WithTimeoutCause(ctx, time.Second*10, errors.New("legacy delete timeout"))
+		defer cancel()
+		_, _, err := d.Legacy.Delete(ctx, name, deleteValidation, options)
+		d.recordLegacyDuration(err != nil, mode3Str, options.Kind, method, startLegacy)
+	}()
+
+	return res, async, err
 }
 
 // Update overrides the behavior of the generic DualWriter and writes first to Storage and then to LegacyStorage.
 func (d *DualWriterMode3) Update(ctx context.Context, name string, objInfo rest.UpdatedObjectInfo, createValidation rest.ValidateObjectFunc, updateValidation rest.ValidateObjectUpdateFunc, forceAllowCreate bool, options *metav1.UpdateOptions) (runtime.Object, bool, error) {
-	log := d.Log.WithValues("name", name)
+	var method = "update"
+	log := d.Log.WithValues("name", name, "kind", options.Kind, "method", method)
 	ctx = klog.NewContext(ctx, log)
-	old, err := d.Storage.Get(ctx, name, &metav1.GetOptions{})
-	if err != nil {
-		log.WithValues("object", old).Error(err, "could not get object to update")
-		return nil, false, err
-	}
 
-	updated, err := objInfo.UpdatedObject(ctx, old)
+	startStorage := time.Now()
+	res, async, err := d.Storage.Update(ctx, name, objInfo, createValidation, updateValidation, forceAllowCreate, options)
 	if err != nil {
-		log.WithValues("object", updated).Error(err, "could not update or create object")
-		return nil, false, err
-	}
-	objInfo = &updateWrapper{
-		upstream: objInfo,
-		updated:  updated,
+		log.Error(err, "unable to update in storage")
+		d.recordLegacyDuration(true, mode3Str, options.Kind, method, startStorage)
+		return res, async, err
 	}
+	d.recordStorageDuration(false, mode3Str, options.Kind, method, startStorage)
 
-	obj, created, err := d.Storage.Update(ctx, name, objInfo, createValidation, updateValidation, forceAllowCreate, options)
-	if err != nil {
-		log.WithValues("object", obj).Error(err, "could not write to US")
-		return obj, created, err
-	}
+	go func() {
+		ctx, cancel := context.WithTimeoutCause(ctx, time.Second*10, errors.New("legacy update timeout"))
 
-	_, _, errLeg := d.Legacy.Update(ctx, name, &updateWrapper{
-		upstream: objInfo,
-		updated:  obj,
-	}, createValidation, updateValidation, forceAllowCreate, options)
-	if errLeg != nil {
-		log.Error(errLeg, "could not update object in legacy store")
-	}
-	return obj, created, err
+		startLegacy := time.Now()
+		defer cancel()
+		_, _, errObjectSt := d.Legacy.Update(ctx, name, objInfo, createValidation, updateValidation, forceAllowCreate, options)
+		d.recordLegacyDuration(errObjectSt != nil, mode3Str, options.Kind, method, startLegacy)
+	}()
+
+	return res, async, err
 }
 
 // DeleteCollection overrides the behavior of the generic DualWriter and deletes from both LegacyStorage and Storage.
 func (d *DualWriterMode3) DeleteCollection(ctx context.Context, deleteValidation rest.ValidateObjectFunc, options *metav1.DeleteOptions, listOptions *metainternalversion.ListOptions) (runtime.Object, error) {
-	log := d.Log.WithValues("kind", options.Kind, "resourceVersion", listOptions.ResourceVersion)
+	var method = "delete-collection"
+	log := d.Log.WithValues("kind", options.Kind, "resourceVersion", listOptions.ResourceVersion, "method", method)
 	ctx = klog.NewContext(ctx, log)
 
-	deleted, err := d.Storage.DeleteCollection(ctx, deleteValidation, options, listOptions)
+	startStorage := time.Now()
+	res, err := d.Storage.DeleteCollection(ctx, deleteValidation, options, listOptions)
 	if err != nil {
-		log.Error(err, "failed to delete collection successfully from Storage")
+		log.Error(err, "unable to delete collection in storage")
+		d.recordStorageDuration(true, mode3Str, options.Kind, method, startStorage)
+		return res, err
 	}
+	d.recordStorageDuration(false, mode3Str, options.Kind, method, startStorage)
 
-	if deleted, err := d.Legacy.DeleteCollection(ctx, deleteValidation, options, listOptions); err != nil {
-		log.WithValues("deleted", deleted).Error(err, "failed to delete collection successfully from LegacyStorage")
-	}
+	go func() {
+		startLegacy := time.Now()
+		ctx, cancel := context.WithTimeoutCause(ctx, time.Second*10, errors.New("legacy deletecollection timeout"))
+		defer cancel()
+		_, err := d.Legacy.DeleteCollection(ctx, deleteValidation, options, listOptions)
+		d.recordStorageDuration(err != nil, mode3Str, options.Kind, method, startLegacy)
+	}()
 
-	return deleted, err
-}
-
-func (d *DualWriterMode3) List(ctx context.Context, options *metainternalversion.ListOptions) (runtime.Object, error) {
-	//TODO: implement List
-	klog.Error("List not implemented")
-	return nil, nil
+	return res, err
 }
 
 func (d *DualWriterMode3) Destroy() {
diff --git a/pkg/apiserver/rest/dualwriter_mode3_test.go b/pkg/apiserver/rest/dualwriter_mode3_test.go
index df5deac228..b32e6fe3da 100644
--- a/pkg/apiserver/rest/dualwriter_mode3_test.go
+++ b/pkg/apiserver/rest/dualwriter_mode3_test.go
@@ -1,72 +1,356 @@
 package rest
 
-// import (
-// 	"context"
-// 	"testing"
-
-// 	"github.com/stretchr/testify/assert"
-// 	metainternalversion "k8s.io/apimachinery/pkg/apis/meta/internalversion"
-// 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-// 	"k8s.io/apimachinery/pkg/runtime"
-// 	"k8s.io/apiserver/pkg/apis/example"
-// )
-
-// func TestMode3(t *testing.T) {
-// 	var ls = (LegacyStorage)(nil)
-// 	var s = (Storage)(nil)
-// 	lsSpy := NewLegacyStorageSpyClient(ls)
-// 	sSpy := NewStorageSpyClient(s)
-
-// 	dw := NewDualWriterMode3(lsSpy, sSpy)
-
-// 	// Create: it should use the Legacy Create implementation
-// 	_, err := dw.Create(context.Background(), &dummyObject{}, func(context.Context, runtime.Object) error { return nil }, &metav1.CreateOptions{})
-// 	assert.NoError(t, err)
-// 	assert.Equal(t, 1, lsSpy.Counts("LegacyStorage.Create"))
-// 	assert.Equal(t, 1, sSpy.Counts("Storage.Create"))
-
-// 	// Get: it should use the Storage Get implementation
-// 	_, err = dw.Get(context.Background(), kind, &metav1.GetOptions{})
-// 	assert.NoError(t, err)
-// 	assert.Equal(t, 0, lsSpy.Counts("LegacyStorage.Get"))
-// 	assert.Equal(t, 1, sSpy.Counts("Storage.Get"))
-
-// 	// List: it should use the Storage List implementation
-// 	_, err = dw.List(context.Background(), &metainternalversion.ListOptions{})
-// 	assert.NoError(t, err)
-// 	assert.Equal(t, 0, lsSpy.Counts("LegacyStorage.List"))
-// 	assert.Equal(t, 1, sSpy.Counts("Storage.List"))
-
-// 	// Delete: it should use call both Legacy and Storage Delete methods
-// 	var deleteValidation = func(ctx context.Context, obj runtime.Object) error { return nil }
-// 	_, _, err = dw.Delete(context.Background(), kind, deleteValidation, &metav1.DeleteOptions{})
-// 	assert.NoError(t, err)
-// 	assert.Equal(t, 1, lsSpy.Counts("LegacyStorage.Delete"))
-// 	assert.Equal(t, 1, sSpy.Counts("Storage.Delete"))
-
-// 	// DeleteCollection: it should delete from both LegacyStorage and Storage
-// 	_, err = dw.DeleteCollection(
-// 		context.Background(),
-// 		func(context.Context, runtime.Object) error { return nil },
-// 		&metav1.DeleteOptions{},
-// 		&metainternalversion.ListOptions{},
-// 	)
-// 	assert.NoError(t, err)
-// 	assert.Equal(t, 1, lsSpy.Counts("LegacyStorage.DeleteCollection"))
-// 	assert.Equal(t, 1, sSpy.Counts("Storage.DeleteCollection"))
-
-// 	// Update: it should update in both storages
-// 	dummy := &example.Pod{}
-// 	uoi := UpdatedObjInfoObj{}
-// 	_, err = uoi.UpdatedObject(context.Background(), dummy)
-// 	assert.NoError(t, err)
-
-// 	var validateObjFn = func(ctx context.Context, obj runtime.Object) error { return nil }
-// 	var validateObjUpdateFn = func(ctx context.Context, obj, old runtime.Object) error { return nil }
-
-// 	_, _, err = dw.Update(context.Background(), kind, uoi, validateObjFn, validateObjUpdateFn, false, &metav1.UpdateOptions{})
-// 	assert.NoError(t, err)
-// 	assert.Equal(t, 1, lsSpy.Counts("LegacyStorage.Update"))
-// 	assert.Equal(t, 1, sSpy.Counts("Storage.Update"))
-// 	assert.NoError(t, err)
-// }
+import (
+	"context"
+	"errors"
+	"testing"
+
+	"github.com/prometheus/client_golang/prometheus"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/mock"
+	"k8s.io/apimachinery/pkg/api/meta"
+	metainternalversion "k8s.io/apimachinery/pkg/apis/meta/internalversion"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/runtime"
+)
+
+func TestMode3_Create(t *testing.T) {
+	type testCase struct {
+		input          runtime.Object
+		setupLegacyFn  func(m *mock.Mock, input runtime.Object)
+		setupStorageFn func(m *mock.Mock)
+		name           string
+		wantErr        bool
+	}
+	tests :=
+		[]testCase{
+			{
+				name:  "creating an object only in the unified store",
+				input: exampleObj,
+				setupLegacyFn: func(m *mock.Mock, input runtime.Object) {
+					m.On("Create", mock.Anything, input, mock.Anything, mock.Anything).Return(exampleObj, nil)
+				},
+				setupStorageFn: func(m *mock.Mock) {
+					m.On("Create", mock.Anything, mock.Anything, mock.Anything, mock.Anything).Return(exampleObj, nil)
+				},
+			},
+			{
+				name:  "error when creating object in the unified store fails",
+				input: failingObj,
+				setupLegacyFn: func(m *mock.Mock, input runtime.Object) {
+					m.On("Create", mock.Anything, failingObj, mock.Anything, mock.Anything).Return(nil, errors.New("error"))
+				},
+				wantErr: true,
+			},
+		}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			l := (LegacyStorage)(nil)
+			s := (Storage)(nil)
+			m := &mock.Mock{}
+
+			ls := legacyStoreMock{m, l}
+			us := storageMock{m, s}
+
+			if tt.setupLegacyFn != nil {
+				tt.setupLegacyFn(m, tt.input)
+			}
+			if tt.setupStorageFn != nil {
+				tt.setupStorageFn(m)
+			}
+
+			dw := NewDualWriter(Mode3, ls, us, p)
+
+			obj, err := dw.Create(context.Background(), tt.input, func(context.Context, runtime.Object) error { return nil }, &metav1.CreateOptions{})
+
+			if tt.wantErr {
+				assert.Error(t, err)
+				return
+			}
+
+			acc, err := meta.Accessor(obj)
+			assert.NoError(t, err)
+			assert.Equal(t, acc.GetResourceVersion(), "1")
+			assert.NotEqual(t, obj, anotherObj)
+		})
+	}
+}
+
+func TestMode3_Get(t *testing.T) {
+	type testCase struct {
+		setupStorageFn func(m *mock.Mock, name string)
+		name           string
+		input          string
+		wantErr        bool
+	}
+	tests :=
+		[]testCase{
+			{
+				name:  "get an object only in unified store",
+				input: "foo",
+				setupStorageFn: func(m *mock.Mock, name string) {
+					m.On("Get", mock.Anything, name, mock.Anything).Return(exampleObj, nil)
+				},
+			},
+			{
+				name:  "error when getting an object in the unified store fails",
+				input: "object-fail",
+				setupStorageFn: func(m *mock.Mock, name string) {
+					m.On("Get", mock.Anything, name, mock.Anything).Return(nil, errors.New("error"))
+				},
+				wantErr: true,
+			},
+		}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			l := (LegacyStorage)(nil)
+			s := (Storage)(nil)
+			m := &mock.Mock{}
+
+			ls := legacyStoreMock{m, l}
+			us := storageMock{m, s}
+
+			if tt.setupStorageFn != nil {
+				tt.setupStorageFn(m, tt.input)
+			}
+
+			p := prometheus.NewRegistry()
+			dw := NewDualWriter(Mode3, ls, us, p)
+
+			obj, err := dw.Get(context.Background(), tt.input, &metav1.GetOptions{})
+
+			if tt.wantErr {
+				assert.Error(t, err)
+				return
+			}
+
+			assert.Equal(t, obj, exampleObj)
+			assert.NotEqual(t, obj, anotherObj)
+		})
+	}
+}
+
+func TestMode3_List(t *testing.T) {
+	type testCase struct {
+		setupStorageFn func(m *mock.Mock, options *metainternalversion.ListOptions)

Leaderboard for this case

medium · 5 of 9 bots fully caught it (56% solve rate) · 81% avg coverage.

#BotResultCaughtCoverageFindingsFalse alarms
1 Claude Code (Opus 5.5) solved 4/4 100.0% 8 0
2 Cubic solved 4/4 100.0% 7 0
3 Drape (Deep Review) solved 4/4 100.0% 9 0
4 Drape (Review) solved 4/4 100.0% 7 0
5 Mjolnir (Sonnet 5.5) solved 4/4 100.0% 7 0
6 cursor partial 3/4 87.5% 4 0
7 Mjolnir (Opus 5.5) partial 3/4 87.5% 8 0
8 greptile partial 3/4 87.5% 5 1
9 qodo miss 0/4 0.0% 0 0

Bot results

Claude Code (Opus 5.5) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 low ✓ exact
d2 low ✓ off by one
d3 medium ✓ exact
d4 high ✓ off by one
valid · impact medium
Before this change, a legacy Create failure was logged (`unable to create object in legacy storage`). Now the call runs in a goroutine that only records a metric, so failures go unlogged. The goroutine also shares `obj` with the request path while the response is being serialized, which can race. The core claim holds. [grounded: diff] [consensus 3/3: valid=3]
valid · matches d2 · impact low
valid · impact low
In Update, the error branch after Storage.Update records the failure with recordLegacyDuration, so storage errors are counted in the legacy metric. [grounded: diff] [consensus 3/3: valid=3]
valid · matches d3 · impact medium
valid · impact low
The DeleteCollection goroutine times the Legacy.DeleteCollection call but records it with recordStorageDuration, so legacy timings and errors land in the storage metric. [grounded: diff] [consensus 3/3: valid=3]
valid · impact medium
The removed code passed `&updateWrapper{upstream: objInfo, updated: obj}` to Legacy.Update, so legacy received the result already applied to storage. The new code passes the raw objInfo, which legacy re-applies to its own copy of the object. The two stores can diverge, and since errors are no longer logged, the divergence is silent. [grounded: diff] [consensus 3/3: valid=3]
valid · matches d4 · impact high
valid · matches d1 · impact low
raw comments (8)
pkg/apiserver/rest/dualwriter_mode3.go:52
The legacy Create now runs in a fire-and-forget goroutine and never logs errors, so a failed write to legacy storage goes unnoticed. The goroutine also passes the same `obj` that Storage.Create just received (Storage may have mutated it, e.g. by setting resourceVersion or uid) while the handler serializes `created`, which may be that same object. That is a data race. Legacy also gets an object carrying unified-store metadata, which can make it reject or mis-store the object. Mode1 deep-copies before handing off to its goroutine; Mode3 does not.
pkg/apiserver/rest/dualwriter_mode3.go:50
Wrong metric on the Create error path: when Storage.Create fails, the code calls recordLegacyDuration instead of recordStorageDuration. Storage failures are reported as legacy failures, which skews both metrics.
pkg/apiserver/rest/dualwriter_mode3.go:121
Wrong metric on the Update error path: when Storage.Update fails, the code calls recordLegacyDuration instead of recordStorageDuration, so storage errors show up in the legacy metric.
pkg/apiserver/rest/dualwriter_mode3.go:101
On a successful Delete, the code passes `name` (the object name) as the `kind` label of recordStorageDuration instead of `options.Kind`. This mislabels the metric and creates unbounded Prometheus label cardinality: one series per deleted object.
pkg/apiserver/rest/dualwriter_mode3.go:165
The DeleteCollection goroutine times the Legacy.DeleteCollection call but records it with recordStorageDuration instead of recordLegacyDuration. Legacy timings and errors are added to the storage metric.
pkg/apiserver/rest/dualwriter_mode3.go:128
Update now passes the client's original `objInfo` straight to Legacy.Update, which re-applies it against the legacy copy of the object. Before this change, the update applied to unified storage was wrapped in `updateWrapper` and forwarded. The two stores can now diverge, and resourceVersion preconditions from the unified store can make the legacy update fail. Because errors are only counted in a metric and never logged, the divergence is silent.
pkg/apiserver/rest/dualwriter_mode3.go:59
The legacy-write goroutines (in Create, Update, Delete and DeleteCollection) derive their context from the request context. The API server cancels that context once the handler returns, so these async legacy writes can be cancelled partway through and never land in legacy storage. Detaching with context.WithoutCancel(ctx) before adding the timeout would avoid this.
pkg/apiserver/rest/dualwriter_mode3.go:95
Delete builds `log` with name, kind and method, but then stores the bare `d.Log` in the context with klog.NewContext(ctx, d.Log). Downstream logs lose those values; the other methods put `log` in the context.
Cubic · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 low ✓ exact
d2 low ✓ off by one
d3 medium ✓ off by one
d4 high ✓ exact
valid · matches d4 · impact high
valid · impact low
The legacy DeleteCollection goroutine records its result under the storage metric. As a result, no legacy duration is recorded, and the storage histogram gets an extra observation that reflects the legacy result. [grounded: tree pkg/apiserver/rest/dualwriter_mode3.go] [consensus 5/5: valid=5]
valid · matches d2 · impact low
valid · impact low
The new Mode3 tests never wait for the legacy-write goroutines and never assert that they ran. Nothing therefore checks that both stores get written, and the goroutines can outlive their subtests. In practice the shared mock is thread-safe and its expectations match, so actual flakiness is unlikely. The real problem is the missing coverage. [grounded: tree pkg/apiserver/rest/dualwriter_mode3_test.go] [consensus 4/5: valid=4, invalid=1]
valid · matches d1 · impact low
valid · matches d3 · impact medium
minor
Both store mocks share one mock.Mock, so the test behaves correctly as written. The finding asks for the failing expectation to move into setupStorageFn for clarity, which is a style point rather than a behaviour bug. [consensus 5/5: minor=5]
raw comments (8)
pkg/apiserver/rest/dualwriter_mode3.go:51
<!-- cubic:v=530dff79-1808-4a67-96d5-0d42f47057d3 --> <!-- cubic:review-run=044219d8-0368-5b69-82b2-04d7266da651 --> <!-- metadata:{"confidence":6} --> P1: These background goroutines inherit the request context, which net/http cancels once the handler returns, so the legacy write/delete can be aborted before it completes and mode 3 silently stops replicating to legacy (the error is only fed to `recordLegacyDuration`, never logged). Derive the goroutine context with `context.WithoutCancel(ctx)` before applying the timeout, and log the legacy error when it is non-nil. <details> <summary>Prompt for AI agents</summary> ```text Check if this issue is valid — if so, understand the root cause and fix it. At pkg/apiserver/rest/dualwriter_mode3.go, line 51: <comment>These background goroutines inherit the request context, which net/http cancels once the handler returns, so the legacy write/delete can be aborted before it completes and mode 3 silently stops replicating to legacy (the error is only fed to `recordLegacyDuration`, never logged). Derive the goroutine context with `context.WithoutCancel(ctx)` before applying the timeout, and log the legacy error when it is non-nil.</comment> <file context> @@ -21,114 +22,151 @@ type DualWriterMode3 struct { - } - return created, nil + go func() { + ctx, cancel := context.WithTimeoutCause(ctx, time.Second*10, errors.New("legacy create timeout")) + defer cancel() + </file context> ``` </details>
pkg/apiserver/rest/dualwriter_mode3.go:166
<!-- cubic:v=ffc68e9a-8911-487f-8a6c-63e341de2bc2 --> <!-- cubic:review-run=044219d8-0368-5b69-82b2-04d7266da651 --> <!-- metadata:{"confidence":9} --> P2: Inside the legacy `DeleteCollection` goroutine, `recordStorageDuration` is called instead of `recordLegacyDuration`. The legacy delete duration is never recorded, while the storage histogram receives an extra observation that can carry `is_error="true"` for a legacy failure. <details> <summary>Prompt for AI agents</summary> ```text Check if this issue is valid — if so, understand the root cause and fix it. At pkg/apiserver/rest/dualwriter_mode3.go, line 166: <comment>Inside the legacy `DeleteCollection` goroutine, `recordStorageDuration` is called instead of `recordLegacyDuration`. The legacy delete duration is never recorded, while the storage histogram receives an extra observation that can carry `is_error="true"` for a legacy failure.</comment> <file context> @@ -21,114 +22,151 @@ type DualWriterMode3 struct { + ctx, cancel := context.WithTimeoutCause(ctx, time.Second*10, errors.New("legacy deletecollection timeout")) + defer cancel() + _, err := d.Legacy.DeleteCollection(ctx, deleteValidation, options, listOptions) + d.recordStorageDuration(err != nil, mode3Str, options.Kind, method, startLegacy) + }() </file context> ``` </details>
pkg/apiserver/rest/dualwriter_mode3.go:45
<!-- cubic:v=e739580d-3b85-48a0-96b1-2de91aef235f --> <!-- cubic:review-run=044219d8-0368-5b69-82b2-04d7266da651 --> <!-- metadata:{"confidence":9} --> P2: On the storage-failure path in both `Create` (line 45) and `Update` (line 129), the code calls `recordLegacyDuration` instead of `recordStorageDuration`. Storage failures are therefore counted in the legacy histogram with a duration measured from `startStorage`, and the storage histogram never records the error. <details> <summary>Prompt for AI agents</summary> ```text Check if this issue is valid — if so, understand the root cause and fix it. At pkg/apiserver/rest/dualwriter_mode3.go, line 45: <comment>On the storage-failure path in both `Create` (line 45) and `Update` (line 129), the code calls `recordLegacyDuration` instead of `recordStorageDuration`. Storage failures are therefore counted in the legacy histogram with a duration measured from `startStorage`, and the storage histogram never records the error.</comment> <file context> @@ -21,114 +22,151 @@ type DualWriterMode3 struct { created, err := d.Storage.Create(ctx, obj, createValidation, options) if err != nil { log.Error(err, "unable to create object in storage") + d.recordLegacyDuration(true, mode3Str, options.Kind, method, startStorage) return created, err } </file context> ``` </details>
pkg/apiserver/rest/dualwriter_mode3_test.go:65
<!-- cubic:v=582a0e8f-1261-40d0-b854-b9321128081b --> <!-- cubic:review-run=044219d8-0368-5b69-82b2-04d7266da651 --> <!-- metadata:{"confidence":6} --> P2: Mode3's core contract is "write to both LegacyStorage and Storage", but Create/Delete/Update/DeleteCollection trigger the Legacy write in a fire-and-forget goroutine and these tests only assert the returned Storage object. The Legacy write is never asserted (no AssertExpectations/AssertNumberOfCalls) and never waited on, so the goroutine can outlive the subtest and keep mutating the shared mock `m` and package-level registry `p` after the test returns, making the tests non-deterministic and giving false confidence that both stores are written. Add synchronization (e.g., a channel/CountDownLatch the goroutine signals, or assert via a small wait) and assert the Legacy call actually happened. <details> <summary>Prompt for AI agents</summary> ```text Check if this issue is valid — if so, understand the root cause and fix it. At pkg/apiserver/rest/dualwriter_mode3_test.go, line 65: <comment>Mode3's core contract is "write to both LegacyStorage and Storage", but Create/Delete/Update/DeleteCollection trigger the Legacy write in a fire-and-forget goroutine and these tests only assert the returned Storage object. The Legacy write is never asserted (no AssertExpectations/AssertNumberOfCalls) and never waited on, so the goroutine can outlive the subtest and keep mutating the shared mock `m` and package-level registry `p` after the test returns, making the tests non-deterministic and giving false confidence that both stores are written. Add synchronization (e.g., a channel/CountDownLatch the goroutine signals, or assert via a small wait) and assert the Legacy call actually happened.</comment> <file context> @@ -1,72 +1,356 @@ + + dw := NewDualWriter(Mode3, ls, us, p) + + obj, err := dw.Create(context.Background(), tt.input, func(context.Context, runtime.Object) error { return nil }, &metav1.CreateOptions{}) + + if tt.wantErr { </file context> ``` </details>
pkg/apiserver/rest/dualwriter_mode3.go:97
<!-- cubic:v=34e0db64-e08e-42db-b46f-a0fcf4817614 --> <!-- cubic:review-run=044219d8-0368-5b69-82b2-04d7266da651 --> <!-- metadata:{"confidence":8} --> P3: The context is built with `d.Log` instead of the scoped `log`, so `d.Storage.Delete`, `d.Legacy.Delete` and anything downstream that reads the logger from the context lose the `name`, `kind` and `method` fields. Use `log`, as the other methods in this file do. <details> <summary>Prompt for AI agents</summary> ```text Check if this issue is valid — if so, understand the root cause and fix it. At pkg/apiserver/rest/dualwriter_mode3.go, line 97: <comment>The context is built with `d.Log` instead of the scoped `log`, so `d.Storage.Delete`, `d.Legacy.Delete` and anything downstream that reads the logger from the context lose the `name`, `kind` and `method` fields. Use `log`, as the other methods in this file do.</comment> <file context> @@ -21,114 +22,151 @@ type DualWriterMode3 struct { +func (d *DualWriterMode3) Delete(ctx context.Context, name string, deleteValidation rest.ValidateObjectFunc, options *metav1.DeleteOptions) (runtime.Object, bool, error) { + var method = "delete" + log := d.Log.WithValues("name", name, "kind", options.Kind, "method", method) + ctx = klog.NewContext(ctx, d.Log) + + startStorage := time.Now() </file context> ``` </details> ```suggestion ctx = klog.NewContext(ctx, log) ```
pkg/apiserver/rest/dualwriter_mode3.go:106
<!-- cubic:v=9b409e0d-7f7d-4694-9c62-088074e5a7d0 --> <!-- cubic:review-run=044219d8-0368-5b69-82b2-04d7266da651 --> <!-- metadata:{"confidence":7} --> P3: The delete success path passes `name` where `recordStorageDuration` expects the kind, so `dual_writer_storage_duration_seconds{kind=...}` records the object name for successful deletes while the error path on line 103 records `options.Kind`. Pass `options.Kind` here for consistent labels. <details> <summary>Prompt for AI agents</summary> ```text Check if this issue is valid — if so, understand the root cause and fix it. At pkg/apiserver/rest/dualwriter_mode3.go, line 106: <comment>The delete success path passes `name` where `recordStorageDuration` expects the kind, so `dual_writer_storage_duration_seconds{kind=...}` records the object name for successful deletes while the error path on line 103 records `options.Kind`. Pass `options.Kind` here for consistent labels.</comment> <file context> @@ -21,114 +22,151 @@ type DualWriterMode3 struct { + d.recordStorageDuration(true, mode3Str, options.Kind, method, startStorage) + return res, async, err } + d.recordStorageDuration(false, mode3Str, name, method, startStorage) - return deleted, async, err </file context> ``` </details> ```suggestion d.recordStorageDuration(false, mode3Str, options.Kind, method, startStorage) ```
pkg/apiserver/rest/dualwriter_mode3_test.go:30
<!-- cubic:v=596010c3-1005-453e-96eb-b655412b3985 --> <!-- cubic:review-run=044219d8-0368-5b69-82b2-04d7266da651 --> <!-- metadata:{"confidence":7} --> P3: The `error when creating object in the unified store fails` case configures the failure in `setupLegacyFn` but has no `setupStorageFn`. Since both stores share the same `mock.Mock`, the Storage.Create call is the one that matches that expectation, so it works only by accident of the shared mock. It is misleading (a reader expects the legacy call to fail) and fragile if the mocks are ever split. Put the failing expectation in `setupStorageFn` for this case. <details> <summary>Prompt for AI agents</summary> ```text Check if this issue is valid — if so, understand the root cause and fix it. At pkg/apiserver/rest/dualwriter_mode3_test.go, line 30: <comment>The `error when creating object in the unified store fails` case configures the failure in `setupLegacyFn` but has no `setupStorageFn`. Since both stores share the same `mock.Mock`, the Storage.Create call is the one that matches that expectation, so it works only by accident of the shared mock. It is misleading (a reader expects the legacy call to fail) and fragile if the mocks are ever split. Put the failing expectation in `setupStorageFn` for this case.</comment> <file context> @@ -1,72 +1,356 @@ + { + name: "creating an object only in the unified store", + input: exampleObj, + setupLegacyFn: func(m *mock.Mock, input runtime.Object) { + m.On("Create", mock.Anything, input, mock.Anything, mock.Anything).Return(exampleObj, nil) + }, </file context> ``` </details>
PR-level
<!-- cubic:review-summary:start --> **7 issues found** across 5 files <!-- cubic:review-summary:end --> <details> <summary>Prompt for AI agents (unresolved issues)</summary> ```text Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately. <file name="pkg/apiserver/rest/dualwriter_mode3.go"> <violation number="1" location="pkg/apiserver/rest/dualwriter_mode3.go:45"> P2: On the storage-failure path in both `Create` (line 45) and `Update` (line 129), the code calls `recordLegacyDuration` instead of `recordStorageDuration`. Storage failures are therefore counted in the legacy histogram with a duration measured from `startStorage`, and the storage histogram never records the error.</violation> <violation number="2" location="pkg/apiserver/rest/dualwriter_mode3.go:51"> P1: These background goroutines inherit the request context, which net/http cancels once the handler returns, so the legacy write/delete can be aborted before it completes and mode 3 silently stops replicating to legacy (the error is only fed to `recordLegacyDuration`, never logged). Derive the goroutine context with `context.WithoutCancel(ctx)` before applying the timeout, and log the legacy error when it is non-nil.</violation> <violation number="3" location="pkg/apiserver/rest/dualwriter_mode3.go:97"> P3: The context is built with `d.Log` instead of the scoped `log`, so `d.Storage.Delete`, `d.Legacy.Delete` and anything downstream that reads the logger from the context lose the `name`, `kind` and `method` fields. Use `log`, as the other methods in this file do.</violation> <violation number="4" location="pkg/apiserver/rest/dualwriter_mode3.go:106"> P3: The delete success path passes `name` where `recordStorageDuration` expects the kind, so `dual_writer_storage_duration_seconds{kind=...}` records the object name for successful deletes while the error path on line 103 records `options.Kind`. Pass `options.Kind` here for consistent labels.</violation> <violation number="5" location="pkg/apiserver/rest/dualwriter_mode3.go:166"> P2: Inside the legacy `DeleteCollection` goroutine, `recordStorageDuration` is called instead of `recordLegacyDuration`. The legacy delete duration is never recorded, while the storage histogram receives an extra observation that can carry `is_error="true"` for a legacy failure.</violation> </file> <file name="pkg/apiserver/rest/dualwriter_mode3_test.go"> <violation number="1" location="pkg/apiserver/rest/dualwriter_mode3_test.go:30"> P3: The `error when creating object in the unified store fails` case configures the failure in `setupLegacyFn` but has no `setupStorageFn`. Since both stores share the same `mock.Mock`, the Storage.Create call is the one that matches that expectation, so it works only by accident of the shared mock. It is misleading (a reader expects the legacy call to fail) and fragile if the mocks are ever split. Put the failing expectation in `setupStorageFn` for this case.</violation> <violation number="2" location="pkg/apiserver/rest/dualwriter_mode3_test.go:65"> P2: Mode3's core contract is "write to both LegacyStorage and Storage", but Create/Delete/Update/DeleteCollection trigger the Legacy write in a fire-and-forget goroutine and these tests only assert the returned Storage object. The Legacy write is never asserted (no AssertExpectations/AssertNumberOfCalls) and never waited on, so the goroutine can outlive the subtest and keep mutating the shared mock `m` and package-level registry `p` after the test returns, making the tests non-deterministic and giving false confidence that both stores are written. Add synchronization (e.g., a channel/CountDownLatch the goroutine signals, or assert via a small wait) and assert the Legacy call actually happened.</violation> </file> ``` </details> <sub>Reply with feedback, questions, or to request a fix.<br /><br />[Re-trigger cubic](https://www.cubic.dev/action/re-review/pr/drape-io/crbench-cubic-2026-09/43/ai_pr_review_github_delivery_699267ad25c301c2306249fe00af474a1994b73919e2c408bd6a3e36f4d42b74?returnTo=https%3A%2F%2Fgithub.com%2Fdrape-io%2Fcrbench-cubic-2026-09%2Fpull%2F43)</sub> <!-- cubic:review-post:ai_pr_review_github_delivery_699267ad25c301c2306249fe00af474a1994b73919e2c408bd6a3e36f4d42b74:d82b79aa30ba698d1eeebc878961b567502ffc6d:044219d8-0368-5b69-82b2-04d7266da651 -->
cursor · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 low ✗ —
d2 low ✓ off by one
d3 medium ✓ exact
d4 high ✓ exact
valid · matches d2 · impact low
valid · matches d3 · impact medium
valid · impact medium
The Create half is not new: the old code also passed the original `obj` to Legacy.Create. The Update half is new. The removed code wrapped the storage result in `updateWrapper{upstream: objInfo, updated: obj}` before calling Legacy.Update. The new code passes the raw `objInfo`, so legacy recomputes the update against its own old object rather than writing what unified storage stored, and the two stores can drift. [grounded: diff] [consensus 5/5: valid=5]
valid · matches d4 · impact high
raw comments (5)
pkg/apiserver/rest/dualwriter_mode3.go:45
### Wrong store recorded in write metrics **Medium Severity** <!-- DESCRIPTION START --> Storage `Create` and `Update` failures call `recordLegacyDuration`, and the async `Legacy.DeleteCollection` path calls `recordStorageDuration`. Mode 3 dashboards then attribute the wrong backend’s latency and errors, which can hide unified-store failures during migration. <!-- DESCRIPTION END --> <!-- BUGBOT_BUG_ID: b59fd5a7-4764-466d-ab2c-3b5c0d030ab7 --> <!-- LOCATIONS START pkg/apiserver/rest/dualwriter_mode3.go#L44-L45 pkg/apiserver/rest/dualwriter_mode3.go#L128-L129 pkg/apiserver/rest/dualwriter_mode3.go#L165-L166 LOCATIONS END --> <details> <summary>Additional Locations (2)</summary> - [`pkg/apiserver/rest/dualwriter_mode3.go#L128-L129`](https://github.com/drape-io/crbench-cursor-2026-09/blob/1b935a7278999bce0d413f8b98dc42bfffe8b5b1/pkg/apiserver/rest/dualwriter_mode3.go#L128-L129) - [`pkg/apiserver/rest/dualwriter_mode3.go#L165-L166`](https://github.com/drape-io/crbench-cursor-2026-09/blob/1b935a7278999bce0d413f8b98dc42bfffe8b5b1/pkg/apiserver/rest/dualwriter_mode3.go#L165-L166) </details> <div><a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9DVVJTT1IiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjkyMzE0OTAxLTY4NzYtNDkwNy1hMGFjLWU1YTIxNTZjNmNlOSIsImVuY3J5cHRpb25LZXkiOiJJMHdYYV9hREl0bm9SNUQyNUotUXpDQzFIaW82SlU0OHRXRGJDQ1lUTm93IiwiYnJhbmNoIjoidDItZ3JhZmFuYS05MDA0NSIsInJlcG9Pd25lciI6ImRyYXBlLWlvIiwicmVwb05hbWUiOiJjcmJlbmNoLWN1cnNvci0yMDI2LTA5IiwicHJvdmlkZXIiOiJnaXRodWIifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/agents?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9XRUIiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjkyMzE0OTAxLTY4NzYtNDkwNy1hMGFjLWU1YTIxNTZjNmNlOSIsImVuY3J5cHRpb25LZXkiOiJJMHdYYV9hREl0bm9SNUQyNUotUXpDQzFIaW82SlU0OHRXRGJDQ1lUTm93IiwiYnJhbmNoIjoidDItZ3JhZmFuYS05MDA0NSIsInJlcG9Pd25lciI6ImRyYXBlLWlvIiwicmVwb05hbWUiOiJjcmJlbmNoLWN1cnNvci0yMDI2LTA5IiwicHJOdW1iZXIiOjQ0LCJjb21taXRTaGEiOiIxYjkzNWE3Mjc4OTk5YmNlMGQ0MTNmOGI5OGRjNDJiZmZmZThiNWIxIiwicHJvdmlkZXIiOiJnaXRodWIifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-web-light.png"><img alt="Fix in Web" width="99" height="28" src="https://cursor.com/assets/images/fix-in-web-dark.png"></picture></a></div> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 1b935a7278999bce0d413f8b98dc42bfffe8b5b1. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
pkg/apiserver/rest/dualwriter_mode3.go:106
### Delete metrics use object name **Medium Severity** <!-- DESCRIPTION START --> Successful `Delete` records `recordStorageDuration` with the object `name` in the `kind` label, while the error path uses `options.Kind`. That splits one histogram into a series per deleted object and can explode Prometheus cardinality. <!-- DESCRIPTION END --> <!-- BUGBOT_BUG_ID: 16ba9939-d490-4790-8471-54ff24fce3cd --> <!-- LOCATIONS START pkg/apiserver/rest/dualwriter_mode3.go#L105-L106 LOCATIONS END --> <div><a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9DVVJTT1IiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OmVjMmIwMTMzLTVmZWYtNDVkMC04NDBjLTc0YWRjZTk1OWQ4MiIsImVuY3J5cHRpb25LZXkiOiJmdkhqYlpqT0I2ZjVwdlZRUkk3WUlDQjFMNFd3NXNPcE1MX2VLV0JTc1BZIiwiYnJhbmNoIjoidDItZ3JhZmFuYS05MDA0NSIsInJlcG9Pd25lciI6ImRyYXBlLWlvIiwicmVwb05hbWUiOiJjcmJlbmNoLWN1cnNvci0yMDI2LTA5IiwicHJvdmlkZXIiOiJnaXRodWIifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/agents?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9XRUIiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OmVjMmIwMTMzLTVmZWYtNDVkMC04NDBjLTc0YWRjZTk1OWQ4MiIsImVuY3J5cHRpb25LZXkiOiJmdkhqYlpqT0I2ZjVwdlZRUkk3WUlDQjFMNFd3NXNPcE1MX2VLV0JTc1BZIiwiYnJhbmNoIjoidDItZ3JhZmFuYS05MDA0NSIsInJlcG9Pd25lciI6ImRyYXBlLWlvIiwicmVwb05hbWUiOiJjcmJlbmNoLWN1cnNvci0yMDI2LTA5IiwicHJOdW1iZXIiOjQ0LCJjb21taXRTaGEiOiIxYjkzNWE3Mjc4OTk5YmNlMGQ0MTNmOGI5OGRjNDJiZmZmZThiNWIxIiwicHJvdmlkZXIiOiJnaXRodWIifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-web-light.png"><img alt="Fix in Web" width="99" height="28" src="https://cursor.com/assets/images/fix-in-web-dark.png"></picture></a></div> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 1b935a7278999bce0d413f8b98dc42bfffe8b5b1. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
pkg/apiserver/rest/dualwriter_mode3.go:57
### Legacy writes omit storage results **High Severity** <!-- DESCRIPTION START --> Async legacy `Create` reuses the original `obj` instead of a copy of `created`, and `Update` passes the same `objInfo` into `Legacy.Update` instead of wrapping the storage result. Legacy then persists a different object than unified storage, so the stores drift. <!-- DESCRIPTION END --> <!-- BUGBOT_BUG_ID: fb91047e-4b3a-4093-b351-7bf417975b3a --> <!-- LOCATIONS START pkg/apiserver/rest/dualwriter_mode3.go#L49-L57 pkg/apiserver/rest/dualwriter_mode3.go#L133-L141 LOCATIONS END --> <details> <summary>Additional Locations (1)</summary> - [`pkg/apiserver/rest/dualwriter_mode3.go#L133-L141`](https://github.com/drape-io/crbench-cursor-2026-09/blob/1b935a7278999bce0d413f8b98dc42bfffe8b5b1/pkg/apiserver/rest/dualwriter_mode3.go#L133-L141) </details> <div><a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9DVVJTT1IiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjViMmVjN2M4LWZhZDUtNGQ5Yi1hZmM1LWZhYTIwYmQ4MDU3ZiIsImVuY3J5cHRpb25LZXkiOiJuOEJLaUhYeEFPQklIdE5fM1Uxa3hVOFpNYkJiaTRqcWUtZGo1U21TNG9RIiwiYnJhbmNoIjoidDItZ3JhZmFuYS05MDA0NSIsInJlcG9Pd25lciI6ImRyYXBlLWlvIiwicmVwb05hbWUiOiJjcmJlbmNoLWN1cnNvci0yMDI2LTA5IiwicHJvdmlkZXIiOiJnaXRodWIifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/agents?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9XRUIiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjViMmVjN2M4LWZhZDUtNGQ5Yi1hZmM1LWZhYTIwYmQ4MDU3ZiIsImVuY3J5cHRpb25LZXkiOiJuOEJLaUhYeEFPQklIdE5fM1Uxa3hVOFpNYkJiaTRqcWUtZGo1U21TNG9RIiwiYnJhbmNoIjoidDItZ3JhZmFuYS05MDA0NSIsInJlcG9Pd25lciI6ImRyYXBlLWlvIiwicmVwb05hbWUiOiJjcmJlbmNoLWN1cnNvci0yMDI2LTA5IiwicHJOdW1iZXIiOjQ0LCJjb21taXRTaGEiOiIxYjkzNWE3Mjc4OTk5YmNlMGQ0MTNmOGI5OGRjNDJiZmZmZThiNWIxIiwicHJvdmlkZXIiOiJnaXRodWIifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-web-light.png"><img alt="Fix in Web" width="99" height="28" src="https://cursor.com/assets/images/fix-in-web-dark.png"></picture></a></div> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 1b935a7278999bce0d413f8b98dc42bfffe8b5b1. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
pkg/apiserver/rest/dualwriter_mode3.go:52
### Request cancel aborts legacy writes **High Severity** <!-- DESCRIPTION START --> Background legacy `Create`, `Update`, `Delete`, and `DeleteCollection` derive their 10s timeout from the incoming request `ctx`. When the HTTP handler returns, that context is canceled, so the timeout never applies and the legacy write is often aborted after the client already received success. <!-- DESCRIPTION END --> <!-- BUGBOT_BUG_ID: b0b29424-3246-4c60-9834-fea11862eea3 --> <!-- LOCATIONS START pkg/apiserver/rest/dualwriter_mode3.go#L49-L52 pkg/apiserver/rest/dualwriter_mode3.go#L107-L111 pkg/apiserver/rest/dualwriter_mode3.go#L133-L135 LOCATIONS END --> <details> <summary>Additional Locations (2)</summary> - [`pkg/apiserver/rest/dualwriter_mode3.go#L107-L111`](https://github.com/drape-io/crbench-cursor-2026-09/blob/1b935a7278999bce0d413f8b98dc42bfffe8b5b1/pkg/apiserver/rest/dualwriter_mode3.go#L107-L111) - [`pkg/apiserver/rest/dualwriter_mode3.go#L133-L135`](https://github.com/drape-io/crbench-cursor-2026-09/blob/1b935a7278999bce0d413f8b98dc42bfffe8b5b1/pkg/apiserver/rest/dualwriter_mode3.go#L133-L135) </details> <div><a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9DVVJTT1IiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjFiMTBmNjdhLTRhM2MtNGRmNi1hNmE2LTI4NjJiNGJjMjU2YSIsImVuY3J5cHRpb25LZXkiOiJmOHBKTVI5cW1VS1RTVWZLTG0zVTV1RzdEOXFkTUNOVGJ0cHB0X2JOYThNIiwiYnJhbmNoIjoidDItZ3JhZmFuYS05MDA0NSIsInJlcG9Pd25lciI6ImRyYXBlLWlvIiwicmVwb05hbWUiOiJjcmJlbmNoLWN1cnNvci0yMDI2LTA5IiwicHJvdmlkZXIiOiJnaXRodWIifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/agents?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9JTl9XRUIiLCJkYXRhIjp7InJlZGlzS2V5IjoiYnVnYm90OjFiMTBmNjdhLTRhM2MtNGRmNi1hNmE2LTI4NjJiNGJjMjU2YSIsImVuY3J5cHRpb25LZXkiOiJmOHBKTVI5cW1VS1RTVWZLTG0zVTV1RzdEOXFkTUNOVGJ0cHB0X2JOYThNIiwiYnJhbmNoIjoidDItZ3JhZmFuYS05MDA0NSIsInJlcG9Pd25lciI6ImRyYXBlLWlvIiwicmVwb05hbWUiOiJjcmJlbmNoLWN1cnNvci0yMDI2LTA5IiwicHJOdW1iZXIiOjQ0LCJjb21taXRTaGEiOiIxYjkzNWE3Mjc4OTk5YmNlMGQ0MTNmOGI5OGRjNDJiZmZmZThiNWIxIiwicHJvdmlkZXIiOiJnaXRodWIifX0" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-web-light.png"><img alt="Fix in Web" width="99" height="28" src="https://cursor.com/assets/images/fix-in-web-dark.png"></picture></a></div> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 1b935a7278999bce0d413f8b98dc42bfffe8b5b1. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
PR-level
<!-- BUGBOT_REVIEW --> Cursor Bugbot has reviewed your changes using high effort and found 4 potential issues. <!-- BUGBOT_FIX_ALL --> <a href="https://cursor.com/open?link=eyJ2ZXJzaW9uIjoxLCJ0eXBlIjoiQlVHQk9UX0ZJWF9BTExfSU5fQ1VSU09SIiwiZGF0YSI6eyJyZWRpc0tleSI6ImJ1Z2JvdC1tdWx0aTpiODAzNDhhZS1lMWIzLTQ2M2ItYjhhZi1kNTk0YmJlNmNlYTgiLCJlbmNyeXB0aW9uS2V5IjoidG5KN2NZV2Z4c1ptdldxdzVsVE9RVXdaT282amlyTFFDbW5tdEpCTGQwbyIsImJyYW5jaCI6InQyLWdyYWZhbmEtOTAwNDUiLCJyZXBvT3duZXIiOiJkcmFwZS1pbyIsInJlcG9OYW1lIjoiY3JiZW5jaC1jdXJzb3ItMjAyNi0wOSIsInByb3ZpZGVyIjoiZ2l0aHViIn19" target="_blank" rel="noopener noreferrer"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/fix-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/fix-in-cursor-light.png"><img alt="Fix All in Cursor" width="115" height="28" src="https://cursor.com/assets/images/fix-in-cursor-dark.png"></picture></a> <!-- /BUGBOT_FIX_ALL --> <!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_BEGIN --> <sup>❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the [Cursor dashboard](https://www.cursor.com/dashboard/bugbot).</sup> <!-- BUGBOT_AUTOFIX_REVIEW_FOOTNOTE_END --> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 1b935a7278999bce0d413f8b98dc42bfffe8b5b1. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
Drape (Deep Review) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 low ✓ exact
d2 low ✓ exact
d3 medium ✓ off by one
d4 high ✓ off by one
valid · matches d3 · impact medium
valid · impact low
In the DeleteCollection goroutine, the result of d.Legacy.DeleteCollection is recorded with recordStorageDuration, so it goes into the storage histogram instead of the legacy one. [grounded: diff] [consensus 3/3: valid=3]
valid · impact medium
The old code passed the legacy store an updateWrapper holding the object storage returned. The new code passes the caller's raw objInfo, so legacy re-applies the update to its own object and can store different content. [grounded: diff] [consensus 3/3: valid=3]
valid · matches d1 · impact low
valid · impact medium
The old code ignored apierrors.IsNotFound from storage and still deleted from legacy. The new code returns early on any storage error, so the legacy delete never runs for objects that only exist in legacy. [grounded: diff] [consensus 3/3: valid=3]
valid · matches d2 · impact low
valid · impact low
In Update, the error path after d.Storage.Update fails calls recordLegacyDuration, so storage failures are reported in the legacy histogram. [grounded: diff] [consensus 3/3: valid=3]
valid · matches d4 · impact high
valid · impact low
The old code logged errors from the legacy writes. The new background goroutines only record a metric and drop the error, so the reason for a legacy failure is lost. [grounded: diff] [consensus 3/3: valid=3]
raw comments (9)
pkg/apiserver/rest/dualwriter_mode3.go:104
**Delete metric labeled with object name, not kind** On successful storage delete, `recordStorageDuration` is passed `name` where every other call passes `options.Kind`, so each deleted object creates a new metric label value. Pass `options.Kind` instead of `name`. ``` pkg/apiserver/rest/dualwriter_mode3.go:104 return res, async, err pkg/apiserver/rest/dualwriter_mode3.go:102 log.Error(err, "unable to delete object in storage") ```
pkg/apiserver/rest/dualwriter_mode3.go:167
**Legacy delete-collection recorded as storage metric** The background legacy `DeleteCollection` records its duration via `recordStorageDuration`, so legacy timing/errors are reported as unified storage. Use `recordLegacyDuration` in the goroutine. ``` pkg/apiserver/rest/dualwriter_mode3.go:166 d.recordStorageDuration(err != nil, mode3Str, options.Kind, method, startLegacy) pkg/apiserver/rest/dualwriter_mode3.go:167 }() pkg/apiserver/rest/metrics.go:54 func (m *dualWriterMetrics) recordLegacyDuration(isError bool, mode string, name string, method string, startFrom time.Time) { pkg/apiserver/rest/metrics.go:55 duration := time.Since(startFrom).Seconds() pkg/apiserver/rest/metrics.go:56 m.legacy.WithLabelValues(strconv.FormatBool(isError), mode, name, method).Observe(duration) pkg/apiserver/rest/metrics.go:57 } pkg/apiserver/rest/metrics.go:59 func (m *dualWriterMetrics) recordStorageDuration(isError bool, mode string, name string, method string, startFrom time.Time) { pkg/apiserver/rest/metrics.go:60 duration := time.Since(startFrom).Seconds() pkg/apiserver/rest/metrics.go:61 m.storage.WithLabelValues(strconv.FormatBool(isError), mode, name, method).Observe(duration) pkg/apiserver/rest/metrics.go:62 } ```
pkg/apiserver/rest/dualwriter_mode3.go:140
**Legacy update can write different content than storage** The legacy Update now gets the caller's raw objInfo, so the patch is re-applied to the legacy object. When legacy and storage already differ, legacy saves a different result from what storage stored. Pass an updateWrapper holding the object storage returned, as before. ``` pkg/apiserver/rest/dualwriter_mode3.go:134 go func() { pkg/apiserver/rest/dualwriter_mode3.go:135 ctx, cancel := context.WithTimeoutCause(ctx, time.Second*10, errors.New("legacy update timeout")) pkg/apiserver/rest/dualwriter_mode3.go:137 startLegacy := time.Now() pkg/apiserver/rest/dualwriter_mode3.go:138 defer cancel() pkg/apiserver/rest/dualwriter_mode1.go:238 updated, err := objInfo.UpdatedObject(ctx, resCopy) pkg/apiserver/rest/dualwriter_mode1.go:239 if err != nil { pkg/apiserver/rest/dualwriter_mode1.go:240 log.WithValues("object", updated).Error(err, "could not update or create object") pkg/apiserver/rest/dualwriter_mode1.go:241 cancel() pkg/apiserver/rest/dualwriter_mode1.go:242 } pkg/apiserver/rest/dualwriter_mode1.go:244 // if the object is found, create a new updateWrapper with the object found ```
pkg/apiserver/rest/dualwriter_mode3.go:95
**Delete context logger loses name/kind fields** `Delete` puts `d.Log` into the context instead of the `log` built with name/kind/method, so downstream logs lack these values (other methods use `log`). Use `klog.NewContext(ctx, log)`. ``` pkg/apiserver/rest/dualwriter_mode3.go:95 var method = "delete" pkg/apiserver/rest/dualwriter_mode3.go:94 func (d *DualWriterMode3) Delete(ctx context.Context, name string, deleteValidation rest.ValidateObjectFunc, options *metav1.DeleteOptions) (runtime.Object, bool, error) { ```
pkg/apiserver/rest/dualwriter_mode3.go:97
**Legacy-only objects can no longer be deleted** Previously a NotFound from unified storage was ignored and the legacy delete still ran; now any storage error returns early, so objects that only exist in legacy (not yet migrated) are never deleted. Keep tolerating `apierrors.IsNotFound` from storage and still delete from legacy. ``` pkg/apiserver/rest/dualwriter_mode3.go:97 ctx = klog.NewContext(ctx, d.Log) pkg/apiserver/rest/dualwriter_mode3.go:99 startStorage := time.Now() pkg/apiserver/rest/dualwriter_mode3.go:100 res, async, err := d.Storage.Delete(ctx, name, deleteValidation, options) pkg/apiserver/rest/dualwriter_mode3.go:101 if err != nil { pkg/apiserver/rest/dualwriter_mode3.go:58 (base) if err != nil { pkg/apiserver/rest/dualwriter_mode3.go:59 (base) if !apierrors.IsNotFound(err) { pkg/apiserver/rest/dualwriter_mode3.go:60 (base) log.Error(err, "could not delete from unified store") pkg/apiserver/rest/dualwriter_mode3.go:61 (base) return deleted, async, err pkg/apiserver/rest/dualwriter_mode3.go:62 (base) } pkg/apiserver/rest/dualwriter_mode3.go:63 (base) } ```
pkg/apiserver/rest/dualwriter_mode3.go:47
**Failed storage create counted as legacy metric** When the unified-storage create fails, the timing is recorded with `recordLegacyDuration`, so legacy error metrics go up for a storage failure and storage errors are never counted. Call `recordStorageDuration` here. ``` pkg/apiserver/rest/dualwriter_mode3.go:44 log.Error(err, "unable to create object in storage") pkg/apiserver/rest/dualwriter_mode3.go:45 d.recordLegacyDuration(true, mode3Str, options.Kind, method, startStorage) pkg/apiserver/rest/dualwriter_mode3.go:46 return created, err pkg/apiserver/rest/dualwriter_mode3.go:47 } pkg/apiserver/rest/metrics.go:54 func (m *dualWriterMetrics) recordLegacyDuration(isError bool, mode string, name string, method string, startFrom time.Time) { pkg/apiserver/rest/metrics.go:55 duration := time.Since(startFrom).Seconds() pkg/apiserver/rest/metrics.go:56 m.legacy.WithLabelValues(strconv.FormatBool(isError), mode, name, method).Observe(duration) pkg/apiserver/rest/metrics.go:57 } pkg/apiserver/rest/metrics.go:59 func (m *dualWriterMetrics) recordStorageDuration(isError bool, mode string, name string, method string, startFrom time.Time) { pkg/apiserver/rest/metrics.go:60 duration := time.Since(startFrom).Seconds() ```
pkg/apiserver/rest/dualwriter_mode3.go:125
**Failed storage update counted as legacy metric** When `d.Storage.Update` fails, the error path calls `recordLegacyDuration`. Storage update errors are reported as legacy errors. Use `recordStorageDuration`. ``` pkg/apiserver/rest/dualwriter_mode3.go:121 var method = "update" pkg/apiserver/rest/dualwriter_mode3.go:122 log := d.Log.WithValues("name", name, "kind", options.Kind, "method", method) pkg/apiserver/rest/dualwriter_mode3.go:123 ctx = klog.NewContext(ctx, log) pkg/apiserver/rest/dualwriter_mode3.go:125 startStorage := time.Now() pkg/apiserver/rest/metrics.go:54 func (m *dualWriterMetrics) recordLegacyDuration(isError bool, mode string, name string, method string, startFrom time.Time) { pkg/apiserver/rest/metrics.go:55 duration := time.Since(startFrom).Seconds() pkg/apiserver/rest/metrics.go:56 m.legacy.WithLabelValues(strconv.FormatBool(isError), mode, name, method).Observe(duration) pkg/apiserver/rest/metrics.go:57 } pkg/apiserver/rest/metrics.go:59 func (m *dualWriterMetrics) recordStorageDuration(isError bool, mode string, name string, method string, startFrom time.Time) { pkg/apiserver/rest/metrics.go:60 duration := time.Since(startFrom).Seconds() ```
pkg/apiserver/rest/dualwriter_mode3.go:51
**Legacy create cancelled when request finishes** The background legacy `Create` takes its context from the request, and that context is cancelled once the handler returns. So the legacy write is often cancelled and the two stores drift apart without anyone noticing. Derive the goroutine context from `context.WithoutCancel(ctx)`. The same mistake is at `pkg/apiserver/rest/dualwriter_mode3.go:110`, `pkg/apiserver/rest/dualwriter_mode3.go:135` and `pkg/apiserver/rest/dualwriter_mode3.go:163`. ``` pkg/apiserver/rest/dualwriter_mode3.go:50 go func() { pkg/apiserver/rest/dualwriter_mode3.go:51 ctx, cancel := context.WithTimeoutCause(ctx, time.Second*10, errors.New("legacy create timeout")) pkg/apiserver/rest/dualwriter_mode3.go:52 defer cancel() pkg/apiserver/rest/dualwriter_mode3.go:54 startLegacy := time.Now() pkg/apiserver/rest/dualwriter_mode3.go:59 return created, err ```
pkg/apiserver/rest/dualwriter_mode3.go:56
**Legacy write failures are no longer logged** The background legacy calls only feed a metric and never log their error, though the old code logged legacy failures. The cause of a store mismatch is lost. Log the error with `log.Error` when the legacy call fails. ``` pkg/apiserver/rest/dualwriter_mode3.go:55 _, errObjectSt := d.Legacy.Create(ctx, obj, createValidation, options) pkg/apiserver/rest/dualwriter_mode3.go:56 d.recordLegacyDuration(errObjectSt != nil, mode3Str, options.Kind, method, startLegacy) pkg/apiserver/rest/dualwriter_mode3.go:112 _, _, err := d.Legacy.Delete(ctx, name, deleteValidation, options) pkg/apiserver/rest/dualwriter_mode3.go:113 d.recordLegacyDuration(err != nil, mode3Str, options.Kind, method, startLegacy) ```
Drape (Review) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 low ✓ exact
d2 low ✓ exact
d3 medium ✓ off by one
d4 high ✓ off by one
valid · impact low
In DeleteCollection, the goroutine times the legacy call with startLegacy but records it with recordStorageDuration. Legacy timings therefore go into the storage metrics, and no legacy metric is recorded. [grounded: diff] [consensus 3/3: valid=3]
valid · matches d2 · impact low
valid · matches d1 · impact low
valid · matches d4 · impact high
valid · impact low
The removed code logged legacy failures, for example "unable to create object in legacy storage". The new goroutines only record a boolean in a metric and never log the error. [grounded: diff] [consensus 3/3: valid=3]
valid · impact medium
The old code wrapped the object Storage actually stored in updateWrapper{upstream: objInfo, updated: obj}. The new code passes the raw objInfo, so legacy recomputes the update from its own old object, and the result can differ from what Storage stored. [grounded: diff] [consensus 3/3: valid=3]
valid · impact medium
Legacy writes used to be synchronous. The change moves them into untracked goroutines with no ordering, so concurrent writes to one object can reach legacy out of order, and Destroy does not wait for them. [grounded: diff] [consensus 3/3: valid=3]
raw comments (7)
pkg/apiserver/rest/dualwriter_mode3.go:167
**Legacy delete-collection duration recorded as storage** The goroutine in DeleteCollection calls recordStorageDuration for the legacy call, so legacy timings/errors pollute storage metrics and legacy metrics are missing. Call recordLegacyDuration. ``` pkg/apiserver/rest/dualwriter_mode3.go:167 }() ```
pkg/apiserver/rest/dualwriter_mode3.go:43
**Storage failures recorded as legacy metrics** On Storage.Create and Storage.Update errors, recordLegacyDuration is called instead of recordStorageDuration, so storage failures are attributed to legacy. Use recordStorageDuration(true, ...). ``` pkg/apiserver/rest/dualwriter_mode3.go:43 if err != nil { pkg/apiserver/rest/dualwriter_mode3.go:103 d.recordStorageDuration(true, mode3Str, options.Kind, method, startStorage) ```
pkg/apiserver/rest/dualwriter_mode3.go:90
**Delete uses object name as metric kind label** Delete records the success metric with `name` instead of options.Kind, creating a metric label per object name (high cardinality), and stores d.Log rather than the enriched log in the context. Use options.Kind and klog.NewContext(ctx, log). ``` pkg/apiserver/rest/dualwriter_mode3.go:82 ctx = klog.NewContext(ctx, log) ```
pkg/apiserver/rest/dualwriter_mode3.go:58
**Legacy writes cancelled when request ends** The background legacy Create/Update/Delete/DeleteCollection goroutines derive their timeout context from the request `ctx`. Once the handler returns, that parent context is cancelled, so the legacy write is cancelled or fails and legacy storage silently falls out of sync. Use context.WithoutCancel(ctx) as the parent of the timeout context in each goroutine. ``` pkg/apiserver/rest/dualwriter_mode3.go:55 _, errObjectSt := d.Legacy.Create(ctx, obj, createValidation, options) pkg/apiserver/rest/dualwriter_mode3.go:56 d.recordLegacyDuration(errObjectSt != nil, mode3Str, options.Kind, method, startLegacy) pkg/apiserver/rest/dualwriter_mode3.go:57 }() pkg/apiserver/rest/dualwriter_mode3.go:59 return created, err ```
pkg/apiserver/rest/dualwriter_mode3.go:58
**Legacy write errors are silently dropped** The goroutines only record a metric; errors from `d.Legacy.Create/Update/Delete/DeleteCollection` are never logged, whereas the old code logged them. Log the error with log.Error in each goroutine. ``` pkg/apiserver/rest/dualwriter_mode3.go:55 _, errObjectSt := d.Legacy.Create(ctx, obj, createValidation, options) pkg/apiserver/rest/dualwriter_mode3.go:56 d.recordLegacyDuration(errObjectSt != nil, mode3Str, options.Kind, method, startLegacy) pkg/apiserver/rest/dualwriter_mode3.go:57 }() pkg/apiserver/rest/dualwriter_mode3.go:59 return created, err ```
pkg/apiserver/rest/dualwriter_mode3.go:98
**Legacy update recomputed from original objInfo** Update now passes the original `objInfo` to `Legacy.Update` instead of a wrapper holding the object actually stored in Storage; the previous code used `updateWrapper` with the stored `obj`. Wrap the stored result in updateWrapper{upstream: objInfo, updated: res} for the legacy call. ``` pkg/apiserver/rest/dualwriter_mode3.go:97 ctx = klog.NewContext(ctx, d.Log) pkg/apiserver/rest/dualwriter_mode3.go:99 startStorage := time.Now() pkg/apiserver/rest/dualwriter_mode3.go:100 res, async, err := d.Storage.Delete(ctx, name, deleteValidation, options) pkg/apiserver/rest/dualwriter_mode3.go:101 if err != nil { ```
pkg/apiserver/rest/dualwriter_mode3.go:108
**Unordered async legacy writes can leave stale data** Each mutation spawns an untracked goroutine for its legacy write. Rapid successive writes to one object can reach legacy storage out of order, and `Destroy` doesn't wait for them. Serialize legacy writes per object, or write synchronously. ``` pkg/apiserver/rest/dualwriter_mode3.go:50 go func() { pkg/apiserver/rest/dualwriter_mode3.go:51 ctx, cancel := context.WithTimeoutCause(ctx, time.Second*10, errors.New("legacy create timeout")) pkg/apiserver/rest/dualwriter_mode3.go:52 defer cancel() pkg/apiserver/rest/dualwriter_mode3.go:54 startLegacy := time.Now() ```
greptile · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 low ✗ —
d2 low ✓ off by one
d3 medium ✓ exact
d4 high ✓ exact
valid · matches d4 · impact high
valid · impact medium
The removed code passed legacy storage an updateWrapper holding the object unified storage had accepted. The new code passes the raw objInfo to both stores, so each store applies the patch to its own old object and the results can diverge. [grounded: diff] [consensus 3/3: valid=3]
valid · impact medium
The removed code ignored a NotFound from unified storage (`if !apierrors.IsNotFound(err)`) and still deleted from legacy storage. The new code returns on every error, so a record that exists only in legacy storage is never deleted. [grounded: diff] [consensus 3/3: valid=3]
invalid
The legacy and storage mocks share the same *mock.Mock `m`. The expectation set up for Delete/DeleteCollection therefore also matches the asynchronous legacy call, which receives the same name and options. No unexpected-call failure occurs, and the goroutine never touches `t`. [invalid: contradiction] [consensus 3/3: invalid=3]
valid · matches d2 · impact low
raw comments (6)
pkg/apiserver/rest/dualwriter_mode3.go:51
<a href="#"><img alt="P1" src="https://greptile-static-assets.s3.amazonaws.com/badges/p1.svg?v=9" align="top"></a> **Request cancellation aborts mirroring** If the API request context is canceled when the request finishes, this timeout context is canceled with it rather than giving the asynchronous legacy write ten seconds to finish. Mode 3 can therefore return success for the primary write while silently aborting the legacy mirror. The same pattern occurs in Delete, Update, and DeleteCollection. Use a context detached from request cancellation while retaining the required logging values and bounded timeout.
pkg/apiserver/rest/dualwriter_mode3.go:139
<a href="#"><img alt="P1" src="https://greptile-static-assets.s3.amazonaws.com/badges/p1.svg?v=9" align="top"></a> **Update is evaluated twice** Mode 3 passes the original `UpdatedObjectInfo` first to unified storage and then to legacy storage. Each backend can evaluate it against its independently fetched old object, so patch- or old-object-dependent updates can produce different results in the two stores. Preserve the object accepted by unified storage and pass it through an `updateWrapper` to legacy storage, as the other dual-write modes do.
pkg/apiserver/rest/dualwriter_mode3.go:104
<a href="#"><img alt="P1" src="https://greptile-static-assets.s3.amazonaws.com/badges/p1.svg?v=9" align="top"></a> **Legacy-only records survive deletion** Returning on every unified-storage delete error removes the previous `NotFound` fallback. During migration, a resource may still exist only in legacy storage; deleting it now returns `NotFound` without invoking `Legacy.Delete`, leaving the stale legacy record behind. Continue to legacy deletion when the primary result is `NotFound`, while retaining early returns for other errors.
pkg/apiserver/rest/dualwriter_mode3_test.go:203
<a href="#"><img alt="P1" src="https://greptile-static-assets.s3.amazonaws.com/badges/p1.svg?v=9" align="top"></a> **Async mock calls are unconfigured** The successful Delete test configures only the primary-storage call, although production now always starts an asynchronous `Legacy.Delete`. The successful DeleteCollection case has the same problem. If either goroutine reaches the mock, testify receives an unexpected call and can fail the test; if it runs after the subtest returns, the timing is nondeterministic. Configure the legacy expectations and wait for or assert their completion before each subtest exits.
pkg/apiserver/rest/dualwriter_mode3.go:45
<a href="#"><img alt="P2" src="https://greptile-static-assets.s3.amazonaws.com/badges/p2.svg?v=9" align="top"></a> **Metrics misclassify storage operations** Several new metric calls attribute operations incorrectly: Create and Update storage failures call `recordLegacyDuration`, successful Delete uses the object name as the kind label, and the asynchronous legacy DeleteCollection calls `recordStorageDuration`. This corrupts backend latency and error attribution, while the per-object Delete label can create high-cardinality metric series. Record each operation against the correct backend and consistently use `options.Kind`.
PR-level
<h3>Greptile Summary</h3> This PR implements and exercises Mode 3 dual-writer operations, adds playlist integration coverage for Mode 3, and refreshes workspace checksums. The production implementation introduces asynchronous legacy mirroring, but currently risks canceled or divergent secondary writes, regresses deletion of legacy-only records, and misattributes several metrics. - Implements unified-storage reads and primary-first writes for Mode 3. - Moves legacy Create, Update, Delete, and DeleteCollection operations to bounded goroutines. - Adds unit and playlist integration tests for Mode 3. - Updates Go workspace checksums. <h3>Confidence Score: 1/5</h3> The PR is not safe to merge until asynchronous legacy writes survive request completion, updates mirror the accepted object consistently, legacy-only deletion remains supported, and the new tests synchronize secondary writes. Mode 3 can return success while request cancellation aborts its legacy mirror, reevaluate updates differently across stores, and leave legacy-only records undeleted; the associated unit tests can also fail nondeterministically because asynchronous mock calls are neither configured nor awaited. **Files Needing Attention:** pkg/apiserver/rest/dualwriter_mode3.go, pkg/apiserver/rest/dualwriter_mode3_test.go <details><summary><h3>Important Files Changed</h3></summary> | Filename | Overview | |----------|----------| | pkg/apiserver/rest/dualwriter_mode3.go | Implements Mode 3 reads, asynchronous dual writes, logging, and metrics, with correctness issues in context lifetime, update reuse, deletion fallback, and metric attribution. | | pkg/apiserver/rest/dualwriter_mode3_test.go | Adds broad Mode 3 unit coverage, but asynchronous legacy operations are not synchronized and two successful cases omit required mock expectations. | | pkg/tests/apis/playlist/playlist_test.go | Adds Mode 3 playlist integration configurations for file and unified storage plus a skipped etcd case. | | pkg/apiserver/rest/dualwriter_mode1_test.go | Removes a shadowing local Prometheus registry declaration. | | go.work.sum | Adds workspace dependency checksums. | </details> <details open><summary><h3>Sequence Diagram</h3></summary> ```mermaid sequenceDiagram participant Client participant Mode3 participant Unified as Unified Storage participant Legacy as Legacy Storage Client->>Mode3: Write(request context) Mode3->>Unified: Primary write Unified-->>Mode3: Success Mode3-->>Client: Success par Asynchronous mirror Mode3->>Legacy: Legacy write using child of request context Note over Mode3,Legacy: Request completion may cancel the parent context Legacy-->>Mode3: Success or cancellation/error end ``` </details> <!-- greptile_other_comments_section --> <sub>Reviews (1): Last reviewed commit: ["Update go.work.sum and 4 more files"](https://github.com/drape-io/crbench-greptile-2026-09/commit/9280088a4c63b59cea860b769216e2974907711e) | [Re-trigger Greptile](https://app.greptile.com/api/retrigger?id=64284501)</sub>
Mjolnir (Opus 5.5) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 low ✗ —
d2 low ✓ exact
d3 medium ✓ off by one
d4 high ✓ off by one
valid · matches d4 · impact high
valid · impact low
In Create, the error branch after d.Storage.Create fails calls recordLegacyDuration, so storage failures are counted as legacy errors. [grounded: diff] [consensus 3/3: valid=3]
valid · impact low
In Update, the error branch after d.Storage.Update fails also calls recordLegacyDuration, so the storage failure is recorded under legacy. [grounded: diff] [consensus 3/3: valid=3]
valid · impact low
The DeleteCollection goroutine times the legacy call but reports it with recordStorageDuration. [grounded: diff] [consensus 3/3: valid=3]
valid · matches d3 · impact medium
valid · impact low
On a successful Delete, the object name is passed where every other call passes options.Kind. That label then grows with each distinct object name. [grounded: diff] [consensus 3/3: valid=3]
valid · impact low
This is the same issue as finding 3: legacy DeleteCollection timing is recorded in the storage metric. [grounded: diff] [consensus 3/3: valid=3]
valid · matches d2 · impact low
raw comments (8)
pkg/apiserver/rest/dualwriter_mode3.go:50
[P2] Async legacy writes inherit the request context and can be cancelled when the handler returns The legacy Create, Update, Delete and DeleteCollection calls now run in goroutines. Each goroutine uses `context.WithTimeoutCause(ctx, ...)`, which derives from the incoming request `ctx`. The method returns right after starting the goroutine. For HTTP-served requests, the apiserver/net/http cancels the request context once the handler finishes, so the legacy SQL write can fail with `context canceled` before it completes, and legacy storage silently diverges from unified storage. The goroutines also no longer log legacy errors (the old code did), so the only trace is the `isError` label on the duration metric. This also applies at lines 108-114, 134-141 and 161-167. Detaching the context (for example with `context.WithoutCancel(ctx)`) before adding the timeout would avoid it.
pkg/apiserver/rest/dualwriter_mode3.go:43
[P2] Storage failures in Create are recorded as legacy-duration metrics When `d.Storage.Create` fails, the error branch calls `d.recordLegacyDuration(true, ...)` instead of `recordStorageDuration`. Unified-storage create failures are therefore counted as legacy errors (and legacy is never called on that path), which hides storage errors and inflates legacy error rates on dashboards and alerts.
pkg/apiserver/rest/dualwriter_mode3.go:127
[P2] Storage failures in Update are recorded as legacy-duration metrics This is the same mistake as in Create: when `d.Storage.Update` returns an error, the code calls `d.recordLegacyDuration(true, ...)` with `startStorage`. The failed unified-storage update is attributed to legacy storage, and the storage error metric never sees it.
pkg/apiserver/rest/dualwriter_mode3.go:165
[P2] Legacy DeleteCollection timing is recorded as a storage metric The goroutine times `d.Legacy.DeleteCollection` but reports it with `d.recordStorageDuration(...)`. Each successful delete-collection therefore records two storage samples and no legacy sample, and legacy failures show up as storage errors. The call should be `recordLegacyDuration`, as in the other async legacy calls.
pkg/apiserver/rest/dualwriter_mode3.go:106
[P3] Successful Delete passes the object name as the metric `kind` label On success, the code calls `d.recordStorageDuration(false, mode3Str, name, method, startStorage)`, which passes `name` where every other call passes `options.Kind`. Every deleted object therefore creates a new label value. That causes unbounded Prometheus cardinality, and these samples don't match the `kind` label used by the error path and the legacy metric.
pkg/apiserver/rest/dualwriter_mode3.go:107
[P2] Delete records its storage metric under the object name, not the kind On a successful delete, `recordStorageDuration` gets `name` where every other call passes `options.Kind`. That fourth argument becomes a Prometheus label, so each deleted object name creates a new time series in the storage-duration histogram. This causes unbounded label cardinality. It also splits delete metrics away from the per-kind series that the error path and the other methods use.
pkg/apiserver/rest/dualwriter_mode3.go:163
[P2] Legacy DeleteCollection timing goes into the storage histogram The background goroutine times `d.Legacy.DeleteCollection` but reports it with `d.recordStorageDuration`. As a result, legacy delete-collection latency and errors are counted as unified-storage observations. The storage numbers become inflated, and legacy failures never appear in the legacy-duration metric. It should call `recordLegacyDuration`, as Create, Update and Delete do.
pkg/apiserver/rest/dualwriter_mode3.go:45
[P3] Storage failures in Create and Update are recorded as legacy failures When `d.Storage.Create` fails, the error branch calls `d.recordLegacyDuration(true, ...)` with the `startStorage` timer. Update does the same at line 127. Unified-storage errors are therefore counted as legacy errors with storage latency, and the storage error series for these methods is never incremented. Dashboards and alerts comparing the two backends during migration will point at the wrong store. These calls should use `recordStorageDuration`.
Mjolnir (Sonnet 5.5) · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 low ✓ exact
d2 low ✓ off by one
d3 medium ✓ exact
d4 high ✓ exact
valid · impact high
The goroutines derive their context from the request ctx without WithoutCancel, so cancellation propagates once the handler returns. Legacy writes that honour ctx can then be aborted. The change introduced this async pattern. [grounded: tree pkg/apiserver/rest/dualwriter_mode3.go] [consensus 3/3: valid=3]
valid · matches d2 · impact low
valid · matches d1 · impact low
valid · matches d4 · impact high
valid · impact medium
The removed code passed an updateWrapper holding the storage result to legacy. The new code passes the raw objInfo asynchronously, so legacy recomputes the update and the two stores can diverge. [grounded: tree pkg/apiserver/rest/dualwriter_mode3.go] [consensus 3/3: valid=3]
valid · matches d3 · impact medium
valid · impact low
Delete puts d.Log into the context instead of the enriched local log, unlike the sibling methods, so downstream logs lose the name/kind/method values. [grounded: tree pkg/apiserver/rest/dualwriter_mode3.go] [consensus 3/3: valid=3]
raw comments (7)
pkg/apiserver/rest/dualwriter_mode3.go:50
[P1] Async legacy writes use the request context, which is cancelled on return In Create, Delete, Update and DeleteCollection, the legacy write runs in a goroutine on a context derived from the request `ctx`. `context.WithTimeoutCause(ctx, ...)` inherits the request's cancellation. Once the handler returns the response, the request context is cancelled, so the goroutine's legacy call can fail with `context canceled`, or never run, depending on timing. With a legacy store that honours ctx (e.g. SQL), legacy writes can be silently dropped. The 10s timeout is also meaningless in that case. Detach the context (e.g. `context.WithoutCancel(ctx)`) before applying the timeout. The same pattern appears at lines 108-114, 134-141 and 161-167.
pkg/apiserver/rest/dualwriter_mode3.go:45
[P2] Legacy/storage metrics recorded under the wrong side or label Several metric calls are mislabelled. In Create (line 45) and Update (line 129), a failure of the primary `Storage` call is recorded with `recordLegacyDuration`. These errors are attributed to the legacy store, and storage error durations are never recorded. In DeleteCollection (line 166), the goroutine that runs `Legacy.DeleteCollection` calls `recordStorageDuration`, so legacy latency and errors are reported as storage. In Delete (line 106), the success path passes `name` instead of `options.Kind`, which creates a high-cardinality label series per object name. Metrics and dashboards for mode 3 will be wrong.
pkg/apiserver/rest/dualwriter_mode3.go:97
[P3] Delete installs the logger-less context instead of the enriched logger `log` is built with name, kind and method, but `klog.NewContext(ctx, d.Log)` stores the base `d.Log`. Storage and Legacy code that reads the logger from the context therefore loses these fields. Every other method uses `log`.
pkg/apiserver/rest/dualwriter_mode3.go:55
[P1] Async legacy writes inherit the request context and get cancelled The goroutine derives its 10s timeout context from the request `ctx` (same pattern in Update, Delete and DeleteCollection). The API server cancels that context once the handler returns, and the handler returns right after the goroutine starts. The legacy write therefore usually runs with an already-cancelled context and fails or is aborted with `context canceled`. Legacy SQL storage is then silently left out of sync, and the failure shows up only as a legacy-error metric. The goroutines should use a detached context, such as `context.WithoutCancel(ctx)` with the timeout, so the best-effort write can finish.
pkg/apiserver/rest/dualwriter_mode3.go:134
[P2] Legacy update is no longer given the object that storage produced The previous code wrapped the storage result in `updateWrapper` so legacy received the same final object. The new code passes the original `objInfo` to `Legacy.Update` in a goroutine, after the request has finished. Legacy therefore recomputes the update against its own current copy. Any resourceVersion/UID preconditions or generated fields in the request can then conflict or diverge between the two stores. The request-scoped `objInfo` is also read after the handler has returned. The write to legacy should use the object already computed by storage.
pkg/apiserver/rest/dualwriter_mode3.go:50
[P2] Duration metrics are recorded to the wrong series and with the wrong label Several calls use the wrong recorder or label: - Storage failures in Create (lines 50-51) and Update (lines 129-130) call `recordLegacyDuration` instead of `recordStorageDuration`. Storage errors are counted as legacy errors. - Delete's success path (line 105) passes `name` instead of `options.Kind` as the kind label. This creates a high-cardinality label per resource name and puts the series in the wrong kind bucket. - The legacy goroutine in DeleteCollection (line 168) calls `recordStorageDuration`, so legacy timings are recorded as storage. Dashboards and alerts built on these metrics will be wrong.
pkg/apiserver/rest/dualwriter_mode3.go:90
[P3] Delete installs the base logger rather than the enriched one `ctx = klog.NewContext(ctx, d.Log)` should use the local `log`, which carries the name, kind and method values. Downstream storage logs lose that context.
qodo · cross-file engine perris 0.11.1+proven.tiered0.25.call.read.subclass.type
DefectSeverityFoundGrade
d1 low ✗ —
d2 low ✗ —
d3 medium ✗ —
d4 high ✗ —
raw comments (2)
PR-level
<h3>Qodo is busy working</h3> Check back in a few minutes. Qodo's code review agents are on it. <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <img src="https://www.qodo.ai/wp-content/uploads/2026/01/anteater-looking-at-ants-01-transparent-bg-croped-ezgif.com-optimize-1.gif" width="30%">
PR-level
<h3>PR Summary by Qodo</h3> Complete Mode 3 unified-first dual-write behavior <code>✨ Enhancement</code> <code>🧪 Tests</code> <code>🕐 20-40 Minutes</code> <img src="https://www.qodo.ai/wp-content/uploads/2025/11/light-grey-line.svg" height="10%" alt="Grey Divider"> <details> <summary>AI Description</summary> <dl> <dd> <br/> ><pre> >• Makes unified storage authoritative for Mode 3 reads and synchronous writes. >• Mirrors successful mutations asynchronously to legacy storage with timeouts and metrics. >• Adds unit and playlist integration coverage across supported storage backends. ></pre> </dd> </dl> </details> <details> <summary>Diagram</summary> <dl> <dd> <br/> ```mermaid sequenceDiagram actor Client participant Mode3 as Mode 3 Writer participant Unified as Unified Store participant Legacy as Legacy Store participant Metrics alt Get or List Client->>Mode3: Read request Mode3->>Unified: Read Unified-->>Mode3: Result Mode3->>Metrics: Record duration Mode3-->>Client: Read result else Mutation Client->>Mode3: Write request Mode3->>Unified: Synchronous write Unified-->>Mode3: Primary result Mode3->>Metrics: Record duration Mode3-->>Client: Primary result Mode3->>Legacy: Async mirror (10s) Legacy-->>Mode3: Best-effort result Mode3->>Metrics: Record mirror duration end ``` </dd> </dl> </details> <details> <summary>High-Level Assessment</summary> <dl> <dd> <br/> >The following are alternative approaches to this PR: <details> <summary><b>1. Synchronous dual writes</b></summary> <dl> <dd> - ➕ Provides immediate confirmation that both stores accepted each mutation. - ➕ Avoids losing legacy writes during process shutdown or request cancellation. - ➖ Adds legacy storage latency to every Mode 3 mutation. - ➖ Allows legacy failures to reduce availability of the authoritative unified store. </dd> </dl> </details> <details> <summary><b>2. Durable background queue</b></summary> <dl> <dd> - ➕ Supports retries and survives request cancellation or process restarts. - ➕ Preserves low client latency while improving legacy convergence guarantees. - ➖ Requires queue or outbox infrastructure and worker lifecycle management. - ➖ Introduces additional operational complexity and observable replication lag. </dd> </dl> </details> >**Recommendation:** Unified-first writes with asynchronous legacy mirroring fit Mode 3&#x27;s migration goal and avoid coupling client availability to the legacy store. The current approach is suitable when mirroring is explicitly best effort; use a lifecycle-managed durable queue instead if legacy convergence must be guaranteed. </dd> </dl> </details> <details> <summary> Files changed (5) <code> +524 / -131 </code> </summary> <dl> <dd> <br/> <details> <summary>Enhancement (1) <code> +98 / -60 </code></summary> <dl> <dd> <details> <summary>dualwriter_mode3.go<code>Implement observable unified-first Mode 3 operations</code> <code>+98/-60</code></summary> <br/> >Implement observable unified-first Mode 3 operations > ><pre> >• Makes unified storage authoritative for Get, List, and all mutation responses. Successful creates, updates, deletes, and collection deletes are mirrored asynchronously to legacy storage with ten-second timeouts, structured logging, and duration metrics. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/42/files#diff-f566900342eda1197a96f2af7efda036f1d4d7c15c99c7a26892c5f47ba72a3e'>pkg/apiserver/rest/dualwriter_mode3.go</a> </details> </dd> </dl> </details> <details> <summary>Tests (3) <code> +410 / -71 </code></summary> <dl> <dd> <details> <summary>dualwriter_mode1_test.go<code>Reuse the shared test metrics registry</code> <code>+0/-1</code></summary> <br/> >Reuse the shared test metrics registry > ><pre> >• Removes a locally shadowed Prometheus registry from the Mode 1 get test so it uses the package-level test registry consistently. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/42/files#diff-523751c29f56877dc6e5c815cc1e77cabae0247d3237f135aeb72212a65bac42'>pkg/apiserver/rest/dualwriter_mode1_test.go</a> </details> <details> <summary>dualwriter_mode3_test.go<code>Add Mode 3 CRUD unit coverage</code> <code>+354/-70</code></summary> <br/> >Add Mode 3 CRUD unit coverage > ><pre> >• Replaces the commented test skeleton with table-driven tests for Create, Get, List, Update, Delete, and DeleteCollection. Covers successful unified-store operations and propagation of primary-store failures. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/42/files#diff-0efd0904d38018163e4832b7f141d10461eef2513483407b0215cd714d2bd953'>pkg/apiserver/rest/dualwriter_mode3_test.go</a> </details> <details> <summary>playlist_test.go<code>Exercise playlists under dual-write Mode 3</code> <code>+56/-0</code></summary> <br/> >Exercise playlists under dual-write Mode 3 > ><pre> >• Adds playlist integration scenarios for Mode 3 using file and unified storage. Also adds a skipped local-etcd scenario with collection cleanup for parity with the other dual-write modes. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/42/files#diff-1cdeb9b90d7fd21df4ef8692cd0e4c6ffc463e9261b270ebfc42b48e2950b959'>pkg/tests/apis/playlist/playlist_test.go</a> </details> </dd> </dl> </details> <details> <summary>Other (1) <code> +16 / -0 </code></summary> <dl> <dd> <details> <summary>go.work.sum<code>Add workspace dependency checksums</code> <code>+16/-0</code></summary> <br/> >Add workspace dependency checksums > ><pre> >• Adds checksums for newly resolved Grafana SDK, Alertmanager, OpenTelemetry, and Google API module versions required by the workspace dependency graph. ></pre> > ><a href='https://github.com/drape-io/crbench-qodo-2026-09/pull/42/files#diff-20cee3a725d1c5d2c595ff5372e6593d3bcaa7aebd8319866f65e90c6924f247'>go.work.sum</a> </details> </dd> </dl> </details> </dd> </dl> </details>