Repository navigation
Conversation
The Task controller added and removed its finalizer with Update, which sends the entire object back to the API server. The Task spec is immutable (the CRD's self == oldSelf rule), so when a stored spec no longer round-trips byte for byte through the current Go types, as with Tasks created before a schema change, the API server rejects the Update with "Task spec is immutable after creation". A deleted Task then keeps its finalizer forever and never finishes deleting. Use a metadata-only merge patch for both the add and the remove, so the spec is never written back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Scott Holodak seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
There was a problem hiding this comment.
1 issue found across 2 files
You’re at about 91% of the monthly reviewed-line limit. You may want to disable incremental reviews to conserve quota. Reviews will continue until that limit is exceeded. If you need help avoiding interruptions, please contact contact@cubic.dev.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="internal/controller/task_controller.go">
<violation number="1" location="internal/controller/task_controller.go:308">
P1: `MergeFrom` replaces the entire `metadata.finalizers` list without checking `resourceVersion`, so a concurrent finalizer update can be silently lost here (and at the add site). Use `MergeFromWithOptions(original, client.MergeFromWithOptimisticLock{})` at both sites so stale patches conflict instead.</violation>
</file>
Reply with feedback, questions, or to request a fix.
View guided diff | Re-trigger cubic
| original := task.DeepCopy() | ||
| controllerutil.RemoveFinalizer(task, taskFinalizer) | ||
| if err := r.Update(ctx, task); err != nil { | ||
| if err := r.Patch(ctx, task, client.MergeFrom(original)); err != nil { |
There was a problem hiding this comment.
P1: MergeFrom replaces the entire metadata.finalizers list without checking resourceVersion, so a concurrent finalizer update can be silently lost here (and at the add site). Use MergeFromWithOptions(original, client.MergeFromWithOptimisticLock{}) at both sites so stale patches conflict instead.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. At internal/controller/task_controller.go, line 308:
<comment>`MergeFrom` replaces the entire `metadata.finalizers` list without checking `resourceVersion`, so a concurrent finalizer update can be silently lost here (and at the add site). Use `MergeFromWithOptions(original, client.MergeFromWithOptimisticLock{})` at both sites so stale patches conflict instead.</comment>
<file context>
@@ -296,9 +298,14 @@ func (r *TaskReconciler) handleDeletion(ctx context.Context, task *kelos.Task) (
+ original := task.DeepCopy()
controllerutil.RemoveFinalizer(task, taskFinalizer)
- if err := r.Update(ctx, task); err != nil {
+ if err := r.Patch(ctx, task, client.MergeFrom(original)); err != nil {
logger.Error(err, "unable to remove finalizer")
return ctrl.Result{}, err
</file context>
|
Can you set the commit author properly? |
What type of PR is this?
/kind bug
What this PR does / why we need it:
The Task controller adds and removes
kelos.dev/finalizerwithUpdate, which sends the whole Task back to the API server.TaskSpecis immutable (+kubebuilder:validation:XValidation:rule="self == oldSelf"). So when a stored spec no longer round-trips byte for byte through the current Go types, the API server rejects theUpdate:This happens, for example, to Tasks created before a schema change. The deletion path then returns that error on every reconcile ("unable to remove finalizer"), and the Task never finishes deleting. On one cluster this left 144 deleted Tasks stuck for up to four weeks. Each had already had its Job removed, but kept its finalizer.
This PR switches both the add and the remove to a metadata-only merge patch (
client.MergeFrom(original)), so the spec is never written back.Tests:
TestReconcile_DeletionRemovesFinalizerWithoutUpdatingSpecandTestReconcile_AddsFinalizerWithoutUpdatingSpecuse a fake client whose interceptor refuses every TaskUpdatewith the same Invalid error the API server returns. Without the change, both fail with that error. With it, the deleted Task is removed and the new Task gets its finalizer.go test ./internal/controller/andgo vetpass.Which issue(s) this PR is related to:
N/A
Special notes for your reviewer:
Only the Task controller is changed. Other controllers that
Updateobjects with an immutable spec could be affected in the same way. I kept this PR focused, but can follow up.Does this PR introduce a user-facing change?