Skip to content

fix: ignore duplicate workflow_job completed events for a job already scaled down - #4594

Open
ankit373 wants to merge 3 commits into
actions:masterfrom
ankit373:fix/4315-workflow-job-completed-event-dedup
Open

ankit373 wants to merge 3 commits into
actions:masterfrom
ankit373:fix/4315-workflow-job-completed-event-dedup

Conversation

@ankit373

@ankit373 ankit373 commented Aug 5, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #4315.

Bug

GitHub resends workflow_job completed events for jobs that already finished in an earlier run of the same workflow run, when a user reruns only the failed jobs instead of all jobs. The webhook server applies every completed event as an unconditional -1 capacity reservation. A burst of resent completions lands in the same batch as the new queued event for the rerun job, nets the reservation to zero, and the rerun job's runner never gets scheduled. It stays queued indefinitely.

The issue includes webhook server logs showing exactly this: one queued event and five resent completed events processed in the same second, netting {"added": 1, "completed": 5, "after": 0}.

Fix

A workflow job ID only completes once. Handle now tracks job IDs it already scaled down for and skips a repeat completed event for the same ID within a bounded window (scaledDownForDedupWindow, one hour). This also covers plain webhook redeliveries, not just the rerun case.

The dedup state is in-memory and per-pod, same as the worker/batch-scaler queue it feeds. Multiple webhook-server replicas without sticky routing already have that limitation for the existing queue, so this isn't a new constraint.

Test plan

Added TestWebhookWorkflowJobCompletedDedup: sends the same completed event through one handler instance twice. First delivery scales down by 1, second is ignored.

Verified the test catches the bug by reverting the fix first: both deliveries scaled down by 1.

  • go build ./controllers/...
  • go vet ./controllers/actions.summerwind.net/...
  • go test -race -count=1 ./controllers/actions.summerwind.net/... (all non-envtest tests pass)
  • gofmt -l clean on both touched files

… scaled down

GitHub resends completed events for jobs that finished in an earlier run
of the same workflow run when a user reruns only the failed jobs. The
webhook server applies every completed event as an unconditional -1
capacity reservation, so a burst of resent completions lands in the same
batch as the new queued event and nets the reservation to zero. The
rerun job's runner never gets scheduled and stays queued indefinitely.

A workflow job ID only completes once, so track IDs we already scaled
down for and skip a repeat completed event for the same ID within a
bounded window. This also covers plain webhook redeliveries.

Dedup state is in-memory and per-pod, matching the worker/batch-scaler
queue it feeds. Multiple webhook-server replicas without sticky routing
already have that limitation, so this isn't a new one.

Added a test that sends the same completed event twice through one
handler: first delivery scales down by 1, second is ignored. Verified
against the unpatched code first, both deliveries scaled down by 1.

Fixes actions#4315
@ankit373
ankit373 force-pushed the fix/4315-workflow-job-completed-event-dedup branch from 2637090 to c680d45 Compare August 5, 2026 09:12
@ankit373
ankit373 requested review from a team and nikola-jokic as code owners September 29, 2026 18:30
@ankit373

Copy link
Copy Markdown
Contributor Author

Friendly ping, this has been open since August. I merged current master in (52 commits behind, no conflicts) and re-ran it on the merged tree: build, go vet, golangci-lint and the controllers/actions.summerwind.net tests all pass locally.

CI has not actually run on this PR (the August runs never started any jobs), so could a maintainer approve a workflow run? @nikola-jokic, would you have a moment to look, or can you point me to whoever owns the webhook autoscaler? Happy to change anything.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Re-run workflow jobs fail to scale runners

1 participant