Repository navigation
Requeue rate-limited scale-set requests without client retries - #4726
Merged
Merged
Conversation
nikola-jokic
requested review from
a team,
Steve-Glass,
mumoshu,
rentziass and
toast-gear
as code owners
October 23, 2026 13:20
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Concurrent authentication pooling and listener acknowledgment changes warrant human validation across recovery and lifecycle paths.
No open issues 🎉
What changed in this PR
This PR addresses #4699 by bounding Actions service calls so rate-limited registrations release controller workers for other work, including completed-runner cleanup. It also migrates the listener to scaleset v0.5.0.
Changes:
- Adds 10-second service-call deadlines, two-retry limits, and rate-limit-aware reconciliation delays.
- Pools SDK clients to isolate concurrent authentication while reusing tokens and HTTP transport.
- Adapts listener message processing and metrics, with retry, concurrency, and recovery tests.
| File | Description |
|---|---|
| go.sum | Updates dependency checksums for the SDK migration. |
| go.mod | Upgrades scaleset to v0.5.0. |
| docs/gha-runner-scale-set-controller/README.md | Documents retry bounds, client isolation, and message acknowledgment. |
| controllers/actions.github.com/scaleset_retry.go | Converts rate-limit errors into reconciliation delays. |
| controllers/actions.github.com/scaleset_retry_test.go | Tests retry limits, deadlines, and concurrent authentication. |
| controllers/actions.github.com/scaleset_retry_controller_test.go | Tests worker release and controller recovery after throttling. |
| controllers/actions.github.com/multiclient/multi_client.go | Creates pooled clients sharing an HTTP transport. |
| controllers/actions.github.com/multiclient/bounded_client.go | Adds client leasing and bounded service calls. |
| controllers/actions.github.com/multiclient/bounded_client_test.go | Tests client reuse and canceled acquisition. |
| controllers/actions.github.com/ephemeralrunnerset_controller.go | Requeues throttled reconciliations and stops throttled cleanup batches. |
| controllers/actions.github.com/ephemeralrunner_controller.go | Applies rate-limit-aware reconciliation delays. |
| controllers/actions.github.com/autoscalingrunnerset_controller.go | Applies rate-limit-aware reconciliation delays. |
| cmd/ghalistener/scaler/scaler.go | Handles SDK messages, job acquisition, scaling, and metrics. |
| cmd/ghalistener/scaler/message_test.go | Tests message processing and failure propagation. |
| cmd/ghalistener/main.go | Wires the session client and metrics into the scaler. |
| cmd/ghalistener/config/config.go | Adapts TLS and proxy configuration to the SDK API. |
| cmd/ghalistener/config/config_client_test.go | Tests proxy routing through HTTP requests. |
Review effort: Balanced
nikola-jokic
changed the title
salmanmkc
self-requested a review
October 23, 2026 14:49
salmanmkc
approved these changes
Oct 23, 2026
salmanmkc
left a comment
salmanmkc
left a comment
Contributor
There was a problem hiding this comment.
Looks great, ready to ship 🚀
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Disable client-side retries for controller service calls. Requeue throttled reconciliations using
Retry-After, falling back to 30 seconds without a usable hint. Cap ordinary controller error backoff at 30 seconds. Keep 10-second call deadlines, isolated authentication, and scaleset v0.5.0 listener compatibility.Validated with controller/listener race tests, retry-policy regressions, build, and vet. The full controller integration suite passed before this timing-only adjustment.
Fixes #4699.