Repository navigation
Runner gets stuck in infinite loop of retries when registration is not found #4191
Description
Describe the bug
When a runner cannot locate the registration, it enters an infinite retry loop.
Expected behavior
Terminate after the set number of retries fail. Do not reset retries and keep going.
Runner Version and Platform
Version of your runner?
Current runner version: '2.330.0'
OS of the machine running the runner? OSX/Windows/Linux/...
Running on EKS 1.32 using bottlerocket-aws-k8s-1.32-x86_64-v1.42.0-5ed15786
What's not working?
Registration error:
[RUNNER 2026-01-09 17:55:48Z ERR BrokerServer] Catch exception during request
[RUNNER 2026-01-09 17:55:48Z ERR BrokerServer] GitHub.Services.OAuth.VssOAuthTokenRequestException: Registration <REDACTED> was not found.
[RUNNER 2026-01-09 17:55:48Z ERR BrokerServer] at GitHub.Services.OAuth.VssOAuthTokenProvider.OnGetTokenAsync(IssuedToken failedToken, CancellationToken cancellationToken)
[RUNNER 2026-01-09 17:55:48Z ERR BrokerServer] at GitHub.Services.Common.IssuedTokenProvider.GetTokenOperation.GetTokenAsync(VssTraceActivity traceActivity)
[RUNNER 2026-01-09 17:55:48Z ERR BrokerServer] at GitHub.Services.Common.IssuedTokenProvider.GetTokenAsync(IssuedToken failedToken, CancellationToken cancellationToken)
[RUNNER 2026-01-09 17:55:48Z ERR BrokerServer] at GitHub.Services.Common.RawHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
[RUNNER 2026-01-09 17:55:48Z ERR BrokerServer] at GitHub.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
[RUNNER 2026-01-09 17:55:48Z ERR BrokerServer] at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
[RUNNER 2026-01-09 17:55:48Z ERR BrokerServer] at Sdk.WebApi.WebApi.RawHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
[RUNNER 2026-01-09 17:55:48Z ERR BrokerServer] at Sdk.WebApi.WebApi.RawHttpClientBase.SendAsync[T](HttpRequestMessage message, Boolean readErrorBody, Object userState, CancellationToken cancellationToken)
[RUNNER 2026-01-09 17:55:48Z ERR BrokerServer] at Sdk.WebApi.WebApi.RawHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Uri requestUri, HttpContent content, IEnumerable`1 queryParameters, Boolean readErrorBody, Object userState, CancellationToken cancellationToken)
[RUNNER 2026-01-09 17:55:48Z ERR BrokerServer] at GitHub.Actions.RunService.WebApi.BrokerHttpClient.GetRunnerMessageAsync(Nullable`1 sessionId, String runnerVersion, Nullable`1 status, String os, String architecture, Nullable`1 disableUpdate, CancellationToken cancellationToken)
[RUNNER 2026-01-09 17:55:48Z ERR BrokerServer] at GitHub.Runner.Common.BrokerServer.<>c__DisplayClass7_0.<<GetRunnerMessageAsync>b__0>d.MoveNext()
[RUNNER 2026-01-09 17:55:48Z ERR BrokerServer] --- End of stack trace from previous location ---
[RUNNER 2026-01-09 17:55:48Z ERR BrokerServer] at GitHub.Runner.Common.RunnerService.RetryRequest[T](Func`1 func, CancellationToken cancellationToken, Int32 maxAttempts, Func`2 shouldRetry)
[RUNNER 2026-01-09 17:55:48Z WARN BrokerServer] Back off 9.418 seconds before next retry. 4 attempt left.
It says it will only retry 4 times, but continues forever. This leaves the pod in the "Running" status so it is never terminated without manual intervention.
It goes from 4 attempt left -> 3 attempt left -> 2 attempt left -> 1 attempt left -> 4 attempt left and repeats forever until the pod is deleted.
Job Log Output
Jobs are not picked up. They remain queued until the runner pods are deleted.
This can impact one set of runners while others remain unaffected at the same time.
Activity
TingluoHuang commentedon Jan 9, 2026
@margeauxg-medium can you share the full runner diag log? 🙇
Also, did you delete the runner via the UI or API?
margeauxg-medium commentedon Jan 10, 2026
Hi @TingluoHuang , attached are the full logs for the runner pod. This ran for ~1 hour until we removed the runner pod and jobs began being picked up again.
I don't delete the runner from GitHub directly, I just
kubectl deletethe pods in the runner scale set so they get recreated.stuck_runner_logs.txt
bkw commentedon Apr 9, 2026
We hit the same issue on GKE with ARC 0.13.0 and runner 2.331.0.
Our scenario: An ephemeral runner (
gke-runner-tests-cpu2-memory4) was created for a job, had pod stability issues during startup (pod recreated once), then successfully registered and started listening at 09:23 UTC. It sat idle for ~51 minutes, the job it was created for was likely assigned to another runner. At 10:14 UTC, the registration was garbage-collected by GitHub, and the runner entered the infinite retry loop:It stayed in this loop for ~11 minutes before eventually hitting a different code path that triggered a clean exit:
Impact: During the retry loop, the pod remained in
Runningstatus. The ARC controller counted it as a live runner (running=1, desired=1→ "nothing to do"), so no replacement was created. Meanwhile GitHub reportedtotalRegisteredRunners: 0for this scale set. Jobs assigned to the scale set were canceled twice by GitHub (~5 min timeout each) before a burst of unrelated jobs forced the controller to scale up and create fresh runners that actually registered.Key observations:
ACTIONS_RUNNER_PRINT_LOG_TO_STDOUT=0(which we use to reduce log volume), the error is invisible in cluster logging, making diagnosis difficult.The ideal fix would be for the runner to exit with a non-retryable error after exhausting retries on "Registration was not found," allowing the ARC controller to clean up the pod and create a replacement.
fix: treat deleted runner registration as non-retryable error