Ephemeral Runners seem to get stuck when job is canceled or interrupted #4307

Activity

1. surgiie

   added

   bugSomething isn't workingSomething isn't working

   needs triageRequires review from the maintainersRequires review from the maintainers

   gha-runner-scale-setRelated to the gha-runner-scale-set modeRelated to the gha-runner-scale-set mode

   on Nov 7, 2025
2. github-actions commented on Nov 7, 2025

   

   github-actionsBot

   on Nov 7, 2025 – with GitHub Actions

   Contributor

   More actions

   Hello! Thank you for filing an issue.

   The maintainers will triage your issue shortly.

   In the meantime, please take a look at the troubleshooting guide for bug reports.

   If this is a feature request, please review our contribution guidelines.
3. badstreff commented on Nov 8, 2025

   

   badstreff

   on Nov 8, 2025

   More actions

   If you do a kubectl get ephemeralrunners do you see a lot of them sitting in a failed state? I have PR - #4272 that may also resolve the issue you see here but has not been getting much traction.
4. surgiie commented on Nov 9, 2025

   

   surgiie

   on Nov 9, 2025

   Author

   More actions

   @badstreff No not really, thats what i find odd about this issue. It seems to happen at random but as far as i can tell inspecting the runners only shows a lot of runners with no status on them and then just the ones in "Running" state that show logs as described above.
5. patrickvinograd commented on Nov 10, 2025

   

   patrickvinograd

   on Nov 10, 2025

   More actions

   As a datapoint, one way we've consistently been able to trigger this is if the CI workflow being run OOMs.

   👍React with 👍2Reacted by Sergio Compean and Javier Gutierrez Gonzalez
6. alexgaganashvili commented on Nov 11, 2025

   

   alexgaganashvili

   on Nov 11, 2025

   Last edited by alexgaganashvili

   More actions

   Just ran into the same problem. Can't cancel a workflow. An EphemeralRunner is showing Running.
7. surgiie commented on Nov 12, 2025

   

   surgiie

   on Nov 12, 2025

   Author

   More actions

   As a datapoint, one way we've consistently been able to trigger this is if the CI workflow being run OOMs.

   Interesting, i will look for OOM failures next time i notice this issue. Like i said, for us it doesnt happen everyday and it seems to happen at random. That said, the runner should be able to clean this up. Hoping the arc team fixes this soon but im getting away with the cronjob i posted above that helps mitigate the issue when it does happen.
8. genisd commented on Nov 19, 2025

   

   genisd

   on Nov 19, 2025

   Last edited by genisd

   Contributor

   More actions

   OOM is definitely not he only cause. We don't have OOMs at all on our runner and we get this as well.
   Infact we have cronjobs which cleanup ephemeral runners which pretty much match: .*Skipping message Job\..*. (in reality we exactly match the log messages which we know of).

   this is the exact regex:

   "(GitHub\.Services\.OAuth.VssOAuthTokenRequestException: Registration .* was not found|Skipping message Job\. Job message not found '.*'\. job (was canceled|not found)|Skipping message Job\. Job message already acquired)"
   

   this regex covers like multiple scenarios

   • job cancelled (manual cancel race condition OR cancelled due to concurrency limit)
   • not found ( 🤷‍♂️ )
   • already acquired
   • runner registration not found (yielding in runner unable to pull job)

   The fact that this particular job fails isn't even the issue imho, but it reserves a slot which is considered "available" but not picking up another job from the queue yielding in starvation :(. That goes for all of these scenarios above

   here a gist with all our "cleaners".

   https://gist.github.com/genisd/54fddf27cbe1bb875ef49412833612c1

   i really need to invest more time into these to be able to get rid of them ;-)
   i trust that with these cleaners (use at your own risk) you will find this (these) particular issue(s) disappearing

   👍React with 👍3Reacted by Sergio Compean, Fabrizio Sabatini and Louis Auzuret
9. surgiie commented on Nov 19, 2025

   

   surgiie

   on Nov 19, 2025

   Author

   More actions

   OOM is definitely not he only cause. We don't have OOMs at all on our runner and we get this as well. Infact we have cronjobs which cleanup ephemeral runners which pretty much match: .*Skipping message Job\..*. (in reality we exactly match the log messages which we know of).

   this is the exact regex:

   "(GitHub\.Services\.OAuth.VssOAuthTokenRequestException: Registration .* was not found|Skipping message Job\. Job message not found '.*'\. job (was canceled|not found)|Skipping message Job\. Job message already acquired)"
   

   this regex covers like multiple scenarios

   • job cancelled (manual cancel race condition OR cancelled due to concurrency limit)
   • not found ( 🤷‍♂️ )
   • already acquired
   • runner registration not found (yielding in runner unable to pull job)

   The fact that this particular job fails isn't even the issue imho, but it reserves a slot which is considered "available" but not picking up another job from the queue yielding in starvation :(. That goes for all of these scenarios above

   here a gist with all our "cleaners".

   https://gist.github.com/genisd/54fddf27cbe1bb875ef49412833612c1

   i really need to invest more time into these to be able to get rid of them ;-) i trust that with these cleaners (use at your own risk) you will find this (these) particular issue(s) disappearing

   Thanks for sharing, may update our cron to cover more areas.
10. nikola-jokic commented on Dec 5, 2025

    

    nikola-jokic

    on Dec 5, 2025

    Collaborator

    More actions

    Hey, could anyone please provide -o yaml of the pod that gets stuck like this? I'd love to understand the state in which the pod is in to see why/how are we not handling this scenario properly.
11. anttijman commented on Dec 9, 2025

    

    anttijman

    on Dec 9, 2025

    More actions

    Sorry about the offtopic but @surgiie are you using the ACTIONS_RUNNER_PRINT_LOG_TO_STDOUT for streaming the runner logs to stdout? Ephemeral runners in my setup are streaming only the log lines without the prefixes to stdout.
12. surgiie commented on Dec 9, 2025

    

    surgiie

    on Dec 9, 2025

    Author

    More actions

    ACTIONS_RUNNER_PRINT_LOG_TO_STDOUT

    Yes. I have this set to 1.
13. surgiie commented on Dec 9, 2025

    

    surgiie

    on Dec 9, 2025

    Author

    More actions

    Hey, could anyone please provide -o yaml of the pod that gets stuck like this? I'd love to understand the state in which the pod is in to see why/how are we not handling this scenario properly.

    I will try to do this soon. Im on holiday break soon so if someone is able to produce logs before that would be great otherwise, ill try to circle back to this soon.
14. 2 remaining items

    Load more

15. atyu1 commented on Jan 5, 2026

    

    atyu1

    on Jan 5, 2026

    More actions

    Same issue happened for us 4 days back, we have a runner on version 2.330.0 , the job failed after 44min and 6s with the same stack trace.

    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer] System.Threading.Tasks.TaskCanceledException: The operation was canceled.
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]  ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]  ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]  ---> System.Net.Sockets.SocketException (125): Operation canceled
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    --- End of inner exception stack trace ---
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory`1 buffer, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    --- End of inner exception stack trace ---
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, HttpConnectionPool connectionPool, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at GitHub.Services.Common.RawHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at GitHub.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    --- End of inner exception stack trace ---
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at Sdk.WebApi.WebApi.RawHttpClientBase.SendAsync(HttpRequestMessage message, HttpCompletionOption completionOption, Object userState, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at Sdk.WebApi.WebApi.RawHttpClientBase.SendAsync[T](HttpRequestMessage message, Boolean readErrorBody, Object userState, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at Sdk.WebApi.WebApi.RawHttpClientBase.SendAsync[T](HttpMethod method, IEnumerable`1 additionalHeaders, Uri requestUri, HttpContent content, IEnumerable`1 queryParameters, Boolean readErrorBody, Object userState, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at GitHub.Actions.RunService.WebApi.BrokerHttpClient.GetRunnerMessageAsync(Nullable`1 sessionId, String runnerVersion, Nullable`1 status, String os, String architecture, Nullable`1 disableUpdate, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at GitHub.Runner.Common.BrokerServer.<>c__DisplayClass7_0.<<GetRunnerMessageAsync>b__0>d.MoveNext()
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer] --- End of stack trace from previous location ---
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at GitHub.Runner.Common.RunnerService.RetryRequest[T](Func`1 func, CancellationToken cancellationToken, Int32 maxAttempts, Func`2 shouldRetry)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer] #####################################################
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer] System.Threading.Tasks.TaskCanceledException: The operation was canceled.
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]  ---> System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]  ---> System.Net.Sockets.SocketException (125): Operation canceled
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    --- End of inner exception stack trace ---
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory`1 buffer, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    --- End of inner exception stack trace ---
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Http.AuthenticationHelper.SendWithNtAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean isProxyAuth, HttpConnection connection, HttpConnectionPool connectionPool, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Http.AuthenticationHelper.SendWithAuthAsync(HttpRequestMessage request, Uri authUri, Boolean async, ICredentials credentials, Boolean preAuthenticate, Boolean isProxyAuth, Boolean doRequestAuth, HttpConnectionPool pool, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at GitHub.Services.Common.RawHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at GitHub.Services.Common.VssHttpRetryMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer] #####################################################
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer] System.IO.IOException: Unable to read data from the transport connection: Operation canceled.
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]  ---> System.Net.Sockets.SocketException (125): Operation canceled
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    --- End of inner exception stack trace ---
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Security.SslStream.EnsureFullTlsFrameAsync[TIOAdapter](CancellationToken cancellationToken, Int32 estimatedSize)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Security.SslStream.ReadAsyncInternal[TIOAdapter](Memory`1 buffer, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer]    at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer] #####################################################
    [RUNNER 2026-01-01 18:12:47Z ERR  BrokerServer] System.Net.Sockets.SocketException (125): Operation canceled
    

    Interesting that at the beginning I have another stack trace:

    [WORKER 2026-01-01 17:27:52Z INFO ProcessInvokerWrapper] Failed to update oom_score_adj for PID: 46.
    [WORKER 2026-01-01 17:27:52Z INFO ProcessInvokerWrapper] System.UnauthorizedAccessException: Access to the path '/proc/46/oom_score_adj' is denied.
    [WORKER 2026-01-01 17:27:52Z INFO ProcessInvokerWrapper]  ---> System.IO.IOException: Permission denied
    [WORKER 2026-01-01 17:27:52Z INFO ProcessInvokerWrapper]    --- End of inner exception stack trace ---
    [WORKER 2026-01-01 17:27:52Z INFO ProcessInvokerWrapper]    at System.IO.RandomAccess.WriteAtOffset(SafeFileHandle handle, ReadOnlySpan`1 buffer, Int64 fileOffset)
    [WORKER 2026-01-01 17:27:52Z INFO ProcessInvokerWrapper]    at System.IO.File.WriteToFile(String path, FileMode mode, String contents, Encoding encoding)
    [WORKER 2026-01-01 17:27:52Z INFO ProcessInvokerWrapper]    at GitHub.Runner.Sdk.ProcessInvoker.WriteProcessOomScoreAdj(Int32 processId, Int32 oomScoreAdj)
    [WORKER 2026-01-01 17:27:52Z INFO ProcessInvokerWrapper] Process started with process id 46, waiting for process exit.
    [WORKER 2026-01-01 17:27:52Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
    [WORKER 2026-01-01 17:27:52Z INFO ProcessInvokerWrapper] STDOUT/STDERR stream read finished.
    

    Thanks!
16. genisd commented on Jan 5, 2026

    

    genisd

    on Jan 5, 2026

    Contributor

    More actions

    from today's harvest, 2 failure scenarios.
    please note we're still on 0.12.1
    gh runner pod tag we use is :latest

    registration was not found:
    pod yaml: https://gist.github.com/genisd/bb028c473b6d83ce1bb506fe796ca16e
    logs: https://gist.github.com/genisd/9fd46e604ee5cd02b93cb55220c01628

    job already acquired:
    pod yaml: https://gist.github.com/genisd/59fdd30abffe2ebc51c8ff2467873df0
    logs: https://gist.github.com/genisd/25ccbfea99f68d06ed6db436b8121c82

    👍React with 👍2Reacted by Sergio Compean and Louis Auzuret
17. denysvitali-kaiko commented on Jan 7, 2026

    

    denysvitali-kaiko

    on Jan 7, 2026

    Last edited by denysvitali-kaiko

    More actions

    Happens to us as well, and it seems similar to actions/runner#3904, actions/runner#3857, and #3953 (comment)

    For reference, we keep on seeing backend errors such as:

    [RUNNER 2026-01-07 13:18:54Z ERR GitHubActionsService] POST request to https://run-actions-2-azure-eastus.actions.githubusercontent.com/33/acquirejob failed. HTTP Status: Conflict 
    [RUNNER 2026-01-07 13:18:54Z INFO Runner] Skipping message Job. Job message already acquired 'ab9b2080-1ae3-553f-88bd-5cb422b4a429'. job assignment is invalid: MissingKey
    
18. margeauxg-medium commented on Jan 22, 2026

    

    margeauxg-medium

    on Jan 22, 2026

    More actions

    I have a similar issue. Full pod logs are included here - actions/runner#4191
19. surgiie commented on Jan 23, 2026

    

    surgiie

    on Jan 23, 2026

    Author

    More actions

    Happens to us as well, and it seems similar to actions/runner#3904, actions/runner#3857, and #3953 (comment)

    For reference, we keep on seeing backend errors such as:

    [RUNNER 2026-01-07 13:18:54Z ERR GitHubActionsService] POST request to https://run-actions-2-azure-eastus.actions.githubusercontent.com/33/acquirejob failed. HTTP Status: Conflict 
    [RUNNER 2026-01-07 13:18:54Z INFO Runner] Skipping message Job. Job message already acquired 'ab9b2080-1ae3-553f-88bd-5cb422b4a429'. job assignment is invalid: MissingKey
    

    Yeah as mentioned, we see this too. Im seeing several job related errors like: "MissingKey", "Job was canceled" , etc and its hard to know if they are all correlated or which one is the root cause to the runners getting stuck. This is why we grep and include all of these in our cron job that deletes them.
20. kornelkalnai commented on Feb 23, 2026

    

    kornelkalnai

    on Feb 23, 2026

    More actions

    I am currently using this workaround, and it seems to be working great.

    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:2.331.0
        imagePullPolicy: IfNotPresent
        # Workaround for this issue: https://github.com/actions/actions-runner-controller/issues/4307
        command:
          [
            "/bin/bash",
            "-c",
            "/home/runner/run.sh 2>&1 | tee /tmp/runner-output.log",
          ]
        livenessProbe:
          exec:
            command:
              - /bin/bash
              - -c
              - "! grep -qE '(GitHub\\.Services\\.OAuth\\.VssOAuthTokenRequestException: Registration .* was not found|Skipping message Job\\. Job message not found .*\\. job (was canceled|not found)|Skipping message Job\\. Job message already acquired)' /tmp/runner-output.log 2>/dev/null"
          initialDelaySeconds: 120
          periodSeconds: 30
          failureThreshold: 1
        env:
          - name: DOCKER_HOST
            value: unix:///var/run/docker.sock
          - name: RUNNER_WAIT_FOR_DOCKER_IN_SECONDS
            value: "120"
        volumeMounts:
          - name: work
            mountPath: /home/runner/_work
          - name: dind-sock
            mountPath: /var/run

    👀React with 👀4Reacted by Margeaux, Ryo Sakamoto, Lukasz Bandzarewicz and Zachary Spar
21. ofirmakmal commented on Apr 15, 2026

    

    ofirmakmal

    on Apr 15, 2026

    More actions

    Confirming this happens on v0.14.0 / runner v2.333.1 with a different trigger: ARC listener restart

    We hit this today and want to contribute a new triggering scenario and a subtle symptom that isn't in the thread yet.

    Trigger: A helm chart update (we modified gha-runner-scale-set values and re-applied) caused the AutoscalingListener pod to be recreated. EphemeralRunners that existed prior to the restart kept running but their OAuth registrations were invalidated server-side. They then spammed this pattern indefinitely:

    ERR POST .../_apis/oauth2/token failed. HTTP Status: BadRequest
    ERR Registration 42eb7d53-033b-420a-b726-2534c896978f was not found.
    ERR Caught exception from acquiring job message: VssOAuthTokenRequestException
    

    So the trigger isn't always a canceled/interrupted job — any listener restart (helm upgrade, pod eviction, node consolidation) can leave already-created EphemeralRunners holding stale tokens.

    Subtle symptom: The EphemeralRunner CRD's .status and the pod's .status.phase both report Running. Nothing in the Kubernetes view indicates a problem — only the runner container's stdout shows the auth loop. This makes the failure mode invisible to standard health checks and dashboards. A pod can sit in Running for hours while its runner process is a zombie.

    Downstream impact: When a real job arrives, the listener routes it to one of these zombies (GitHub sees them as valid runners), the zombie can't claim it, and the job hangs at "Waiting for a runner to pick up this job…" until someone manually `kubectl delete ephemeralrunner …`.

    Suggested fix at the controller level: When the AutoscalingListener restarts and re-registers with GitHub, the controller should reconcile existing EphemeralRunners — either re-issue registrations or mark them for replacement. Right now the controller treats them as healthy because their pods are Running.

    In the meantime, @kornelkalnai's livenessProbe workaround upthread works well; we're adopting it. Consolidated list of known error signatures to catch (from this incident and earlier posts):

    • `Registration .* was not found`
    • `Skipping message Job. Job message not found .*. job was canceled`
    • `Skipping message Job. Job message not found`
    • `Skipping message Job. Job message already acquired`

    Environment:

    • Controller: `ghcr.io/actions/gha-runner-scale-set-controller:0.14.0`
    • Runner: `2.333.1`
    • Deployment: Helm via Terraform
    • K8s: EKS 1.35
22. JWhitleyWork commented on Apr 23, 2026

    

    JWhitleyWork

    on Apr 23, 2026

    More actions

    I am seeing this issue as well but I got some stuck runners even with the work-around. Claude tells me that this is because some instances were being stopped and then restarted, which would clear their logs and keep the liveliness check from being effective. However, since I am using scale-to-zero and a new runner (at least in my cluster) almost always picks up a job immediately after being spawned, the system uptime is a good measure of how long the runner has been awake and can be compared against the job logs to determine if a job has been picked up within a reasonable time of being spawned. As such, Claude added the following check to my setting YAMLs:

    livenessProbe:
      exec:
        command:
          - /bin/bash
          - -c
          - |
            grep -qE '(GitHub\.Services\.OAuth\.VssOAuthTokenRequestException: Registration .* was not found|Skipping message Job\. Job message not found .*\. job (was canceled|not found)|Skipping message Job\. Job message already acquired)' /tmp/runner-output.log 2>/dev/null && exit 1
            if ! grep -q "Running job:" /tmp/runner-output.log 2>/dev/null; then
              uptime=$(ps -o etimes= -p 1 2>/dev/null | tr -d ' ')
              [ -n "$uptime" ] && [ "$uptime" -gt 150 ] && exit 1
            fi
            exit 0
23. USBAkimbo commented on Apr 27, 2026

    

    USBAkimbo

    on Apr 27, 2026

    Last edited by USBAkimbo

    More actions

    The liveness probe from @kornelkalnai seems to do the trick

    My arc-runners values.yml looks like this now in my AKS cluster (deployed with ArgoCD ApplicationSets)

    gha-runner-scale-set:
      githubConfigUrl: "https://github.com/my-org"
      githubConfigSecret: arc-runners-secret
    
      minRunners: 0
      maxRunners: 10
    
      runnerScaleSetName: "aks-agents"
    
      controllerServiceAccount:
        namespace: arc-controller
        name: arc-controller-gha-rs-controller
    
      template:
        spec:
          containers:
            - name: runner
              image: custom.azurecr.io/devops/gha-runner:latest
              imagePullPolicy: Always
              # Tee run.sh output to file so liveness probe can grep for stale-registration errors (#4307)
              command: ["/bin/bash", "-c", "/home/runner/run.sh 2>&1 | tee /tmp/runner-output.log"]
              livenessProbe:
                exec:
                  command:
                    - /bin/bash
                    - -c
                    - |
                      ! grep -qE '(VssOAuthTokenRequestException.*Registration.*was not found|Skipping message Job.*job (was canceled|not found)|Job message already acquired)' /tmp/runner-output.log 2>/dev/null
                initialDelaySeconds: 20
                periodSeconds: 10
                failureThreshold: 1
              resources:
                requests:
                  cpu: "10m"
                  memory: "50Mi"
                limits:
                  cpu: "4"
                  memory: "8Gi"
    
    

    There's 1 more problem though - if I ever edit the spec to change image or whatever ArgoCD syncs it and a new EphemeralRunnerSet gets created but the AutoscalingListener isn't reconciled automatically

    So I currently have to kubectl -n arc-controller delete autoscalinglistener aks-agents-*-listener after spec changes

    This same issue happens in 0.13.1 as well

    EDIT: Found a fix for the listener refresh by deploying this - just runs as an ArgoCD post-sync hook

    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: arc-listener-refresh
      namespace: arc-runners
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRole
    metadata:
      name: arc-listener-refresh
    rules:
      - apiGroups: ["actions.github.com"]
        resources: ["autoscalinglisteners"]
        verbs: ["list", "delete"]
    ---
    apiVersion: rbac.authorization.k8s.io/v1
    kind: ClusterRoleBinding
    metadata:
      name: arc-listener-refresh
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: ClusterRole
      name: arc-listener-refresh
    subjects:
      - kind: ServiceAccount
        name: arc-listener-refresh
        namespace: arc-runners
    ---
    # Runs after every ArgoCD sync. Deletes the AutoscalingListener so the controller
    # recreates it pointing at the freshly-hashed EphemeralRunnerSet. Without this,
    # any spec change produces a new EphemeralRunnerSet hash that the existing
    # AutoscalingListener doesn't know about, causing a permanent listener crash loop.
    apiVersion: batch/v1
    kind: Job
    metadata:
      name: arc-listener-refresh
      namespace: arc-runners
      annotations:
        argocd.argoproj.io/hook: PostSync
        argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
    spec:
      template:
        spec:
          serviceAccountName: arc-listener-refresh
          restartPolicy: Never
          containers:
            - name: kubectl
              image: bitnami/kubectl:latest
              command:
                - /bin/sh
                - -c
                - |
                  kubectl delete autoscalinglistener -n arc-controller \
                    -l actions.github.com/scale-set-name=aks-agents --ignore-not-found
    

    With all of this, my ARC runners now work, though sadly I can't have 1 standby agent online at all times (which I can live without)
24. jingles commented on May 21, 2026

    

    jingles

    on May 21, 2026

    More actions

    @nikola-jokic wondering if this got lost in the holiday shuffle, the logs were posted as you requested, but issue remains open. additional help in the actual code base (if appropriate) would be much appreciated!

    👍React with 👍2Reacted by Nate Wales and Sergio Compean
25. pauldraper commented on Jul 7, 2026

    

    pauldraper

    on Jul 7, 2026

    More actions

    Hit this today on controller 0.13.1 / runner 2.334.0, org-level scale sets on EKS.

    Data points that might help triage:

    • 67 of ~180 runner pods were stuck this way across two scale sets (one had 53 of 104). Each had acknowledged a job request, gotten 404 from acquirejob, logged Skipping message Job. Job message not found '...'. job was canceled, and then sat "Listening for Jobs" indefinitely — one for 18+ hours. No Runner.Worker process in any of them.
    • The cancellations clustered around a burst of PR pushes, so this looks reproducible via concurrency groups canceling queued runs after assignment but before acquire.
    • The listener never recovers the capacity: it logs Calculated target runner count {"assigned job": 49, "decision": 49, "currentRunnerCount": 49} so it won't scale up, while its own stats show only 26 registered / 24 busy at GitHub. The gap between currentRunnerCount and totalRegisteredRunners is exactly the stuck runners, and queued jobs wait even though the scale set is far below max.
    • Easy fingerprint from outside the pod: the stuck EphemeralRunners never get status.jobDisplayName patched. Deleting those CRs recovered everything within a minute.

    Stopgap we're using until there's a fix — reap by that fingerprint, no log scraping needed:

    kubectl get ephemeralrunners -n arc-runners -o json \
      | jq -r --arg cutoff "$(date -u -d '-15 minutes' +%Y-%m-%dT%H:%M:%SZ)" \
        '.items[] | select(.status.jobDisplayName == null) | select(.metadata.creationTimestamp < $cutoff) | .metadata.name' \
      | xargs -r -n10 kubectl delete ephemeralrunner -n arc-runners --wait=false

    The age cutoff avoids reaping runners that are still starting up.
26. brtkwr commented on Jul 23, 2026

    

    brtkwr

    on Jul 23, 2026

    More actions

    For anyone tracking this: actions/runner#4312 was closed as completed on 2026-07-15, but we hit this exact hang on runner v2.336.0 (about six hours after rolling it out), so the v2.336.0 cancellation fixes do not fully cover it. Filed actions/runner#4569 with the timeline and log signature. Interim mitigation that works for us: activeDeadlineSeconds: 21600 on the runner pod template (6h, matching the default job timeout-minutes, so it can never kill a legitimate job) so the kubelet reaps zombies and ARC replaces them.
27. antoinedeschenes commented on Jul 23, 2026

    

    antoinedeschenes

    on Jul 23, 2026

    More actions

    On our side we mostly solved controller stalls by cutting down the scaleset client timeout #4473
28. vnazzaro-cpu

    added 2 commits that reference this issue on Jul 27, 2026

    Force-remove EphemeralRunner registration finalizer after deletion ti…

    ...

    3af222a

    Force-remove EphemeralRunner registration finalizer after deletion ti…

    ...

    11e1a44