Repository navigation
An error occurred: Runner not found #3857
Description
Describe the bug
Hello. I am maintaining Linux Kernel BPF CI.
We use self-hosted runners on various hardware. Recently they started failing with the following vague error: An error occurred: Runner not found. It affects all architectures that we use (x86_64, aarch64 and s390x). This then causes the runner systemd service to restart, and then the GH App quickly runs out of tokens due to frequent restarts.
Sometimes some runners are able to pick up a job when the tokens reset, but most fail with this error.
As far as I can tell, runner not found message indicates RunnerNotFoundException caught at https://github.com/actions/runner/blob/main/src/Runner.Listener/Program.cs#L148-L153
The only place it is thrown from is https://github.com/actions/runner/blob/main/src/Sdk/WebApi/WebApi/BrokerHttpClient.cs#L118-L119
Which means that github broker endpoint returns this error.
However, there is no clue what it means exactly. The runners are ephemeral with no reuse, and "replace" themselves on each run.
There were no recent changes to the self-hosted infra, or runner updates, or in the workflows.
From systemd logs on the hosts, it appears that these errors first started to appear some time in April, but in the last few days they became much more frequent to the point of blocking the testing pipeline.
To Reproduce
N/A
Expected behavior
Runners should pick up and run jobs normally.
Runner Version and Platform
v2.323.0 on Linux x86_64, aarch64 and s390x
What's not working?
Runners fail to pick up jobs after successful authentication.
Job Log Output
May 13 00:07:18 bpf-ci-runner-s390x-01 systemd[1]: Started actions-runner-kernel-patches@01.service - Ephemeral GitHub Actions Runner Container for kernel-patches - 01.
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: Runner reusage is disabled
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: Obtaining the token of the runner
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: Ephemeral option is enabled
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: Disable auto update option is enabled
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: Configuring
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: --------------------------------------------------------------------------------
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: | ____ _ _ _ _ _ _ _ _ |
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: | / ___(_) |_| | | |_ _| |__ / \ ___| |_(_) ___ _ __ ___ |
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: | | | _| | __| |_| | | | | '_ \ / _ \ / __| __| |/ _ \| '_ \/ __| |
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: | | |_| | | |_| _ | |_| | |_) | / ___ \ (__| |_| | (_) | | | \__ \ |
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: | \____|_|\__|_| |_|\__,_|_.__/ /_/ \_\___|\__|_|\___/|_| |_|___/ |
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: | |
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: | Self-hosted runner registration |
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: | |
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: --------------------------------------------------------------------------------
May 13 00:07:19 bpf-ci-runner-s390x-01 docker[5099]: # Authentication
May 13 00:07:21 bpf-ci-runner-s390x-01 docker[5099]: √ Connected to GitHub
May 13 00:07:21 bpf-ci-runner-s390x-01 docker[5099]: # Runner Registration
May 13 00:07:22 bpf-ci-runner-s390x-01 docker[5099]: A runner exists with the same name
May 13 00:07:22 bpf-ci-runner-s390x-01 docker[5099]: √ Successfully replaced the runner
May 13 00:07:22 bpf-ci-runner-s390x-01 docker[5099]: √ Runner connection is good
May 13 00:07:22 bpf-ci-runner-s390x-01 docker[5099]: # Runner settings
May 13 00:07:22 bpf-ci-runner-s390x-01 docker[5099]: √ Settings Saved.
May 13 00:07:24 bpf-ci-runner-s390x-01 docker[5099]: √ Connected to GitHub
May 13 00:07:24 bpf-ci-runner-s390x-01 docker[5099]: Current runner version: '2.323.0'
May 13 00:07:24 bpf-ci-runner-s390x-01 docker[5099]: 2025-05-13 00:07:24Z: Listening for Jobs
May 13 00:07:25 bpf-ci-runner-s390x-01 docker[5099]: An error occurred: Runner not found
May 13 00:07:25 bpf-ci-runner-s390x-01 systemd[1]: actions-runner-kernel-patches@01.service: Main process exited, code=exited, status=1/FAILURE
May 13 00:07:25 bpf-ci-runner-s390x-01 systemd[1]: actions-runner-kernel-patches@01.service: Failed with result 'exit-code'.
Runner and Worker's Diagnostic Logs
Didn't have a chance to capture these. Will add a comment when I do.
Activity
TingluoHuang commentedon May 13, 2025
Can you share more detail about your runner?
ex: what's the name of your runner and how it get setup and replaced?
kargerova commentedon May 13, 2025
Hello, we are facing the same Issue at our organization with two of three our runners.
All three runners are:
The two failing runners are ephemeral, registered via GitHub personal access token (token is valid until 2026),
started by system service running docker run
ExecStartPre=-/usr/bin/docker stop %N ExecStartPre=-/usr/bin/docker rm %N ExecStart=/usr/bin/docker run --rm \ --env-file /etc/ephemeral-github-actions-runner.env \ -e RUNNER_NAME=%H \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /tmp/runner:/tmp/runner \ --name %N \ --security-opt label=disable \ myoung34/github-runner:latestThe one functional runner is not emphemeral,
started directly by docker compose up
services: github-runner: image: myoung34/github-runner:latest container_name: github-runner restart: unless-stopped security_opt: - label:disable volumes: - /var/run/docker.sock:/var/run/docker.sock - /tmp/runner:/tmp/runner env_file: - .envThe issue occurs regardless of the runner version - with latest as with older '2.322.0'
All runners have been working properly until today. Nothing changed on our infrastructure.
Job log output
kargerova commentedon May 13, 2025
I removed the runners from organization settings and they (as they are ephemeral) registered again. Now runners are working properly.
So this is not an issue anymore, but don't know what really happened.
Hope this helps.
theihor commentedon May 13, 2025
@TingluoHuang
Very similar setup to what @kargerova described above.
All checks. Only we customize myoung34/github-runner image, see Dockerfile.
Runners are provisioned with ansible scripts. In short, a systemd service is set up to maintain runner containers:
Runner names have a format like this:
<prefix>-<host>-worker-<i>. Examples:bm-i-0a77d6f5721623905-worker-01,bpf-ci-runner-s390x-03-worker-01.When I say "replaced", I mean that normally I'd see
Runner successfully addedin the log, which I presume means that technically a new "runner" from the pov of github is created every time the container starts. Log snippet from a successful job run:theihor commentedon May 13, 2025
@kargerova I will try this, thanks for the tip.
Exit with special status code in token.sh on error
theihor commentedon May 13, 2025
For anyone interested, a working mitigation is to use unique runner names when runner container starts.
If you're using myoung34/github-runner image, you can achieve this by:
These are handled by entrypoint.sh.
Given that this helps, I suspect the reason github broker might return this error is when a runner name is reused (maybe with "unexpected" access token?).
Exit with special status code in token.sh on error
djtarazona commentedon May 20, 2025
We're experiencing this issue as well with Tartelet. Manually deleting the runner in GitHub settings fixes the issue for a while. It will eventually happen again.
patrickvinograd commentedon May 29, 2025
I believe this is the same or similar as #3819 - e.g. see this comment with the "Runner not found" error in the self-hosted runner logs.
atsu85 commentedon Jun 3, 2025
Hi
It seems to me, that it doesn't matter if you're using some fork of runner base image or the official one from this repository, because
The first Failed EphemeralRunners was created couple of hours after after we updated the runner base image!
Although I'm not using fork of the runner, I'm seeing very similar issues and logs myself as well from runner:
GChanathip commentedon Jun 3, 2025
I got the same issue using tartelet with runner version 2.324.0.
austinorth commentedon Jun 3, 2025
Same here, version 2.324.0.
mrballcb commentedon Jun 5, 2025
We were experiencing this issue where tasks were queued for 10+ minutes before executing, starting yesterday afternoon PDT. We restarted the listeners trying to mitigate, but no change. This morning we just upgraded our local runners to use 2.235.0 hoping to fix the tasks stalling problem. For about an hour, tasks started runners quickly and ran them to completion., but just now a task hung again, with its runner logging:
So it's still behaving this way for us.
8 remaining items
rentziass commentedon Jun 6, 2025
Our fix is now fully rolled out, and new runners should now stop failing (especially those that ran a job that was canceled) due to
Runner not founderrors. Please let us know if you see any repeats and thanks again for your patience ❤pulkitanz commentedon Jun 10, 2025
@rentziass @Link-
The issue of Failed Ephemeral Runners persisting is gone. They do get deleted but we are seeing a different issue for some of the runner now.
There's a lot of runner pods which are not getting deleted after being finished. The jobs on these pods have been finished but the EphemeralRunner is still stuck in the Running state. On the pod these are the logs I can see
[RUNNER 2025-06-10 03:45:46Z ERR GitHubActionsService] POST request to https://run-actions-1-azure-eastus.actions.githubusercontent.com/51/acquirejob failed. HTTP Status: NotFound
[RUNNER 2025-06-10 03:45:46Z INFO Runner] Skipping message Job. Job message not found 'e7ba9489-0333-5830-b74b-9ee48f4fe123'. job not found
Once this happens, the runner goes into an endless loop of
[RUNNER 2025-06-10 03:45:47Z INFO ConfigurationStore] HasCredentials()
[RUNNER 2025-06-10 03:45:47Z INFO ConfigurationStore] stored True
[RUNNER 2025-06-10 03:45:47Z INFO CredentialManager] GetCredentialProvider
[RUNNER 2025-06-10 03:45:47Z INFO CredentialManager] Creating type OAuth
[RUNNER 2025-06-10 03:45:47Z INFO CredentialManager] Creating credential type: OAuth
This is only happening to about 50% of the jobs. This is introducing a lot of pods which should've been cleaned up in a stuck state, are there any ongoing issues with the GitHubActionsService still?
rentziass commentedon Jun 10, 2025
@pulkitanz thanks for the report, I don't see a particular uptick in 404s for that endpoint, do you have workflow run URLs you are able to share for jobs that ran on runners that got stuck? Those would really help a ton, if you can't share them publicly can you send them to me at
rentziass at github dot complease?pulkitanz commentedon Jun 10, 2025
Apologies, @rentziass after digging more into these stuck EphemeralRunners, I can see that no jobs were picked up by the pod since startup. I will correct myself here.
On startup of the pod, It gets connected to Github then starts listening for jobs, then we see the above errors and it goes into a neverending loop. The interesting bit is that it doesn't go into a
FailingbutRunningstate.For a normal running job, which is working,
the message is something like
Job request 0 for plan 8e9b3a43-6b53-4fbe-9bf0-2a147d41ca2ae job a895ce2c-be9b-5c11-8f3f-ca7h3d12b5vd9 received.for a stuck Ephemeral Runner I am seeing
Skipping message Job. Job message not found 'e7bb9489-0233-5831-b74b-9ea48f4fe123'. job not foundtylermarien commentedon Jun 10, 2025
We are running into a lot of similar issues with this now. We are seeing 2 different errors. This is causing nearly all of our runners to fail and jobs to get backed up.
The first successfully completes the Job but then fails and causes the Pod to not shut down properly. The error looks like this:
Then there is a different error that appears to happen before the job starts and causes the Job to fail and the Pod to not shut down properly. That error looks like:
patrickvinograd commentedon Jun 11, 2025
Echoing this (I think the same as the last couple comments) - we are seeing a new failure mode where we now have a large number of runners showing in "Running" status for on the order of hours, but with no
JOBWORKFLOWREFassigned, and therefore seemingly doing no work.Looking at the logs, it also appears that we see this log message from the scale-set-controller:
before the runner is even started up:
patrickvinograd commentedon Jun 11, 2025
This is even worse than the previous failure mode because the pods stay running/consuming resources on our cluster.
pulkitanz commentedon Jun 12, 2025
The fix that I used is to create a cron job to clean up all the EphemeralRunner Resources which are stuck in a running state and have no jobRepositoryName assigned with age > 15 mins
Deleting Stuck Ephemeral Runners has helped us, we are waiting for a fix from GitHub end.
atsu85 commentedon Jun 13, 2025
I'd like to comment that
then they don't seem to cause big issues - maybe longer GHA job pickup time (but i didn't bother checking it)?
3) The script that @pulkitanz mentioned would kill also runners that are waiting idle due to
minRunnersorAutoscalingRunnerSet. The script didn't find anything useful for us even for runners that don't haveminRunnersconfiguredrentziass commentedon Jun 13, 2025
Hey everyone, sorry for the lack of updates on this one: resolving the 404 that would kill runners revealed another bug if a job was assigned to an ephemeral runner but got cancelled before the runner picked it up; the runner on its own wouldn't know it had been "used", but assignment wouldn't consider it again for other jobs. We have finished rolling out a backend fix for this yesterday (14:31 UTC) where we would delete these runners when their job completes, regardless if they started it or not.
For ARC runners this means ARC will automatically clean those runners up, while standalone runners will eventually be unable to refresh their session and shutdown. If you have any remaining runners in this state they will automatically cleaned and fall under that behaviour in the next ~24 hours.
Longer term we also want to introduce a way to gracefully tell runners they should shutdown (i.e. handle 404 without throwing as an example).
Once again sorry this impacted you and thank you for your patience ❤
blackliner commentedon Jun 24, 2025
EDIT: Confirmed, it was a misconfigured autoscaler lambda that deleted idle runners.
I just lost a fleet of 18 onprem runners toAn error occurred: Runner not found. bare metal, non-ephemeral. Can this be related to what has been discussed in this issue?I read somewhere that when you have very CPU intensive workloads that steal CPU time from the runner process, after a certain time the runner will be marked as stale from the server (github.com) side and won't be able to reconnect. Could this explain some of these occurences?UPDATE:These runners don't even last 15 minutes!
patrickvinograd commentedon Jun 26, 2025
Hi @rentziass , things seemed better a couple weeks ago, but this week we are seeing a ton of self-hosted runners hanging out in the
Runningstate but apparently not doing anything. There's a mix of those with a workflow assigned and those without.We have introduced various workarounds to prune runners, those without workflows assigned are straightforward but those without we have to exclude various ETL jobs, it's definitely not optimal. Any insights of what's going on on the backend?
ansible: disable runner name randomization
ansible: disable runner name randomization
denysvitali-kaiko commentedon Dec 9, 2025
Hey @rentziass!
Is it possible that there is still a backend bug about this?
We're randomly seeing listeners not receiving the message from the GH backend, see actions/actions-runner-controller#3953 (comment)