Repository navigation
EphemeralRunner left stuck Running after node drain/pod termination #4148
Description
Checks
- I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
- I am using charts that are officially provided
Controller Version
0.12.0
Deployment Method
Helm
Checks
- This isn't a question or user support case (For Q&A and community support, go to Discussions).
- I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
To Reproduce
1. Start a long-running GHA job
2. Run `kubectl drain <node-name>` on the EKS node running the pod for the allocated EphemeralRunner. (Directly deleting the runner pod with `kubectl delete pod <pod-name>` also has the same effect, but isn't what we normally do/experience.)
3. Observe the Runner in GHE list of active runners goes away
4. Observe the EphemeralRunner in K8s stays in `Running` state for everDescribe the bug
While the Runner (as recorded by the GitHub Actions list of org-attached Runners in the Settings page) goes away, the EphemeralRunner stays allocated forever.
This messes with AutoScalingRunnerSets thinking it doesn't need to scale up any more & we observe long wait times for new Runners to be allocated to Jobs. Until this is fixed we have to delete the ERs manually with a script like the below:
#!/usr/bin/env bash
set -euo pipefail
STUCK_RUNNERS=$(kubectl get ephemeralrunners -n gha-runner-scale-set -o json \
| jq -r '.items[] | select(.status.phase == "Running" and .status.ready == false and .status.jobRepositoryName != null) | .metadata.name' \
| tr '\n' ' ')
if [ -z "$STUCK_RUNNERS" ]; then
echo "No stuck EphemeralRunners."
exit 0
fi
echo "Deleting: $STUCK_RUNNERS"
kubectl delete ephemeralrunners -n gha-runner-scale-set $STUCK_RUNNERSDescribe the expected behavior
For the ARC to at least notice the Runner has disappeared & to delete the stuck EphemeralRunner automatically.
The best solution would be for the ARC to resubmit the job for a re-run if it sees this condition, or at least emit an specific K8s event such that we could add such automation on top easily via a custom watcher.
Additional Context
Here's the redacted YAML for the ER itself, complete with status: https://gist.github.com/tyrken/7810a7c51739511585abcced460176ab#file-x86-64-h8696-runner-jzgnk-yaml
Controller Logs
See https://gist.github.com/tyrken/7810a7c51739511585abcced460176ab#file-controller-logs-txt - the node was drained around 19:11 UTC.
See also the listener logs if of any interest: https://gist.github.com/tyrken/7810a7c51739511585abcced460176ab#file-listener-logs-txt
Runner Pod Logs
See https://gist.github.com/tyrken/7810a7c51739511585abcced460176ab#file-runner-logs-tsv
(Note copied from logging server as runner/pod is deleted during bug reproduction)
Activity
[-]EphemeralRunner left Running after node drain led to pod termination[/-][+]EphemeralRunner left stuck Running after node drain/pod termination[/+]luislongom commentedon Jun 27, 2025
@nikola-jokic - does this PR also fix the problem stated here? Thanks
kennedy-whytech commentedon Jun 27, 2025
Same question as @luislongom
I haven't tried all scenario. With 0.12.1, I removed the runner pod manually and the ephemeralrunners were removed along with it afterwards. 👍
kennedy-whytech commentedon Jun 27, 2025
Created a separate issue here.
With 0.12.1:
However, when a container is OOMKILL, the pod just hangs there.
This might be even worse since it shows that it's healthy.
Here's the status
luislongom commentedon Jun 30, 2025
@kennedy-whytech - were you able to reproduce the same issue with 0.12.0? Thanks
kennedy-whytech commentedon Jun 30, 2025
@luislongom In version 0.12.0, the issue appears slightly different — the pod was permanently killed, but the EphemeralRunners were left behind.
In version 0.12.1, the pod remains in a “healthy” state, but we still see those BrokerMessageListener messages and not able to pick up new jobs.
I’ve also seen similar behavior after Karpenter drain nodes and reschedule the pods- but not everytime
alexanderkranga commentedon Jul 2, 2025
We are experiencing the same issue on 0.12.1
We use Karpenter but we think it does not cause this issue. Our pods have enough ram and are never killed because of OOM.
Also, the stuck EphemeralRunners are pointing to workflow files in github, and after verification of those workflows, we saw no recent failing jobs, so there must be something else that causes EphemeralRunners to stuck in running
It is hard to reproduce, I will update this comment if I find something else
EDIT: correction, we had this issue on 0.12.0, not 0.12.1. We have upgraded all our clusters (we have 10+ ARC deployments) to 0.12.1 yesterday and saw no runners failing this way.
There was a fix merged into 0.12.1 that fixed this issue
Thank you!
Tal-E commentedon Jul 7, 2025
Hi, I still face this issue even on version v0.12.1
rajesh-dhakad commentedon Jul 11, 2025
I see the same issue with version v0.12.1 : listed down here: #4168 (comment)
ArdiannS commentedon Jul 21, 2025
Hi,
We're also experiencing an issue with actions-runner-controller version 0.12.1. We're using Karpenter for autoscaling nodes, but this does not appear to be an autoscaler-related problem.
Scenario Description
When a new GitHub Actions workflow is triggered, an ephemeral runner pod is correctly created and begins executing the job. However, if the node is deleted (either manually or by Karpenter), the following behavior occurs:
The runner pod is terminated, and the GitHub runner is correctly deregistered.
However, the corresponding EphemeralRunner resource remains in Running state.
A new runner pod is then continuously spawned with the same name on a different node.
Even though the GitHub workflow fails, the ephemeral runner continues to respawn indefinitely.
Additional Observations
The GitHub job being executed is simply sleep 5000, so this is not caused by OOMKilled.
We're allocating 5 vCPUs and 17 GB of memory to both the runner and DinD containers.
After node deletion, the original runner pod briefly enters a Completed state.
The GitHub UI shows the job has failed, and there are no active workflow runs, yet the runner continues to respawn.
After Node Deletion:
Runner pod goes to Completed
EphemeralRunner remains in Running
Node deletion proceeds normally
This leads to orphaned and perpetually recreated runner pods, even though no jobs are pending.
Thank you!
WZHRRTH commentedon Dec 17, 2025
Hi,
Is there any update on this?
nikola-jokic commentedon Dec 17, 2025
Hey everyone, we are aiming to release☺️
0.13.1soon and #4340 should fix this scenario, based on testing. If anyone has any concerns with it, feel free to raise it before we merge it and issue a next release. I am confident it should fix it, but I want to make sure everything is covered just in case