Track job assignment in the cluster-autoscaler safe-to-evict annotation

[jaimeph]

Problem

Idle runners kept around by minRunners sit on a node doing nothing, and that node never scales down.

The cluster autoscaler will not drain a node that hosts a pod with local storage unless the pod declares itself evictable through cluster-autoscaler.kubernetes.io/safe-to-evict: "true". Runner pods mount emptyDir volumes (work, dind-sock, dind-externals, see _helpers.tpl) and carry no such annotation, so the autoscaler treats every runner as non-evictable, busy or not.

Hardcoding safe-to-evict: "true" in the runner template is not a fix: the autoscaler would then be free to evict a runner in the middle of a job.

Change

The controller manages the annotation on the runner pod, following job assignment:

Runner state	Annotation
Idle (no Status.JobID)	"true"
Job assigned by the listener	"false"
Template sets it explicitly	left untouched

This is opt-in. Nothing changes on upgrade until it is turned on:

# gha-runner-scale-set-controller values.yaml
flags:
  manageSafeToEvictAnnotation: true

Implementation notes

• safeToEvictFor() derives the value and decides whether the controller owns the annotation at all. If the EphemeralRunner or its PodTemplateSpec already carries it, the function reports "not managed" and the controller keeps its hands off — an explicit value is a deliberate choice by whoever wrote the template.
• The annotation is applied after the pod is built, so it stays out of the pod-template-hash. Its value changes during the pod's life and must not register as template drift.
• The transition is one-way: Status.JobID is written once and never cleared, so the value goes from "true" to "false" and stays there; the pod dies with the job it ran. The reconcile is kept idempotent regardless, so it converges on its own after a controller restart.
• reconcileSafeToEvictAnnotation() patches only when the value differs from what is on the pod. A runner that never picks up a job costs a single write, the one at pod creation.
• No new watch is needed: EphemeralRunnerReconciler already reconciles on every EphemeralRunner change, and the listener's write of Status.JobID is one of them. The pod patch does not feed back into the loop because ephemeralRunnerOwnedPodPredicate drops updates that only touch metadata.
• Pods with a deletionTimestamp are left alone: the annotation only matters while the pod is a real eviction candidate.
• The annotation is reconciled off the EphemeralRunner, not off the pod. Removing it from a running pod does not wake the controller by itself, since ephemeralRunnerOwnedPodPredicate drops pod updates that only touch metadata. A pod that already existed when the flag was turned on is repaired by the next reconcile of its runner, which a controller restart produces for every runner it owns.

Scope: what this does not cover

This addresses cluster autoscaler scale-down only. It does not protect a runner with a job in flight from:

• a manual kubectl drain
• managed node auto-upgrade (AKS/EKS/GKE) and provider-side cordon+drain
• Karpenter consolidation (karpenter.sh/do-not-disrupt has the same limitation)

Those paths go through the Eviction API, which honors PodDisruptionBudget and nothing else. safe-to-evict is read only by the cluster autoscaler when it decides to remove a node.

PDBs do not currently work on ARC v2 either: the runner pod's direct owner is the EphemeralRunner CR, which declares only the status subresource (apis/actions.github.com/v1alpha1/ephemeralrunner_types.go:29). Without a scale subresource the disruption controller cannot resolve the budget and the PDB ends up SyncFailed. Closing that gap means giving EphemeralRunner a scale subresource or reparenting runner pods to a set-level owner, which is an architectural change and deliberately out of scope here. See #4618 for the full gap and #4493 for the PDB discussion.

Tests

• ephemeralrunner_safe_to_evict_test.go: 5 cases on value derivation (flag off, idle, with a job, explicit annotation on the runner, explicit annotation on the template) and 5 on the reconcile (annotates an idle runner, flips to false when the job arrives, does not patch when already at the desired value, does nothing with the flag off, skips a terminating pod).

• TestTemplate_ControllerDeployment_ManageSafeToEvictAnnotation: the flag is rendered only with manageSafeToEvictAnnotation: true.

• ephemeralrunner_controller_test.go: two envtest specs driving the real reconcile against an API server. One follows an idle runner to "true" and then to "false" on job assignment; the other strips the annotation off a live pod and checks the next reconcile restores it. Both verified to fail when the call to reconcileSafeToEvictAnnotation is removed.

• TestTemplate_CreateManagerRole: pins the pod verbs on the Role created in the runner namespace, including patch.

go build ./... and go vet ./... clean.

Operational note

There is a narrow window between the listener writing JobID and the controller setting the annotation to false. The autoscaler's scale-down-unneeded-time (10 minutes by default) covers it comfortably in practice, but it is not zero.

RBAC

The Role created in the runner namespace gains patch on pods. It previously granted create, delete and get, which was enough because nothing patched a runner pod until now.

This is worth calling out because of how it fails: the chart installs, Helm reports the release as deployed, the flag shows up in the controller args, and pods created after the rollout do carry the annotation, since that value goes into the pod spec before Create. Only the later patch is forbidden, so pods that already existed never get annotated and the true -> false flip never lands. The failure is visible only in the controller logs.

Field results

Deployed on a production CI cluster running ~100 runners across three scale sets.

The annotation converged on all 106 pre-existing runner pods within 3 minutes of the rollout, and followed job assignment from then on: on one node, six runners flipped to "false" within 30 seconds as they picked up jobs, and back to "true" as they finished.

A node that had been up for 3h15m holding a single idle runner was then retired by the autoscaler. Its last runners went idle at 16:46, the autoscaler marked it DeletionCandidateOfClusterAutoscaler immediately and ToBeDeletedByClusterAutoscaler two minutes later. Both runners still on the node at that point were idle with an empty JobID; no in-flight job was disrupted. Runner pods were not recycled by the upgrade, since the change does not touch the pod template.

Related to #2562.