Repository navigation
devspace: Log in over SSH as an unprivileged runner-sandbox user - #16
Open
cgwalters-bot wants to merge 3 commits into
Open
cgwalters-bot wants to merge 3 commits into
cgwalters-bot wants to merge 3 commits into
Conversation
A pinned SHA is a hash someone has to bump by hand and read past in review. The shared bootc-dev Renovate config (config:recommended, no digest pinning) keeps major tags current instead, the way the rest of bootc-dev references actions. The checkout test now only checks that it's a major tag. This is also a version bump: checkout goes from v4.2.2 to v7, and the Tailscale action from v4.1.2 to the current v4 (4.2.0). Generated-by: AI Signed-off-by: Colin Walters <walters@verbum.org>
Prep for SSH sessions losing sudo, after which they can't install packages themselves. Until now the bot's bot-devspace installed this list over SSH after every start; having the workflow do it also means every devspace, not only the bot's, gets the same toolchain. python3-virt-firmware (for bcvk's Secure Boot key enrollment) and python3-pip are new in the list. bcvk itself is left out: EPEL has 0.10.0 while bootc CI pins 0.19.0, and bot-devspace provision installs that release into the user's ~/.local/bin, which needs no sudo. Generated-by: AI Signed-off-by: Colin Walters <walters@verbum.org>
Everything on a devspace ran as runner: the SSH sessions, and so every build script or test they start, as well as the workflow steps. The running keepalive step's environment holds ACTIONS_ID_TOKEN_REQUEST_URL and ACTIONS_ID_TOKEN_REQUEST_TOKEN (the job has id-token: write for the Tailscale login), readable from /proc/PID/environ by any process of the same uid (ptrace_scope is 0 too), so untrusted code run over SSH could mint GitHub OIDC tokens for this workflow, which the Tailscale OAuth client trusts with writable auth_keys. With passwordless sudo, no narrower fix inside the runner uid holds. So sessions now log in as a separate user, runner-sandbox, with no sudo. Being another uid is what keeps it out of the steps' /proc entries. The name isn't agent-specific on purpose: the same unprivileged user may well be useful for deterministic CI too. Otherwise the stock runner setup stays. A few exposures do need fixing: the runner's .credentials and the hosted compute agent's token in /opt/hca were world-readable, and system files root loads code from were world-writable (runner-sandbox could add a polkit rule granting itself root). tailscaled's LocalAPI socket is 0666 as well, and its dial call has no permission check, so any user could reach every tailnet node the runner's tag may through it (tailscale nc) and list the whole tailnet; /var/run/tailscale becomes root's only. The image's /etc/environment also sets runner's XDG_RUNTIME_DIR for every PAM session, which breaks runner-sandbox's session bus and rootless podman; that one line is removed, and should really be fixed in the runner image. And ssh refuses to run with the group-writable crypto-policies config it includes, so that one file loses its group write permission. runner-sandbox is in no sudoers rule or admin group anyway, but the two obvious ways to root are closed explicitly: a sudoers rule denies it everything, and a polkit rule (when polkit is installed) says no to all its requests, pkexec included. Undoing more of the image's setup (such as stripping every setuid bit) is left out on purpose; it turns into a pile of hacks against the runner image. The uid separation is the boundary. As defense in depth, the long-lived keepalive step (the one SSH sessions live under) re-executes itself without the request token, since GitHub can't make it single-use but the Tailscale login is its only use (tailscaled, started by the action with sudo -E, keeps it in its environment, which only root can read). ptrace_scope goes to 1, which stops runner's own processes from attaching to each other, though not from reading each other's /proc/PID/environ. Cockpit is turned off, since it listened on the tailnet, and runner-sandbox can't reach the instance metadata service. What's left is anything that runs as runner or root during the job (the runner itself, the actions it uses, package scriptlets), and root processes already running (tailscaled, sshd). runner-sandbox gets subordinate IDs for rootless podman and the kvm group, but not libvirt: its polkit rule grants all of qemu:///system, which is root-equivalent. Rootless VMs (bcvk ephemeral) cover the tests that need root. The authorized key moves to a root-owned file under /etc/ssh, so a session can't add keys, and the runner's home no longer needs to be traversable. scripts/runner-sandbox.mjs is the one place that knows the user and how a step runs something as it: run0, so it gets a login session of its own like an SSH login does (XDG_RUNTIME_DIR, the user's systemd manager, so rootless podman uses systemd cgroups), with a fixed environment and outside the step's cgroup. machinectl shell gives the same session, but always through a pty: the exit status is lost, stderr is merged into stdout, and with stdin a pipe nothing got through, which scripted use and an agent speaking a protocol on stdio can't live with. The setup script uses it to install the bot's dotfiles (just init), and agent runs will use it too. Generated-by: AI Signed-off-by: Colin Walters <walters@verbum.org>
This was referenced Oct 22, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SSH sessions on a devspace ran as
runner, the same uid (with passwordless sudo) as the workflow steps. The running keepalive step's/proc/PID/environholdsACTIONS_ID_TOKEN_REQUEST_TOKEN/_URL, so any build script started over SSH could mint GitHub OIDC tokens that the Tailscale OAuth client trusts.runner-sandbox, with no sudo. Another uid can't readRunner.Worker's or any step's environ or memory.scripts/setup-runner-sandbox.mjs(Node) creates it (kvm group and subuids; not libvirt, whoseqemu:///systemaccess is root-equivalent). It also fixes the real exposures of the umask-000 image:/home/runnerand/opt/hcabecome private, since the runner's.credentialsand the compute agent's token were world-readable;runner-sandboxcould otherwise add a polkit rule granting itself root;/etc/environmentloses runner'sXDG_RUNTIME_DIR(XDG_RUNTIME_DIR is set system-wide in /etc/environment, breaking sessions of other users actions/runner-images#14649), so logins get their own runtime directory;runner-sandbox ALL=(ALL:ALL) !ALL) makes that explicit, and when polkit is installed a rule answers no to all its requests,pkexecincluded.ACTIONS_ID_TOKEN_REQUEST_*andACTIONS_RUNTIME_TOKEN.tailscaled, started by the action withsudo -E, still holds the token, readable only by root.ptrace_scope=1, Cockpit's socket disabled, and no instance metadata service forrunner-sandboxor its rootless containers.The toolchain moves into the workflow (
packages.txt), since sessions can't install packages.scripts/runner-sandbox.mjsis the single definition of the user and of how a step runs something as it (run0, so it gets its own login session like an SSH login's, with a fixed environment);setup-runner-sandbox.mjs --init Justfileuses it for the dotfiles, and #3's agent.yml runs the same setup script. A first commit references the workflow's actions by major tag (checkout v7, tailscale v4) instead of SHAs, for Renovate.What's left: anything running as
runneror root during setup (the runner, the actions it uses, package scriptlets), and root processes that are already running (tailscaled, sshd).NoNewPrivilegesfor the runner itself isn't possible from inside a job, becauseRunner.Listener/Workerrun under the image'shosted-compute-agent.serviceand restarting that would end the job. Narrowing the Tailscale federated identity's claims and keepingtag:bootc-dev-sandbox's ACL tight limit what a minted token is worth.Testing, at this head (0edf5c1), on a 16-core RHEL 10 devspace started from main (run 36275733071; main predates runner-sandbox, so this branch was pushed there):
cargo fmt --check,clippy --all-targets -D warningsandcargo test(15 passed);actionlintandnode --checklocally.setup-runner-sandbox.mjsran twice cleanly. Then runner-sandbox logged in over SSH, through a PAM sshd with this workflow's auth settings (on 127.0.0.1:2222, as its own unit, so the login got a systemd session). Theresudo -n trueandsudo -lfail,pkexecsays "Not authorized",systemctl restart sshdgets "Access denied" (without the polkit rule, an earlier run showed it asking for interactive authentication instead), andsuto root or runner fails authentication (both passwords are locked in the image, so nothing to lock). Rootless podman ran a container,bcvk ephemeral run-sshbooted centos-bootc:stream10, and runner keeps its sudo. Script and logs: https://gist.github.com/cgwalters-bot/619656d679aad001f51cfa7b806942d1-xdevsweep,/run,/dev/shmand/mnthave no world-writable non-sticky files other than sockets./mntis empty, and there is no swap on this runner size.setup-runner-sandbox.mjs --init Justfile(runner-sandbox's dotfiles) and Sync common files from infra repository #3's isolation check (12 checks) passed at 558171a (run 36240484839). The token scrub and sshd admitting only runner-sandbox were covered by run 36206810916. None of these changed here./var/run/tailscaleroot's only, since tailscaled's LocalAPI socket is 0666 and lets any usertailscale ncthrough the node andtailscale statusthe tailnet; the keepalive now runssudo tailscale status --self):cargo test(15 passed) on a 16-core devspace (run 36415814928). There, with the directory at the image's 0755,tailscale nc <tailnet host> 22as runner-sandbox connected andtailscale statuslisted the tailnet; after the setup script both fail with permission denied, andsudo tailscale statusstill works for runner.runner-sandbox.mjsusesrun0instead of plainsystemd-run, per review): on a 16-core RHEL 10 devspace (run 36487000792),setup-runner-sandbox.mjs --init Justfilefor a freshly created user, Sync common files from infra repository #3's isolation check (15 ok, applied on Sync common files from infra repository #3's head), rootlesspodman runthroughasSandbox(systemd cgroup manager, runroot in/run/user/UID), exit statuses passed through, andcargo test(15 passed). machinectl shell, run0 and systemd-run compared in a bcvk CentOS Stream 10 VM and on the runner: https://gist.github.com/cgwalters-bot/f932edce0622dba38e1b9018c01c6c31env -udropping theSUDO_USER/SUDO_UID/SUDO_GIDrun0 sets, andCollectMode=inactive-or-failed, since run0 otherwise leaves a failed unit for every nonzero exit): on a 16-core RHEL 10 devspace (run 36488428943),setup-runner-sandbox.mjs --init Justfilefor a fresh user,asSandboxenv has noSUDO_*, rootlesspodman run(systemd cgroups, runroot in/run/user/1002, exit 5 passed through), no failed units left,cargo fmt --check/clippy -D warnings/cargo test(15 passed). Sync common files from infra repository #3's and devspace: Initialize runners for cgwalters-bot sessions with pinned agent CLIs #5's checks at their restacked heads are in those PRs.homegit's bot-devspace now logs in as
runner-sandboxand falls back torunner(cgwalters-bot/homegit#7).The
Signed-off-by: Colin Walters <walters@verbum.org>on these commits was added on cgwalters's approval of the review draft: cgwalters-forge#2 (review)Generated-by: https://github.com/cgwalters/#llms