Skip to content

devspace: Log in over SSH as an unprivileged runner-sandbox user - #16

Open
cgwalters-bot wants to merge 3 commits into
bootc-dev:mainfrom
cgwalters-forge:bot/unprivileged-ssh-user
Open

cgwalters-bot wants to merge 3 commits into
bootc-dev:mainfrom
cgwalters-forge:bot/unprivileged-ssh-user

Conversation

@cgwalters-bot

Copy link
Copy Markdown
Collaborator

SSH sessions on a devspace ran as runner, the same uid (with passwordless sudo) as the workflow steps. The running keepalive step's /proc/PID/environ holds ACTIONS_ID_TOKEN_REQUEST_TOKEN/_URL, so any build script started over SSH could mint GitHub OIDC tokens that the Tailscale OAuth client trusts.

  • Separate uid (the boundary). Sessions log in as a new user, runner-sandbox, with no sudo. Another uid can't read Runner.Worker's or any step's environ or memory. scripts/setup-runner-sandbox.mjs (Node) creates it (kvm group and subuids; not libvirt, whose qemu:///system access is root-equivalent). It also fixes the real exposures of the umask-000 image:
  • No sudo or polkit for runner-sandbox. It is in no sudoers rule or admin group anyway; a sudoers rule (runner-sandbox ALL=(ALL:ALL) !ALL) makes that explicit, and when polkit is installed a rule answers no to all its requests, pkexec included.
  • Use the token once. The keepalive step, which lives as long as the devspace, re-executes itself without ACTIONS_ID_TOKEN_REQUEST_* and ACTIONS_RUNTIME_TOKEN. tailscaled, started by the action with sudo -E, still holds the token, readable only by root.
  • Hardening on top: ptrace_scope=1, Cockpit's socket disabled, and no instance metadata service for runner-sandbox or its rootless containers.

The toolchain moves into the workflow (packages.txt), since sessions can't install packages. scripts/runner-sandbox.mjs is the single definition of the user and of how a step runs something as it (run0, so it gets its own login session like an SSH login's, with a fixed environment); setup-runner-sandbox.mjs --init Justfile uses it for the dotfiles, and #3's agent.yml runs the same setup script. A first commit references the workflow's actions by major tag (checkout v7, tailscale v4) instead of SHAs, for Renovate.

What's left: anything running as runner or root during setup (the runner, the actions it uses, package scriptlets), and root processes that are already running (tailscaled, sshd). NoNewPrivileges for the runner itself isn't possible from inside a job, because Runner.Listener/Worker run under the image's hosted-compute-agent.service and restarting that would end the job. Narrowing the Tailscale federated identity's claims and keeping tag:bootc-dev-sandbox's ACL tight limit what a minted token is worth.

Testing, at this head (0edf5c1), on a 16-core RHEL 10 devspace started from main (run 36275733071; main predates runner-sandbox, so this branch was pushed there):

  • cargo fmt --check, clippy --all-targets -D warnings and cargo test (15 passed); actionlint and node --check locally.
  • setup-runner-sandbox.mjs ran twice cleanly. Then runner-sandbox logged in over SSH, through a PAM sshd with this workflow's auth settings (on 127.0.0.1:2222, as its own unit, so the login got a systemd session). There sudo -n true and sudo -l fail, pkexec says "Not authorized", systemctl restart sshd gets "Access denied" (without the polkit rule, an earlier run showed it asking for interactive authentication instead), and su to root or runner fails authentication (both passwords are locked in the image, so nothing to lock). Rootless podman ran a container, bcvk ephemeral run-ssh booted centos-bootc:stream10, and runner keeps its sudo. Script and logs: https://gist.github.com/cgwalters-bot/619656d679aad001f51cfa7b806942d1
  • Outside the setup's -xdev sweep, /run, /dev/shm and /mnt have no world-writable non-sticky files other than sockets. /mnt is empty, and there is no swap on this runner size.
  • setup-runner-sandbox.mjs --init Justfile (runner-sandbox's dotfiles) and Sync common files from infra repository #3's isolation check (12 checks) passed at 558171a (run 36240484839). The token scrub and sshd admitting only runner-sandbox were covered by run 36206810916. None of these changed here.
  • At 619398e (this head; the change since 0edf5c1 makes /var/run/tailscale root's only, since tailscaled's LocalAPI socket is 0666 and lets any user tailscale nc through the node and tailscale status the tailnet; the keepalive now runs sudo tailscale status --self): cargo test (15 passed) on a 16-core devspace (run 36415814928). There, with the directory at the image's 0755, tailscale nc <tailnet host> 22 as runner-sandbox connected and tailscale status listed the tailnet; after the setup script both fail with permission denied, and sudo tailscale status still works for runner.
  • At 391d342 (runner-sandbox.mjs uses run0 instead of plain systemd-run, per review): on a 16-core RHEL 10 devspace (run 36487000792), setup-runner-sandbox.mjs --init Justfile for a freshly created user, Sync common files from infra repository #3's isolation check (15 ok, applied on Sync common files from infra repository #3's head), rootless podman run through asSandbox (systemd cgroup manager, runroot in /run/user/UID), exit statuses passed through, and cargo test (15 passed). machinectl shell, run0 and systemd-run compared in a bcvk CentOS Stream 10 VM and on the runner: https://gist.github.com/cgwalters-bot/f932edce0622dba38e1b9018c01c6c31
  • At 7cb2947 (this head; the same plus env -u dropping the SUDO_USER/SUDO_UID/SUDO_GID run0 sets, and CollectMode=inactive-or-failed, since run0 otherwise leaves a failed unit for every nonzero exit): on a 16-core RHEL 10 devspace (run 36488428943), setup-runner-sandbox.mjs --init Justfile for a fresh user, asSandbox env has no SUDO_*, rootless podman run (systemd cgroups, runroot in /run/user/1002, exit 5 passed through), no failed units left, cargo fmt --check/clippy -D warnings/cargo test (15 passed). Sync common files from infra repository #3's and devspace: Initialize runners for cgwalters-bot sessions with pinned agent CLIs #5's checks at their restacked heads are in those PRs.
  • devspace.yml itself (including the Tailscale action at v4 and checkout at v7) can only run from the default branch, so it is untested as a whole until merged.

homegit's bot-devspace now logs in as runner-sandbox and falls back to runner (cgwalters-bot/homegit#7).

The Signed-off-by: Colin Walters <walters@verbum.org> on these commits was added on cgwalters's approval of the review draft: cgwalters-forge#2 (review)

Generated-by: https://github.com/cgwalters/#llms

A pinned SHA is a hash someone has to bump by hand and read past in
review. The shared bootc-dev Renovate config (config:recommended, no
digest pinning) keeps major tags current instead, the way the rest of
bootc-dev references actions. The checkout test now only checks that
it's a major tag.

This is also a version bump: checkout goes from v4.2.2 to v7, and the
Tailscale action from v4.1.2 to the current v4 (4.2.0).

Generated-by: AI
Signed-off-by: Colin Walters <walters@verbum.org>
Prep for SSH sessions losing sudo, after which they can't install
packages themselves. Until now the bot's bot-devspace installed this
list over SSH after every start; having the workflow do it also means
every devspace, not only the bot's, gets the same toolchain.

python3-virt-firmware (for bcvk's Secure Boot key enrollment) and
python3-pip are new in the list. bcvk itself is left out: EPEL has
0.10.0 while bootc CI pins 0.19.0, and bot-devspace provision installs
that release into the user's ~/.local/bin, which needs no sudo.

Generated-by: AI
Signed-off-by: Colin Walters <walters@verbum.org>
Everything on a devspace ran as runner: the SSH sessions, and so every
build script or test they start, as well as the workflow steps. The
running keepalive step's environment holds ACTIONS_ID_TOKEN_REQUEST_URL
and ACTIONS_ID_TOKEN_REQUEST_TOKEN (the job has id-token: write for the
Tailscale login), readable from /proc/PID/environ by any process of the
same uid (ptrace_scope is 0 too), so untrusted code run over SSH could
mint GitHub OIDC tokens for this workflow, which the Tailscale OAuth
client trusts with writable auth_keys. With passwordless sudo, no
narrower fix inside the runner uid holds.

So sessions now log in as a separate user, runner-sandbox, with no sudo.
Being another uid is what keeps it out of the steps' /proc entries. The
name isn't agent-specific on purpose: the same unprivileged user may
well be useful for deterministic CI too. Otherwise the stock runner
setup stays. A few exposures do need fixing: the runner's .credentials
and the hosted compute agent's token in /opt/hca were world-readable,
and system files root loads code from were world-writable
(runner-sandbox could add a polkit rule granting itself root).
tailscaled's LocalAPI socket is 0666 as well, and its dial call has no
permission check, so any user could reach every tailnet node the
runner's tag may through it (tailscale nc) and list the whole tailnet;
/var/run/tailscale becomes root's only. The image's /etc/environment
also sets runner's XDG_RUNTIME_DIR for every PAM session, which breaks
runner-sandbox's session bus and rootless podman; that one line is
removed, and should really be fixed in the runner image. And ssh refuses
to run with the group-writable crypto-policies config it includes, so
that one file loses its group write permission.

runner-sandbox is in no sudoers rule or admin group anyway, but the two
obvious ways to root are closed explicitly: a sudoers rule denies it
everything, and a polkit rule (when polkit is installed) says no to all
its requests, pkexec included. Undoing more of the image's setup (such
as stripping every setuid bit) is left out on purpose; it turns into a
pile of hacks against the runner image.

The uid separation is the boundary. As defense in depth, the
long-lived keepalive step (the one SSH sessions live under)
re-executes itself without the request token, since GitHub can't make
it single-use but the Tailscale login is its only use (tailscaled,
started by the action with sudo -E, keeps it in its environment, which
only root can read). ptrace_scope goes to 1, which stops runner's own
processes from attaching to each other, though not from reading each
other's /proc/PID/environ. Cockpit is turned off, since it listened on
the tailnet, and runner-sandbox can't reach the instance metadata
service. What's left is anything that runs as runner or root during the
job (the runner itself, the actions it uses, package scriptlets), and
root processes already running (tailscaled, sshd).

runner-sandbox gets subordinate IDs for rootless podman and the kvm
group, but not libvirt: its polkit rule grants all of qemu:///system,
which is root-equivalent. Rootless VMs (bcvk ephemeral) cover the tests
that need root. The authorized key moves to a root-owned file under
/etc/ssh, so a session can't add keys, and the runner's home no longer
needs to be traversable.

scripts/runner-sandbox.mjs is the one place that knows the user and how
a step runs something as it: run0, so it gets a login session of its own
like an SSH login does (XDG_RUNTIME_DIR, the user's systemd manager, so
rootless podman uses systemd cgroups), with a fixed environment and
outside the step's cgroup. machinectl shell gives the same session, but
always through a pty: the exit status is lost, stderr is merged into
stdout, and with stdin a pipe nothing got through, which scripted use
and an agent speaking a protocol on stdio can't live with. The setup script uses it
to install the bot's dotfiles (just init), and agent runs will use it
too.

Generated-by: AI
Signed-off-by: Colin Walters <walters@verbum.org>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant