feat: P6 on Kubernetes, slice 2 — external P4 hosts over TLS (#424)

[pdettori]

Summary

Slice 2 of epic #426 (closes #424). P6 runs in the cluster; only the P4 microVM tier needs a KVM host. A host outside the cluster runs microvm-worker against the in-cluster relay over TLS. It connects through an L4 sidecar (ghostunnel) behind a passthrough Route. The cluster issues each host's token, and the relay reads it from a mounted Secret on every attach, so adding a host restarts nothing.

Spec: docs/specs/2026-10-04-p6-on-kubernetes-slice2-design.md. It is updated in this branch to match what was built; the corrections are listed under "Spec corrections".

Live acceptance passed on OpenShift 4.20.8 with the shared KVM rig. Results: #424 (comment)

What's in it

Relay (packages/sandbox-relay)

• Token directory: SH_RELAY_TOKEN_DIR is read on each attach, so the file <dir>/<id> takes effect without a restart. The lookup order is:
  1. the file;
  2. SH_RELAY_TOKEN_<id>;
  3. SH_RELAY_TOKEN.
• What falls through: only ENOENT falls through to the next step. Any other read error refuses the attach, and so does an empty file.
• Which IDs use the directory: only IDs matching ^[A-Za-z_][A-Za-z0-9_]*$. Others, such as the container tier's dashed moca-sandbox-N, take the environment path exactly as before.
• Reserved name: DIR is refused outright, because SH_RELAY_TOKEN_DIR sits in the SH_RELAY_TOKEN_<id> namespace (found in review).
• Exec-token guard: a directory token equal to the exec token is refused on that attach.

Workers (remote-worker)

• RELAY_CA_FILE is supported in both microvm-worker and worker, through one shared helper, session.TransportCredentials.
• The file adds to the system pool and never replaces it.
• The worker refuses to start on an unreadable file, a file with no PEM, or a CA file set without TLS.

Manifests

• base/relay.yaml: an optional mount of Secret moca-relay-sandbox-tokens at /run/relay-tokens, mode 0400.
• New OpenShift component overlays/ocp/p4-relay/, rendered only when there are P4 IDs:
  • a ghostunnel sidecar :8444 → 127.0.0.1:9443, the attach listener only, so SandboxExec is unreachable from outside. It runs with --alpn h2, which grpc-go ≥ 1.67 requires;
  • Service sandbox-relay-tls;
  • passthrough Route moca-relay with timeout: 5m;
  • NetworkPolicy sandbox-relay-from-router, admitting the router on TCP 8444 only;
  • default-container: sandbox-relay, so plain kubectl logs still reaches the relay rather than the sidecar.

deploy/k8s/setup.sh

• Inputs:
  • SH_P4_SANDBOX_IDS is sticky, and an explicitly empty value clears it.
  • IDs are trimmed, glob-safe and validated; duplicates and DIR are refused.
  • One tier per stack: setting IDs with SH_SANDBOX_COUNT > 0 is refused, naming the fix. P4 on kind is refused.
  • --relay-tls-cert/--relay-tls-key supply an operator certificate.
• Tokens:
  • One 64-hex token per ID, generated once and written off argv.
  • Revocation is an idempotent merge patch.
  • A failed read aborts the run.
  • A token is never allowed to equal the exec token.
• Relay certificate: an operator's, or self-signed (825 days), installed through a shared route_cert that the supervisor's TLS now uses too.
• Bundles: one per host, in .generated/ocp/p4/<id>/ (0700/0600), holding worker.env and, for a self-issued certificate, relay-ca.crt. setup.sh prints the exact copy and install commands.
• Portability: works on macOS bash 3.2.

deploy/microvm/setup-microvm.sh --remote <bundle>

• Validates the bundle, including RELAY_ADDR as host:port.
• Renders the worker env from the bundle; the cluster's token replaces a local one. Installs the CA (0644) and swaps 50-moca-p6.conf for 50-moca-remote.conf.
• Touches nothing of a local P6: no relay drop-in, no podman, no Redis.
• Attach check: scoped to the unit's current InvocationID, with a settle window.
  • Why the window: the worker logs attached when the stream opens, before the relay checks the token. A rejected token shows up as stream ended within milliseconds, and the check catches that (found in review).
  • Bound: the check is bounded by the wall clock.
• render_env now passes values through the environment, not awk -v argv.
• Running it again without --remote switches back to the local relay and restores the local token.

deploy/k8s/smoke.sh --tier p4 (OpenShift only)

• P2: records.
• P3: a turn's kernel is no node's kernel.
• P4: the workspace persists across turns.
• P6: a gRPC probe through the Route, from the control-plane pod, with the exec token on stdin. A wrong attach token must be refused at the relay, and exec must be UNIMPLEMENTED.
• P7: adding a scratch host must leave the relay pod unchanged, and the run restores itself.

Docs: deploy/k8s/README.md gains §11, "P4 on Kubernetes" (switch, install, verify, add/revoke and their timing, cleanup, troubleshooting). deploy/microvm/P4-ON-P6.md gets a pointer section and token-source corrections.

Spec corrections (made deliberately, all in the spec now)

• §2.1: the ID regex gates only the directory lookup. Refusing non-matching IDs would break the container tier.
• §2.4: the sidecar needs --alpn h2.
• §5: the attach check uses the InvocationID plus a settle window, not a --since timestamp. Local mode restores the token from the relay's copy.
• §6 claim 6: a refused attach ends with status OK (the relay calls stream.end()), not UNAUTHENTICATED.
• §4.5: relay-ca.crt ships whenever the certificate is self-issued.

Test plan

• Relay vitest (55), supervisor test/k8s manifest tests (60), make test, make typecheck, make lint
• go test ./... in remote-worker
• setup.test.sh, smoke.test.sh, and every deploy/microvm/tests/*.test.sh, under bash 5 and macOS /bin/bash 3.2
• Live on OpenShift 4.20.8 plus the KVM rig. See the results comment:
  • P4 smoke passed 8/8, including P7 (token reached the relay in 42 s, relay unchanged);
  • the guest kernel was 6.18.44+ vs the nodes' 5.14;
  • the Route probe returned attach=0 exec=12;
  • ALPN h2 was negotiated live;
  • an 11-minute idle stretch with a single attach;
  • two mocactl users;
  • both sides were restored, the rig binary byte-identical, and the container smoke passed 11/11 afterwards.
• One live-run defect was fixed in this branch: P3 compared the whole JSON tool-result envelope and could never fail. It now extracts the kernel line, and a regression test uses the real envelope format.

Follow-ups (not in this PR)

• Deterministic replacement for the timing-based attach-bound test in setup-microvm.test.sh, which could flake on a loaded CI host.
• A relay server keepalive, so a host whose network drops is noticed before the router's 5 m timeout.
• Running the bash suites under bash 3.2 in CI.
• More than one P4 host per stack, or both tiers together: Sandbox tier selection and session-to-sandbox affinity in selectPoolSandbox (P6-on-K8s slice 3) #425.

🤖 Generated with Claude Code

[pdettori]

Solid, thoroughly tested slice. The relay's token-directory lookup is carefully hardened (ENOENT-only fallthrough, empty-file and exec-token collision refusal, reserved DIR, constant-time compare, regex-gated path access), tokens never touch argv or child-process environments, bundles are 0700/0600, and the ghostunnel sidecar is digest-pinned with a hardening securityContext and an attach-only target. Test coverage is unusually comprehensive — real TLS handshakes in tls_test.go, argv-hygiene assertions, SSA-prune idempotency, bash 3.2 paths, and failure-injection cases for each claimed behavior.

Two nits inline. One design note, not blocking: the Route path uses --disable-authentication on ghostunnel, so L4 TLS is encryption-only and authentication rests entirely on the 64-hex constant-time-compared attach token with no rate limiting — entropy is the mitigation, which the spec accepts.

(Posted as COMMENT rather than APPROVE — GitHub does not allow approving one's own PR.)

[pdettori]

for id in ${ids//,/ } splits $ids unquoted — but here ids comes from the cluster ConfigMap (moca-setup), not from the validated SH_P4_SANDBOX_IDS input, so it's not covered by normalize_p4_ids's identifier check. A tampered/garbage ConfigMap value containing a glob would undergo pathname expansion against the CWD. Low severity (cluster-local, worst case a bogus P2 failure), but read -ra with IFS=',' would be tighter and revalidate the identifier here.

[pdettori]

Fixed in ea18150. P2 now splits the list with IFS=',' read -ra, so there is no pathname expansion, and re-checks each id against ^[A-Za-z_][A-Za-z0-9_]*$. An invalid id is a distinct FAIL (moca-setup SH_P4_SANDBOX_IDS holds invalid id(s): '*'). The expansion uses ${a[@]+…}, so it's safe on bash 3.2.

The regression test in smoke.test.sh sets MOCK_P4_IDS='moca_microvm_0,*'. Against the old code it reproduced exactly what you described: the * expanded to the repo's own filenames (buf.gen.yaml buf.yaml CLAUDE.md …), reported as missing records. It passes now under bash 5 and /bin/bash 3.2.

[pdettori]

If x509.SystemCertPool() errors, the code silently falls back to NewCertPool() — the system roots are dropped with no log line, so a worker could start trusting only the relay CA without any trace of why. Practically unreachable on Linux, but a one-line log (or folding it into the returned error) would make the failure mode visible.

[pdettori]

Fixed in 2503419. The fallback is now logged: session: WARNING the system cert pool is unavailable (<err>); RELAY_CA_FILE <path> is the only trusted CA. I kept the fallback rather than failing, because trusting the given CA is still correct. The log makes the "system roots dropped" case visible.

x509.SystemCertPool is now a package variable, so a new subtest can make it fail. The subtest asserts that the warning names the error and the file, and that a real TLS dial still succeeds with the CA file alone. It fails with the log line removed.

[pdettori]

Thanks for the review. Both inline nits are fixed: ea18150 (P2 splits the list without globbing and re-checks each id) and 2503419 (the system-pool fallback is logged). Each has a regression test that fails without its fix.

On the design note about --disable-authentication: agreed, and it's deliberate. The sidecar gives TLS for confidentiality and server identity; the client is authenticated by the per-host 64-hex attach token, checked with a constant-time, length-checked compare. The current mitigations are:

• the Route reaches only the attach listener, never SandboxExec. The live probe confirmed this: exec=12 with the real exec token;
• a refused attach ends before anything is parked;
• every host's token is unique and revocable.

If we want more than entropy, two options fit without changing the shape:

• mTLS at the sidecar: ghostunnel --allow-cn, or a client CA, with per-host client certificates issued alongside the bundle;
• relay-side rate limiting on failed attaches, per source address.

Either can be a follow-up issue if you'd like one.

[pdettori]

Opened #441 for the mTLS follow-up (client certificates at the ghostunnel sidecar, with the revocation-without-restart and cert-to-token binding questions spelled out).