feat: sandbox tiers and session-to-sandbox affinity — harness, relay, workers (P6.3 PR 1, #425)

[pdettori]

Summary

PR 1 of 3 for #425 (P6.3, sandbox tiers and session-to-sandbox affinity).

• Workers advertise their sandbox tier as the moca.dev/tier presence label.
• The relay records when each sandbox leaves (its detach mark).
• selectPoolSandbox keeps a session in its tier and on the sandbox that served its previous turn. It answers 503 while that sandbox is busy or briefly gone. Past a grace period, it moves the session within its tier and records a workspace reset.
• A turn reports where it ran, and streams a workspace_reset frame when the session lands on a fresh workspace. mocactl displays it.

Spec: docs/specs/2026-10-04-p6-on-kubernetes-slice3-design.md (v1.2; §0.1 and §0.2 list the corrections made during implementation).

What changed

1. Workers advertise their tier. worker and microvm-worker read SANDBOX_TIER (defaults container and microvm; set but empty sends no label) into Hello.labels["moca.dev/tier"]. No proto change.
2. Tier configuration and the affinity store (harness).
   • SH_SANDBOX_TIERS / SH_SANDBOX_DEFAULT_TIER are parsed and validated. The parser is pinned by a shared JSON fixture, which the control plane will reuse in PR 2.
   • New settings SH_SANDBOX_AFFINITY_TTL_SECONDS (default 86400) and SH_SANDBOX_AFFINITY_GRACE_SECONDS (default 60).
   • RedisAffinityStore keeps sh:sandbox:affinity:<sessionId> and sh:sandbox:detached:<sandboxId>. Its claim and detach-mark Lua scripts are atomic, and garbage or future marks are replaced by "now".
3. The relay's detach mark. Teardown overwrites the mark after the presence remove, and Hello clears it before the presence put. Both go through the same Redis client, so their order holds.
4. Tier filtering.
   • Only records in the session's tier are candidates. Pods count as the default tier.
   • An unlabelled or mislabelled worker is excluded, with one log line per sandbox id.
   • A new SandboxAffinityPendingError subclasses SandboxPoolSaturatedError, so it is a retryable 503 on /turn and "saturated" on leaves.
5. Session-to-sandbox affinity.
   • Busy: if the session's sandbox is at capacity, the turn gets a 503 naming it.
   • Absent: the turn gets a 503 "pending" for the grace period. This also covers a relay restart, which wipes every record.
   • Gone past the grace, or moved to another declared tier: the session falls back to the least-loaded sandbox in its tier, and the reset is recorded.
   • Sandbox record from another tier (including those written before tiers were on): honoured if its sandbox is in the session's tier.
   • Concurrent first turns: they converge on one sandbox through a compare-and-claim.
   • Leases: every acquired lease is returned or released on every path.
6. Placement and workspace_reset.
   • The tier is threaded through /turn as TurnConfig.sandboxTier. PR 2 sets it from the control plane's exchange.
   • A turn result carries sandbox: { id, tier, workspaceReset? }, and the runtime report gets sandboxId / sandboxTier / workspaceResetAt / workspaceResetFrom at turn end.
   • The workspace_reset frame goes out before any model output. mocactl shows it as a warning notice; older clients ignore unknown frames.
7. End-to-end test (packages/sandbox-relay/test/tier-affinity.integration.test.ts). It uses real Redis and a real relay, with two tiers of two fake workers each. It covers:
   • tier pinning;
   • an exact least-loaded spread;
   • affinity across turns;
   • surviving a relay restart;
   • the grace-period wait;
   • the in-tier fallback, and the TTL on the relay's mark.

Compatibility

• With SH_SANDBOX_TIERS unset nothing is filtered. Affinity alone applies, which fixes the container-tier workspace hop (two container sandboxes, with turn 2 landing on the other one).
• The pods-only path is unchanged: no affinity store is built or called.
• Error messages without a tier are byte-for-byte unchanged.

Follow-ups (not in this PR)

• PR 2: the control plane records each session's tier and returns it on the exchange; mocactl gets a tier option; the resources view shows placement. It will also record a reset when the fallback turn itself fails.
• PR 3: deployment wiring and startup validation of SH_SANDBOX_TIERS.
• Separate issue: two concurrent turns of one session that both outlast the grace period can each move it. The workspace has already been reset by then; a compare-and-set replace would close this.

Test results (rebased on upstream/main fc021ba, which includes #436 and #437)

make lint                          EXIT:0
make typecheck                     EXIT:0
make test                          EXIT:0   (harness 575 passed, sandbox-relay 66, knative-server 423, mocactl 559, control-plane 453, supervisor 187, …)
cd remote-worker && go test ./...  EXIT:0

Part of #425.

🤖 Generated with Claude Code

[pdettori]

Reviewed P6.3 PR 1 — sandbox tiers and session-to-sandbox affinity (harness, relay, workers, mocactl).

Checked: tier parsing/validation (parseSandboxTiers), the Redis affinity store (atomic claim/detach-mark Lua, corrupt-value and clock-skew handling), tier filtering and the affinity state machine in selectPoolSandbox (busy → 503, absent-within-grace → pending 503, past grace / retiered → in-tier fallback with workspace_reset), first-turn claim-race convergence with single re-run, lease release on every path, relay detach-mark ordering through the shared FIFO client, Go TierLabels (incl. set-but-empty semantics), knative-server 503 mapping and runtime-report fields, mocactl frame handling, and the test suite — including the real-Redis/real-relay integration test. Dependency addition (redis devDep in sandbox-relay) is expected for that test.

No blockers found. Two notes:

• check and k8s-kind-e2e were still pending at review time — confirm green before merge.
• Nit: the title runs ~95 chars, past the 72-char advisory.

(Posting as a comment — GitHub won't let the author's own token submit an approval; a second maintainer can approve.)

[pdettori]

suggestion — related to this mapping: turnErrorHeaders (below) advertises the global saturation Retry-After (default 5s) for SandboxAffinityPendingError too, so a session waiting out its (default 60s) grace polls ~12 times before the grace expires and the fallback runs. The error already carries retryInMs — could turnErrorHeaders take the error and use it (perhaps clamped, e.g. min(retryInMs, 10s)) so a client that honours Retry-After backs off proportionally? If the fixed 5s is deliberate — polling reacts fastest to a sandbox returning early — happy to see this land in PR 2/3 instead.

[pdettori]

Done in f485b72 (+ ae287f3): turnErrorHeaders(status, err?) now sets Retry-After for a SandboxAffinityPendingError to min(10, max(1, ceil(retryInMs / 1000))) — proportional to the grace left, capped at 10 s so a sandbox that returns early is still noticed quickly; a non-finite retryInMs falls back to the configured value. Every other 503 keeps KAGENTI_SYNC_SATURATION_RETRY_AFTER_S. Both call sites (sync catch and the SSE pre-first-frame catch) pass the error. Tests in turn-error-status.test.ts.