docs(spec): MU1 multi-user control plane — owned sessions, per-user credentials

[pdettori]

What

Specs MU1 — Multi-User Control Plane: an always-on trusted tier fronting the existing
scale-to-zero data plane, so many users can share one deployment while each authenticates, holds
their own credentials, and sees only their own sessions.

Docs only — no code changes. Adds:

• docs/specs/2026-09-08-multi-user-control-plane-design.md
• docs/adrs/0033-multi-user-control-plane.md
• registry entries in docs/specs/README.md and docs/adrs/README.md

Shape of the design

Placement	new always-on Deployment (@sh/control-plane), deliberately not Knative — it holds an OAuth client, mints tokens for cron-fired runs with no client present, and is the trusted tier
Data path	control path only. Client gets a session token, then POSTs turns and streams SSE directly to the Knative Service. The control plane never sees a prompt or a model response
Token	Ed25519, carrying sub + sid and no secret. The harness holds only the public key, so a compromised harness can verify but never mint — the trust tier dictates the algorithm
Credentials	per-user K8s Secrets under AES-256-GCM envelope encryption, own namespace, no list verb on the serving path. Open registry keyed by consumer tier, destination-bound
Ownership	one assertOwner() choke point; 404 (not 403) across tenants, to avoid an existence oracle

Three findings from tracing the tree at b533d87

These changed the design mid-flight and are the spec's main value:

1. The credential path reads the environment in three places, not one —
   run-turn.ts:306 (|| fallback), :310-311 (the write-once-if-absent seed), :313. Removing
   only the seed leaves the || fallbacks intact, so /turn would still fail open. All three go.
2. Redis is fully ephemeral — deploy/knative/redis.yaml has no PVC and no appendonly, so
   credentials cannot live there. That is what moved them to Secrets.
3. There is no session → sandbox reverse index — the lease ZSET is keyed by run id, not session
   id (sandbox-lease.ts:3-6), so the "view my compute resources" requirement had nothing to read.
   The harness self-reports a runtime record, which the spec marks display-only, never authz input.

Two limitations named rather than glossed

• Slice 1 ships a shared sandbox pool. Isolation holds at the API, the session store, and the
  inference credential — not the sandbox. The demo narration says so. The tenant-labelled
  partition is slice 2 and is blocked on Decide whether a workload's sandbox pool selector should bound its kind:'prompt' leaves #237.
• The control plane holds both identity and credentials, which diverges from
  Z1 §2's "orchestrator holds no secrets".
  Accepted in ADR-0033 to avoid blocking on kagenti-extensions, contained by a CredentialStore
  seam plus RBAC scoping, and retired by Z3/Z5 in slice 3.

Relationship to #228

MU1 is adjacent to, not blocked on, the P5 session-isolation spec in #228, whose implementation is
a separate contributor's track on a different timeline. §3.5 sets the ownership boundary:

• MU1 owns /turn and buildConfig's signature.
• P5 owns the leaf ScaledJob and CLI paths, the reachability pins for the four inert globals,
  and the scrub.

Both touch run-turn.ts:306-313, so whichever lands second rebases; neither ordering invalidates the
other's design. MU1's cut is a strict subset of P5's step 2. P5 reserves ADR-0032, so this takes
ADR-0033 — note that leaves 0032 unused if #228 does not land.

Filed alongside

• Decide whether a workload's sandbox pool selector should bound its kind:'prompt' leaves #237 — decide whether a workload's pool selector should bound its kind:'prompt' leaves. Deferred
  at server.ts:308-320 under ADR-0028; it gates slice 2's tenant partition, and it also affects the
  non-multi-user /runs path, so the spec records it as owed rather than deciding it.

Open questions for review

2. Status: Proposed in both the spec and the registry cell (design (proposed)) — bump together
   on acceptance.

Verification

make lint passes (prettier, shellcheck, hadolint, gitleaks, yaml). No code touched, so no test run
applies.

🤖 Generated with Claude Code

[pdettori]

I think this should be moved to its own track

[pdettori]

Done — moved to its own track. Rebasing to do it surfaced that #228 merged in the meantime, and reading the merged P5 spec contradicted something load-bearing in §3.4, so this push is both changes.

Own track. Z8 → MU1, first entry in a new MU (multi-user service) track with its own registry section and dependency notes. Phase 2 is a security architecture; this is a product surface with its own consumers and compatibility obligations, and the two meet only at the credential. MU2/MU3 are recorded so the former slices 2 and 3 have ids. Filename unchanged, per the registry's own convention that the canonical id lives in the registry rather than the filename.

§3.4 was wrong and is corrected. The draft said "pass the key as an explicit argument and pi's per-session apiKey does the rest." P5 §3.3 and run-turn.ts:150-155 show there is no such seam — createAgentSession cannot inject a per-session key, and pi resolves the key by provider name from ANTHROPIC_API_KEY. What is per-session is the model object, rebuilt each turn at :497 by applyModelGateway, which installs Authorization: Bearer from config and already prefers config over the environment at :306.

Two consequences, both good for the split:

• MU1 touches no line of run-turn.ts. Its work is making config carry the right subject's token, which is server.ts plus the control plane.
• Deleting the :310-312 seed is not MU1's to do — alone it makes ANTHROPIC_API_KEY absent and breaks gateway mode outright. P5's step 3 sentinel is what makes that deletion safe.

§3.5 is now composition, not a conflict boundary. P5 step 1 already reserved Authorization for "may this caller use the harness … ADR-0011's lock-down implies caller auth is coming" — which is exactly this spec — and carries whose-work-it-is in X-SH-Subject. MU1 adopts that split unchanged and adds the one thing P5 cannot: making the subject trustworthy rather than asserted. P5 reads X-SH-Subject from the request, which is right for a trusted orchestrator but is the spoofable-header pattern Z1 §3.2 warns about once arbitrary users can call the API. So: when a session token is present the subject is token.sub and an inbound X-SH-Subject is ignored; both present and conflicting is a 400 subject_conflict rather than a precedence rule, since a silent winner there is a cross-tenant bug waiting to be written.

One honesty change worth flagging. Pre-P5, MU1 enforces fail-closed by policy at two points it owns (session creation and the exchange both refuse a subject with no resolvable credential), not by construction (nothing ambient reachable). My earlier draft implied the stronger property. §3.5, §8.1 and §9.3 test 1 now say which is in force, and test 1 asserts the policy form today and tightens to the sentinel form once P5's implementation lands — so the weaker assertion doesn't have to be deleted later. P5's own §5 makes the reason concrete: asserting the sentinel alone would stay green while an ANTHROPIC_OAUTH_TOKEN outranked it.

Also fixed the seed citation to :310-312, matching P5 and the closing brace.

Rebased onto 02109bc; the docs/adrs/README.md conflict was purely Prettier reflowing every column width once ADR-0032's longer link landed. make lint passes.

Still open from the PR description: Status: Proposed in both the spec and the registry cell — bump together on acceptance.

[pdettori]

Added §3.6 — What MU1 owes P5's implementation, since P5's design is merged and shouldn't be rewritten to accommodate a later spec. Three interactions, one of which would have bitten:

The Bearer payload is one-or-the-other. P5 targets Bearer <inert placeholder> swapped by static-inject; MU1's interim direct mode carries Bearer <real token> with no injector. Same field, incompatible contents — a P5 implementation that unconditionally sets the placeholder silently overwrites MU1's token and the request fails with nothing configured to swap it.

Resolution: TurnConfig carries a tagged credential rather than a bare string, and the control plane declares the mode at exchange time with placeholder mode winning wherever an injector is configured — so adding an injector strictly narrows what the harness may hold, and MU3 deletes direct mode. The tag isn't ceremony: with a bare string both failure directions are silent (placeholder sent upstream when the injector is misconfigured; real key rewritten when an injector is added later). P5 §3.4's own argument — that an ambient placeholder is as dangerous as an ambient key, because the injector faithfully swaps in whichever tenant it names — applies just as much to a mislabelled one.

Direct mode diverges from P5 §5, not just Z1/Z3. §5 asserts no real provider credential is reachable from the harness process; direct mode puts one there per turn. Now named in both the spec and ADR-0033's accepted-cost line, so P5's implementation doesn't assert an invariant MU1 knowingly breaks. Suggested there that the assertion be scoped to the environment, which MU1 never writes.

Inbound X-SH-Subject becomes conditional, so P5 shouldn't pin "always honoured".

Coordination note posted on #228 (#228 (comment)) rather than a P5 tracking issue — the issues replacing #220 don't exist yet, so that needs carrying into whichever one gets filed.

Also aligned naming now the track exists: former slices 2 and 3 are MU2/MU3, and §2.2 no longer reads as though MU1 performs P5's deletions.

make lint passes. Three commits now: the spec, the review response, and this.

[cwiklik]

Approving. Docs-only, and unusually well-traced: every file:line citation in this spec checks out against source — the three credential reads (run-turn.ts:306, :310-312, :313), createAgentSession at :499-504, the model rebuild at :497, buildConfig and its four call sites, resolveRunWorkload at server.ts:300-324, :553, :482-494, leaf-job.ts:15-18, cli.ts:9-14, redis.yaml (no PVC / no appendonly / 128Mi), sandbox-lease.ts:3-6. The ADR-0028 deferral is quoted verbatim and correctly.

I had drafted this as REQUEST_CHANGES against 3d5ac7dc on four points; the last two commits resolved all four, so this is an approval of f29a83b5:

• §8.1's mechanism. The earlier row attributed "a turn runs on its own subject's key or not at all" to no env fallback on /turn, which did not follow: with no credential, run-turn.ts:314 returns baseModel unchanged rather than throwing, leaving pi's withEnvApiKey and service.yaml:45-49's real key reachable — while the scrub that would close it was assigned to P5. §8.1 now says enforced by policy pre-P5, by construction once P5's sentinel lands, §3.5's "If P5's implementation has not landed" draws the cannot vs does not line explicitly, and ADR-0033 records it as an accepted cost. That is the honest form of the claim.
• §9.3 test 1 previously set only ANTHROPIC_AUTH_TOKEN — the one variable the change stops reading — so it would have stayed green while ANTHROPIC_API_KEY still authenticated. Now two-phase, asserting ANTHROPIC_OAUTH_TOKEN/ANTHROPIC_AUTH_TOKEN absent and citing the precedence.
• The merge conflict from #228's docs/adrs/README.md column reflow, and the four stale "#228, open against main" statements — both gone; no Z8 or "whichever lands second" leftovers anywhere.

Two things worth calling out as better than what I asked for. §3.4's correction — that createAgentSession exposes no seam for a per-session key, so the per-turn model object is the only per-subject surface — is the right reading of the code and reverses a plausible-sounding first draft. And §3.6 catches a collision I had not raised: P5's inert placeholder and MU1's real token contend for the same Authorization: Bearer field, in which a bare string makes both failure directions silent. The tagged UpstreamCredential with placeholder-mode-wins is the correct resolution, and "adding an injector strictly narrows what the harness may hold" is the property that makes it safe.

Four non-blocking comments inline: two suggestions (§8.2's #237 dependency looks inverted; §6.2's descriptor has no field for the base URL §5.3 returns) and two nits.

Author: pdettori (MEMBER — maintainer)
Areas reviewed: Docs/specs, ADRs
Agent/IDE config (.claude/.vscode): none
Commits: 3, all signed off
CI status: passing (12/12)

Assisted-By: Claude (Anthropic AI) noreply@anthropic.com

[cwiklik]

suggestion — I think this dependency is inverted, and the source cited two paragraphs up says so directly. server.ts:308-311:

A prompt leaf DOES lease a pool sandbox now, and honors an envelope sandboxPoolSelector (ADR 0028 amendment, 2026-09-01) — but a workload-addressed one still ignores the workload's own selector.

Two different selectors. #237 governs only the workload-addressed one — whether a workload's own sandboxSelector should bound its prompt leaves. But the mechanism this section proposes is the envelope: "the control plane supplies a sandboxPoolSelector per tenant through the seam at server.ts:553/:300-324". That path already works for prompt leaves — sandboxEnvironment() (run-leaf.ts:124-128) is not gated on kind, and runPromptLeaf reaches it at :390.

If the control plane injects the tenant selector into the leaf envelope after the :553 scrub — which is exactly what that comment describes as the intended post-boundary path — the partition is enforceable today. Worth re-checking, because it may unblock MU2's headline item entirely.

Same claim in §11.1 and in docs/specs/README.md (the MU2 row's "gated on #237", and the MU2 is gated bullet), so it propagates to the track table.

[pdettori]

You're right, and it's the more useful kind of wrong — it unblocks MU2's headline item. Traced it before changing anything:

• sandboxEnvironment() (run-leaf.ts:124-128) checks only env.sandboxPoolSelector, not gated on kind; runPromptLeaf reaches it at :390.
• selectPoolSandbox takes its environment as an argument (select-sandbox.ts:75), so the selector is already request-scoped rather than process-global — which is exactly what a per-tenant partition needs, and I'd assumed had to be built.

So #237 governs only the workload-addressed selector, as you say. §8.2 now carries the two-selector table with your quoted source, and #237 stays open on its own merits for the /runs + workloadId path.

One correction to the conclusion, though — the partition isn't quite enforceable today on MU1's path. sandboxPoolSelector is a LeafEnvelope field on the /runs path; /turn never leases from the pool at all — run-turn.ts:57 resolves a single pod via resolveSandboxConfig, never selectPoolSandbox. Since MU1's interactive path is /turn, MU2's work is making that path take a request-scoped selector, which is the same substitution runPromptLeaf already made and which its own comment at run-leaf.ts:385-387 calls "a superset, not a behavior swap" (no selector set ⇒ identical single-pod fallback).

Net: not a deferred cross-cutting decision, just plumbing on a path this spec already touches. Propagated to §11.1, the MU2 registry row, and the MU2 is not gated bullet.

[cwiklik]

suggestion — the descriptor has no field for the gateway base URL, but §5.3's exchange returns one:

◀── { anthropicAuthToken, anthropicBaseUrl }

destination.hosts is a host allow-list (["api.github.com", "github.com"]), not a base URL — applyModelGateway needs a full origin plus path, e.g. https://litellm.internal/v1. So for consumer: inference it is unclear where anthropicBaseUrl is stored, or whether it is deployment-wide rather than per-credential.

This matters because of the || at run-turn.ts:313: if config.anthropicBaseUrl comes back undefined, gatewayBase falls through to process.env.ANTHROPIC_BASE_URL, and if that is also unset the model gets Authorization: Bearer <subject token> with no baseUrl override — the subject's gateway token sent to the default endpoint. Either add an optional base-URL field to the descriptor, or state that it stays deployment-level and the exchange simply echoes it.

[pdettori]

Real gap, fixed. Added an optional endpoint field to the descriptor, meaningful only for consumer: inference, with the reasoning you gave: destination.hosts is a host allow-list, not an origin, so it can't serve as the gateway address applyModelGateway needs.

Resolution order is credential endpoint → deployment default → refuse with endpoint_unresolved. The refusal is the substantive part, and it's your :313 trace that makes the case: with config.anthropicBaseUrl undefined, gatewayBase falls through to process.env.ANTHROPIC_BASE_URL, and with that unset the model gets Authorization: Bearer <subject's token> and no baseUrl override — one user's gateway token sent to the default Anthropic endpoint. That's a misdirected secret rather than a degraded request, so it fails closed instead of defaulting.

Per-credential rather than deployment-only because a user may hold keys for different gateways, and endpoint is non-secret metadata so it lists without decryption like the rest of the descriptor.

Threaded through §5.3 (the exchange now returns {mode, anthropicAuthToken, anthropicBaseUrl} and never an undefined base), §6.2, §9.1's taxonomy, and §9.2's fail-closed list.

[cwiklik]

nit — the chain is missing a step that this section's conclusion depends on. getApiKeyEnvVars("anthropic") returns ["ANTHROPIC_OAUTH_TOKEN", "ANTHROPIC_API_KEY"] — OAuth first, so it outranks the API key.

That is load-bearing for the diagram's ANTHROPIC_API_KEY = P5's inert sentinel ← satisfies pi's existence check: the sentinel only ends up being what pi resolves if ANTHROPIC_OAUTH_TOKEN is absent, which is precisely why P5's step 3 deletes it rather than only overwriting ANTHROPIC_API_KEY. §9.3 test 1 already has this right ("asserting the sentinel alone would stay green while an OAuth token outranked it") — worth one clause here so the two sections agree.

[pdettori]

Added. §3.4 now states that getApiKeyEnvVars('anthropic') returns ["ANTHROPIC_OAUTH_TOKEN", "ANTHROPIC_API_KEY"] in that order — OAuth first — so the sentinel is only what pi resolves once ANTHROPIC_OAUTH_TOKEN is deleted, which is why P5's step 3 deletes rather than merely overwrites. Also added the deletion to the diagram, so §3.4 and §9.3 test 1 now say the same thing rather than one of them carrying the load alone.

[cwiklik]

nit — kagenti-extensions has been renamed and now redirects to rossoctl/cortex. Pre-existing (the merged P5 spec and ADR-0032 both carry the old name, so this is a shared docs nit rather than anything introduced here), but these are new lines, so worth using the current name in them — also in ADR-0033's Context and Consequences, and in the MU3 row of docs/specs/README.md.

[pdettori]

Updated to rossoctl/cortex on the lines this PR adds — §3.3 of the spec, and ADR-0033's Context and Consequences. The MU3 registry row now says rossoctl/cortex too.

Left the merged P5 spec and ADR-0032 alone, and also the pre-existing kagenti-extensions mention in the Phase-2 note in docs/specs/README.md — that one predates this PR, and renaming it here would mix an unrelated docs sweep into a spec review. Worth a separate pass if you want the whole tree consistent.