fix(link): report stale Child connection readiness

devlog/_plan/261107_orphaned_child_link/010_design.md

@@ -0,0 +1,39 @@
+# Orphaned Remote Link Child (#8243)
+
+## Failure and hypotheses
+
+`src/client/runtime.ts` binds a startup connection snapshot; `src/client/machine-listener.ts`
+reports tunnel readiness without checking that snapshot against the durable connection. Meanwhile,
+`src/client/connect.ts` refuses disconnect after the record disappears unless a disconnect receipt
+already reached the clearing phase. Pinned sync in `src/client/sync-expectation.ts` correctly refuses
+the missing connection and must keep doing so without writes.
+
+Possible causes of the missing record include an old whole-config writer, a concurrent config
+mutation, or an external replacement. An audit of config persistence and regression fixtures can
+exclude or demonstrate individual cooperating writers; it cannot identify the historical external
+writer without evidence from the affected machine. The runtime symptom is falsified if readiness
+changes when only durable ownership changes; the disconnect symptom is falsified if the existing
+locked transaction can complete from a missing record and safely identified artifacts.
+
+## Design and verification
+
+Use durable connection identity as the authority for listener readiness and status. A missing,
+invalid, or different owner/link is distinct from a tunnel failure. Do not derive a connection
+from health, tunnel state, or catalog. Revalidate under the existing lifecycle and config locks
+before any cleanup; preserve unknown or replaced credentials, sidecars, Desktop state, and Codex
+state. Only remove artifacts with independently established ownership, and make interruption and
+concurrent reconnect safe. Preserve pinned sync's exit 5 and no-write contract.
+
+Add failing-first focused tests for the stale listener, orphan cleanup, replacement races and
+negative ownership cases. Audit whole-config writers and protect a proven cooperating overwrite
+where feasible. Update the owning structure and user-facing docs. Run focused tests, typecheck,
+`test:changed`, structure and privacy gates, and the docs build; document unrun coverage.
+
+## Open recovery question
+
+The existing `client-link.json` carries a link ID and tunnel details, but no token fingerprint.
+The token file and this sidecar can each have been replaced independently. An absent connection
+therefore does not establish that the present token belongs to the sidecar, and a token read
+followed by compare-and-swap alone only proves that the token did not change during cleanup. A
+recovery implementation needs an independent owner witness and a safe path for legacy orphans;
+it must not interpret sidecar presence as proof of token ownership.

docs-site/src/content/docs/guides/remote-link.md

@@ -73,7 +73,9 @@ A failed link does not silently switch to a local provider.
 The Child returns a retryable `503` without forwarding the link key or request when its tunnel is failed, stopped, or not supervised.
 It also keeps the key local when another process owns the tunnel port; free that port so the Child's SSH tunnel can bind it.
 
-On a Child, `GET /healthz` only says that OpenCodex is running. `GET /readyz` (and `ocx ready`) also says whether the link can carry a request right now: while it cannot, it answers `503` with `Retry-After: 1`, `"status": "pending"` and a `link` object such as `{ "state": "unavailable", "reason": "home_unreachable" }`, and while it can, `200` with `"status": "ready"` and `{ "state": "available" }`. The reasons are `tunnel_connecting`, `tunnel_failed`, `tunnel_idle`, `tunnel_unavailable`, `sidecar_invalid`, `home_unreachable`, `home_unauthorized`, `link_auth_failed`, `link_credential_unavailable` and `link_unverified`. A Child that joined from its own dashboard answers from its tunnel. A Child that its Home added has no tunnel of its own to look at, so it asks the Home through the tunnel with the same credential-free check every relayed request starts with; it repeats that check at most once every five seconds, and answers `link_unverified` when the Home has not replied within half a second.
+On a Child, `GET /healthz` only says that OpenCodex is running. `GET /readyz` (and `ocx ready`) also says whether the link can carry a request right now: while it cannot, it answers `503` with `Retry-After: 1`, `"status": "pending"` and a `link` object such as `{ "state": "unavailable", "reason": "home_unreachable" }`, and while it can, `200` with `"status": "ready"` and `{ "state": "available" }`. The reasons are `tunnel_connecting`, `tunnel_failed`, `tunnel_idle`, `tunnel_unavailable`, `sidecar_invalid`, `home_unreachable`, `home_unauthorized`, `link_auth_failed`, `link_credential_unavailable`, `link_unverified` and `client_state_changed`. A Child that joined from its own dashboard answers from its tunnel. A Child that its Home added has no tunnel of its own to look at, so it asks the Home through the tunnel with the same credential-free check every relayed request starts with; it repeats that check at most once every five seconds, and answers `link_unverified` when the Home has not replied within half a second.
+
+The reason `client_state_changed` means the running Child's saved connection is missing or now names another owner or link. Its machine and link status endpoints also return `503` with this reason. Stop or restart the Child and check its connection before re-linking; this readiness result does not itself disconnect the relay.
 
 To confirm which Home is serving a Child, compare the `x-opencodex-served-by` header on the Child's `GET /v1/models` with the `instanceId` in the Home's `GET /healthz`. The Child passes the Home's header through unchanged, so the two match while the link works. The value is an opaque id, never a hostname; see [Home identity and the served-by header](/reference/management-api/#home-identity-and-the-served-by-header).
 

src/client/machine-listener.ts

@@ -41,6 +41,7 @@ import { linkRouteAllowed } from "../link/routes";
 import { linkHomeName, linkHomePhrase } from "./link-home";
 import { relayLinkManagementRead } from "./link-management-relay";
 import type { ClientRestartAdmission } from "./runtime-restart";
+import { clientRestartRollbackAction } from "./runtime-restart";
 import { isLinkConnectionGeneration } from "../link/relay-auth";
 
 const VERSION = packageVersion("0.0.0");
@@ -136,6 +137,7 @@ export function startMachineListener(
     return state.value;
   })();
   const linkMode = isLinkConnection(connection);
+  const connectionChanged = () => clientRestartRollbackAction(connection, readClientConnectionState()) === "recycle";
   if (linkMode && !connection.link) throw new Error("link machine listener requires link transport metadata");
   const managementAuth = deps.managementAuthState ?? initializeManagementAuthState(config);
   let hubReachability: HubReachability = "unknown";
@@ -199,9 +201,17 @@ export function startMachineListener(
         const identity = { service: "opencodex", version: VERSION, role: "client" } as const;
         const facts = { uptime: process.uptime(), pid: process.pid, port: server.port, protocolVersion: 1 };
         let link: LinkReadiness | null = null;
+        if (linkMode && connectionChanged()) {
+          return Response.json({ ...identity, status: "pending", ...facts, link: { state: "unavailable", reason: "client_state_changed" } },
+            { status: 503, headers: { "Retry-After": "1", "Cache-Control": "no-store" } });
+        }
         if (linkMode && deps.linkReadiness) {
           try { link = await deps.linkReadiness(); } catch { link = { ready: false, reason: "link_unverified" }; }
         }
+        if (linkMode && connectionChanged()) {
+          return Response.json({ ...identity, status: "pending", ...facts, link: { state: "unavailable", reason: "client_state_changed" } },
+            { status: 503, headers: { "Retry-After": "1", "Cache-Control": "no-store" } });
+        }
         if (link && !link.ready) {
           // The /readyz contract: not ready is 503 with Retry-After and status pending or failed.
           return Response.json({ ...identity, status: "pending", ...facts, link: { state: "unavailable", reason: link.reason } },
@@ -253,6 +263,9 @@ export function startMachineListener(
         if (managementPrincipal(req, managementAuth, config) !== "gui-session") {
           return Response.json({ error: "opencodex machine GUI session required" }, { status: 401 });
         }
+        if (linkMode && connectionChanged() && (url.pathname === "/api/machine/status" || url.pathname === "/api/machine/clients" || url.pathname === "/api/link/status")) {
+          return Response.json({ error: "client_state_changed" }, { status: 503, headers: { "Cache-Control": "no-store" } });
+        }
         // A loopback dashboard session proves possession, not user presence: any local
         // process can fetch the dashboard bootstrap and replay its token and CSRF value.
         // Keep the connected listener useful for status/diagnostics, but never let that