fix(cloud): a busy Mac chat moves to a new Mac before its deadline and keeps working

[andrewcai8]

Problem

An instance-engine Mac chat could not run a turn longer than about 4h47m. Before Namespace's five-hour deadline, upkeep saves the chat and releases its Mac. The host then marked every released chat paused, busy or not, and nothing reopened it. A turn still running at the deadline stopped and stayed stopped until a client woke the chat.

Fix

• NamespaceMacRuntime.upkeep now releases a Mac for sleep only when the chat reads idle (that callback replaces busy, so an unreadable chat no longer counts as idle). A chat still working at the forced-release threshold returns a new outcome, reopen. So does any awake chat found off its Mac.
• upkeepCloudChats handles reopen while it still holds the box. It keeps the lease awake and brings the chat back on a new Mac through wake, which is the same driver.resume + markActive path a client's resume uses (now shared with resume). It does not wait for a client. On the new Mac, the guest's restart continuation (continueThreadsAfterServerUpdate, on for boxes) picks the interrupted turn back up.
• Idempotent across host restarts. An awake lease whose chat is off its Mac, or on a new Mac it was never restored onto, comes back as reopen. If the host dies at any point of a move, the next upkeep pass finishes it on the Mac the move already created. A failed reopen leaves the lease awake for the next pass. An expired snapshot marks the lease missing.
• Move cap. A chat moves at most 3 times in a row (MAX_UNWATCHED_MOVES) with no client heartbeat or resume since the first of them. On the next deadline it is released for sleep and marked paused, as before this PR, and opening it continues the turn. The count lives on the lease (unwatchedMoves), so it survives host restarts. It rises only when a move lands, so a failed or redone move counts once. A client heartbeat or resume clears it; the reaper keeping a busy box awake does not.
• Idle releases behave as before: the chat sleeps until someone opens it.

All changes are in apps/server/src/environmentControl/ plus two sentences in docs/operations/cloud-provisioning.md.

Tests

The failing tests land first (032faac). NamespaceMacRuntime.test.ts covers four cases:

• A busy chat at its deadline returns reopen with its final snapshot saved.
• A second pass, standing in for a restarted host, still returns reopen.
• An awake chat whose Mac died returns reopen.
• An idle chat at its deadline returns released.

A fifth case covers a host that died mid-restore: the next upkeep returns reopen, and the reopen restores onto the existing Mac without creating a second one.

EnvironmentControl.test.ts covers the host side, including the cap: three moves land (a failed one is not counted), the fourth deadline pauses the chat, and a client resume or heartbeat starts the count again. ProvisionedLeaseRegistry.test.ts checks that the reaper's keep-awake touch keeps the count and a client touch clears it. reopen keeps the lease active and retries after a failed resume. Once the move lands, the lease records the new proxy. An expired snapshot marks the lease missing.

Live proof

The run used a local manager built from this branch with a throwaway, uncommitted patch: Mac lifetime 40 min, idle window 10 min, template builds off so no builder Mac. The forced-move threshold stayed at 8 min. The repo was andrewcai8/t3code on the instance engine, with Claude running a 90-tick job (one tick every 30 s, about 45 min). Each tick logs the Mac's hostname and boot time. Only one test Mac existed at a time.

Timeline (UTC, from the manager log and the report):

• 00:57:06. Mac qdt1o044i1beo created. The turn started at 00:58.
• 01:30:04. namespace mac released before its deadline (leftMs 421818, asleep: false). Final save of 11.5 MB took 4.6 s. The Mac departed at 01:30:13.
• 01:30:25. The host created Mac l5b738cbuqb7o with no client involved. It materialized at 01:31:25 from snapshot generation 7.
• The lease read active throughout (move.neverPaused). The child reconnected at the same origin 51 s after the new Mac appeared, and the thread read running.
• tick 63 ran on qdt1o044i1beo at 01:30:00. tick 64 ran on l5b738cbuqb7o at 01:32:06. The job reached tick 90 and printed DONE host=l5b738cbuqb7o at 01:45:16. A follow-up turn on the same chat then answered in 15 s.

Every check passed (move.progress 90/90 on 2 Macs) except host.skills.claudeAgent, which tests local skill config and has nothing to do with this change. Both test leases were disposed through the manager's dispose path, and no test Mac is left running.

Evidence lives on the machine that ran the test, under /tmp/mac-move-proof/: report.json, run-final.log, manager-move-excerpt.log, progress-excerpt.txt, plus the harness and the short-deadline.patch.

Risks

• An awake chat whose Mac dies out of band is now reopened instead of paused. The same goes for a host crash that lands between an idle release and its markPaused. Each costs one Mac until the chat goes idle near the new deadline, or until the move cap puts it to sleep.
• A chat whose activity can't be read is no longer released early as idle. It waits until the forced threshold and then moves.
• After a host restart, a client heartbeat that reaches renew before the first upkeep pass can still pause a chat that was mid-move. A client resume then continues it.

🤖 Generated with Claude Code

[github-actions]

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider	Metric	Main baseline	This PR	Impact	PR ceiling
Codex	Total thread wire	4.9 KiB	4.9 KiB	0 B (0.0%)	6.8 KiB	✅
Codex	Thread snapshot wire	3.7 KiB	3.7 KiB	0 B (0.0%)	4.9 KiB	✅
Codex	Live turn WebSocket wire	1.2 KiB	1.2 KiB	0 B (0.0%)	2.0 KiB	✅
Codex	Live turn WebSocket decoded	20.4 KiB	20.4 KiB	0 B (0.0%)	29.3 KiB	✅
Codex	Live turn messages	2	2	0 (0.0%)	8	✅
Claude	Total thread wire	4.9 KiB	4.9 KiB	0 B (0.0%)	6.8 KiB	✅
Claude	Thread snapshot wire	3.7 KiB	3.7 KiB	0 B (0.0%)	4.9 KiB	✅
Claude	Live turn WebSocket wire	1.2 KiB	1.2 KiB	0 B (0.0%)	2.0 KiB	✅
Claude	Live turn WebSocket decoded	20.7 KiB	20.7 KiB	0 B (0.0%)	29.3 KiB	✅
Claude	Live turn messages	1	1	0 (0.0%)	8	✅

Baseline: 8e2d5c6 · PR result: 77f04e8 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

• Codex decoded thread snapshot: 106.1 KiB
• Claude decoded thread snapshot: 106.4 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

[andrewcai8]

Verdict: PASS+NOTES (head 1e8d072afd3c0cad7159099a837b4396f6dcbac5)

Independent review. Code and evidence only, no Macs started.

Correctness. The decision logic holds. A host-driven move runs entirely under the per-box save lock (EnvironmentControl.ts:651). A client resume, pause, renew, reap or cleanup during the move is refused, not doubled. A client resume after the move hits planLive open on a ready record, which is done, so no second Mac. A client resume that wins the lock first creates the Mac itself, and upkeep then sees live. E2B and devbox leases are unaffected: upkeepChat still returns null for them. activity never throws (unknown on any failure), so an unreadable chat is treated as not idle and moves at 8 min, as described. #155's proxy restore still feeds the reaper's busy read (EnvironmentControl.ts:308).

Notes (none blocking):

1. Cost is unbounded for a chat that stays busy. The reaper keeps a confirmed-busy box awake forever (EnvironmentControl.ts:308). Before this PR, the Mac deadline was the only hard cap on an unwatched busy instance chat. Now each move also sends the guest's "Continue where you left off." prompt, so a chat that never settles costs one Mac per ~4h52m, plus provider usage, with no limit. Consider capping host-driven moves that happen with no client heartbeat (for example, one move) and pausing after that.
2. A host restart during materialize does not finish the move. The record is live with cache: "unknown". upkeep falls through to periodicSave, which skips (namespaceChat.ts:237) and returns kept (NamespaceMacRuntime.ts:735). Nothing completes the open, so the reaper pauses the chat (its read returns unknown), or a client resume finishes it. That window is the longest step (~60 s in the proof). Returning reopen for a live record that is not ready would close it. The PR body's "the next upkeep pass finishes the move" holds only for crashes between the release and the create.
3. The quota-full retry is bounded, but by a pause. A failed create (resource_exhausted) leaves the record idle. The next client heartbeat renew calls mac.touch, which returns released and becomes paused (EnvironmentControl.ts:990). Without a client, the reaper pauses the chat. So there is no create loop and no spend, but "retries next pass" in practice means a client resume retries it.
4. Near a retentionDeadline, the new Mac's deadline is capped at retention, so the last ~8 min can repeat the move each pass. This is API-only (no client sets it), so it is minor.

Tests. The tests assert outcomes and lease state. The test files are identical at 032faac and HEAD. Run against the pre-fix source, 3 of them fail (expected 'released' to be 'reopen', and others). At HEAD, vp test run src/environmentControl passes 42 files / 676 tests. Local typecheck is blocked by machine policy, and CI Typecheck plus all Test jobs are green. Notes 2 and 3 have no test coverage.

Live proof (/tmp/mac-move-proof, local times are UTC-7). The guest ran revision 1e8d0… with the patch at 40 min lifetime and 10 min idle window. The busy release came at leftMs: 421818, asleep: false (01:30:04Z). The final save was gen 7 (11.5 MB, 4.6 s), and qdt1o044i1beo departed at 01:30:13. With no client, l5b738cbuqb7o was created at 01:30:25 and materialized at 01:31:25 from gen 7. Lease active>active. Tick 63 ran on the old host at 01:30:00 and tick 64 on the new host at 01:32:06. The job reached 90/90 with DONE host=l5b738cbuqb7o. Confirmed. The aborted first run's Mac d1gu4foojj0qs was only disposed at 01:46. The files don't show whether it was already released by then.

Mergeability. MERGEABLE and the trial merge with main is clean. #159 and #161 touch nothing in environmentControl. #159 changed the guest's RestartContinuation (queued runs, a user interrupt suppresses continuation), so the proof ran a pre-#159 guest. The plain continue path is unchanged, though.

[andrewcai8]

Re: review. Notes 1 and 2 are fixed in 559b2b0 and 77f04e8, after merging origin/main (55470cb).

1. Unbounded cost (77f04e8). A chat now moves at most MAX_UNWATCHED_MOVES = 3 times in a row while no client heartbeats or resumes it. On the next deadline upkeep releases it for sleep and marks it paused, as before this PR, and opening it continues the turn. The count is unwatchedMoves on the lease, so it survives host restarts. markActive raises it only when a host move lands, so a failed or redone move counts once. A client heartbeat (touch) or resume (markActive) clears it. The reaper's keep-awake touch passes "host" and leaves it alone. Past the cap, upkeep calls driver.pause before markPaused, so a half-restored Mac is released too.

2. Host restart mid-restore (559b2b0). upkeep now returns reopen for a live record with cache: "unknown". The reopen runs drive("open"), which plans materialize on the Mac already recorded, so no second Mac is created.

Tests. Each new test failed against the pre-fix source and passes now:

• NamespaceMacRuntime.test.ts: a recorded Mac that was never restored gets reopen (it returned kept before the fix). The resume restores agent.txt onto that same Mac, and live() lists only that Mac.
• EnvironmentControl.test.ts: a failed move plus three landed moves keep the lease active, and the fifth pass pauses it. A client resume or heartbeat starts the count again.
• ProvisionedLeaseRegistry.test.ts: a host touch keeps the count and a client touch clears it.

vp test run src/environmentControl: 678/679 passed. The one failure was remotePreparation.test.ts, a timing test this PR does not touch, and it passed when rerun alone. CI is green on 77f04e8. Notes 3 and 4 are unchanged.