Skip to content

Recovery: roll back an install that does not come up, and a hand-back runbook - #20179

Closed
rawphp wants to merge 230 commits into
pingdotgg:mainfrom
original-solutions:t3/factory-issue-59
Closed

rawphp wants to merge 230 commits into
pingdotgg:mainfrom
original-solutions:t3/factory-issue-59

Conversation

@rawphp

@rawphp rawphp commented Oct 16, 2026

Copy link
Copy Markdown

Closes #59

Contract

  • done: after restart_job, install.sh waits up to T3FAC_HEALTH_WAIT s (90) for a TCP connect on the port and a factory started or no factory config line appended to the log after the restart (size noted before; a smaller file is read from offset 0). Either missing: last 50 log lines, rollback.sh, exit 1 with install of <version> failed its health check; rolled back to <previous>; no previous: exit 1 and says so. rollback.sh runs the same check on the restored version and exits 1 naming both versions. README gains a Recovery section.
  • out: log rotation (Fix ChatView header overflow for long thread titles #60), handover.md (Show pulsing terminal activity icon in thread sidebar #21), plist or port changes.
  • risk: none
  • test: vp test run apps/server/src/factory/installScripts.test.ts → pass (31 tests)

Acceptance criteria

  • install.sh waits up to T3FAC_HEALTH_WAIT (default 90) for the port and a fresh log line — wait_healthy in factory/scripts/lib/t3fac.sh; tests keeps a healthy install, rolls back an install whose port does not accept connections, rolls back an install whose server never logs that it started, keeps an install whose server logs that it has no factory config
  • Missing: last 50 log lines, rollback.sh, exit 1 with the message — rolls back an install whose server never logs that it started (asserts line 11 shown, line 10 cut)
  • No previous version: exit 1 and says so — exits 1 and says so when an unhealthy first install has nothing to roll back to
  • rollback.sh checks the restored version and names both versions — rollback.sh exits 1 naming both versions when the restored version is unhealthy too, install.sh reports when the rollback fails its health check as well
  • Old factory started line does not count; truncated log reads from offset 0 — does not count a factory started line that was there before the restart, counts a line written after the log was truncated to a size below the noted one
  • Four cases with a stub launchctl, a stub listener and a written log — apps/server/src/factory/installScripts.test.ts, describe install.sh and rollback.sh health check
  • README "Recovery" section: rolling back by hand, hand-back to the Rust factory, moving back, telling the fork is down — verified factory/README.md section ## Recovery

Assumption: the issue says tests in factory/scripts/; the existing script tests live in apps/server/src/factory/installScripts.test.ts, which is what the gate runs, so the new cases are there. T3FAC_PORT is a test-only override for the health check port; the plist still serves 3876.

Gate

  • factory/scripts/check.sh → exit 0 (seams, 33 test files / 1138 tests, server and web typecheck)

Models

  • maker: claude-sonnet-5-5 (host subagent); the Check-fix commit (tests only) was made by the orchestrating session
  • checker: claude-opus-5-5

Review

  • cycles: 1
  • roster: correctness (approve), reliability (approve), testing (12/12 named mutants killed; one extra survivor: no test for the no factory config line)
  • fixed: test for the no factory config line; the log-tail assertion now proves the 50-line bound (both mutants re-run and killed by the orchestrator)
  • leftovers: none

Lessons

  • Push the branch by HEAD, not by the name in the launch snapshot. Symptom: git push -u origin <branch> fails with "src refspec does not match any". Cause: the git status snapshot at session start named a different branch (t3/8c8fa6d2) from the worktree's real one (t3/factory-issue-59). Default: read git branch --show-current first, or push HEAD, and never pass the snapshot's name.
  • Read skill references with absolute paths, never cd into the skill directory. Symptom: after cd <skill>/references && cat ... the harness reports the primary working directory moved off the worktree. Cause: the session's cwd follows the shell. Default: cat /abs/path/file from the worktree; every command keeps starting with cd <worktree> &&.

Commits

  • 520aaab0b test: install and rollback wait for the server to come up — Make slice (red: 6 failed)
  • b33d8dfcd feat(factory): roll back an install that fails its health check — Make slice (green)
  • 626a4971b docs(factory): recovery section — Make slice (docs, no red phase)
  • 51fe118b3 test: a server that logs no factory config is healthy, and the log tail is 50 lines — Check cycle 1 (tests for behavior that already worked, no red phase)

Files

  • factory/scripts/lib/t3fac.sh — log, port (T3FAC_PORT), log_before noted in restart_job, new wait_healthy
  • factory/scripts/install.sh — health check after the restart; rollback or say there is nothing to roll back to; exit 1
  • factory/scripts/rollback.sh — health check on the restored version; exit 1 naming both versions
  • apps/server/src/factory/installScripts.test.ts — shared stub listener, LAUNCHCTL_HEALTHY / LAUNCHCTL_HEALTHY_LINE / LAUNCHCTL_TRUNCATE_LOG in the launchctl stub, nine health-check cases
  • factory/README.md — Recovery section; install and rollback one-liners mention the health check

Isolate: the factory's worktree was behind origin/main (ancestor, clean tree); fast-forwarded with git merge --ff-only origin/main before provisioning. Repo labels: all seven existed; none created.

rawphp and others added 30 commits October 15, 2026 19:44
Red: ./index.ts does not exist yet, so the import fails.

Issue: #1
Green: factory started is logged once.

Issue: #1
Red: factory/scripts/check-seams.sh does not exist yet, so both cases fail.

Issue: #1
Green: both check-seams cases pass.

Issue: #1
No red phase: the tree already complies, so the real-tree test passes on
arrival. The synthetic cases show each rule catching a violation.

Issue: #1
Typecheck could not resolve the bare vitest module; the repo imports it via vite-plus/test.

Issue: #1
…order-free

Red: "names an upstream file renamed into an allowed path" fails because
check-seams.sh lets git rename detection hide the deleted upstream path.
The fail case now changes server.ts itself, and the boundary real-tree
test asserts server.ts imports ./factory/index.ts; both already pass.

Issue: #1
A git mv of an upstream file into an allowed path listed only the new
path, so the deleted upstream file passed the seam check. --no-renames
lists both sides.

Issue: #1
- boundary.test.ts flags factory code outside factory/t3/ that imports the
  server's workspace packages (@t3tools/*, effect-acp, ...), read from
  apps/server/package.json.
- check-seams.sh captures the diff before the loop, so a failed git diff
  fails the script instead of passing with nothing checked.
- README and the script header say the check covers committed HEAD only.
- check-seams tests cover apps/web/src/factory/ and a failing diff.
- Node built-ins imported as NodeFS/NodePath/... per namespace-node-imports.
Factory skeleton: an empty FactoryLayer wired in through one seam
… plainly

Review nits: the cd to the repo root was never exercised, and the expected
calls rebuilt the constants they came from. No red phase: both behaviors
already held.

Issue: #2
Red on purpose: install.sh, rollback.sh and build.sh do not exist yet.

Issue: #3
… tail

Red: apps/server/src/factory/process.ts does not exist yet.

Issue: #8
…ests

Ports the run id, next_attempt, format_ts and parse_ts tests from softfac's
ledger.rs, plus the issue key parse and refuse cases from issue_key.rs.

Red: runId.test.ts and time.test.ts fail because runId.ts, time.ts and
errors.ts (LedgerError, used by testSupport.ts) do not exist yet.

Issue: #9
Ports softfac's RunId, IssueKey, next_attempt, format_ts and parse_ts to
plain TypeScript, and LedgerError with the Rust Display messages.

- Run ids are lowercase owner/name plus a canonical issue (up to u64 max) and
  attempt (up to u32 max); file names that are not exactly what this code
  writes are not run ids.
- Lowercasing is ASCII-only so a Kelvin sign cannot fold into a legal name.
- validateRunId also checks the issue and attempt, since they end up in the
  file name too.

Issue: #9
Ports the run ledger tests from softfac's ledger.rs: one exact line for every
one of the 30 event types, create/open semantics (create-new, newline repair
of a torn last line, symlinks and directories refused), the read rules (torn
last line, blank lines, invalid lines counted, unknown events and odd data
kept as written, causes kept as written), finish (total_ms and gate_attempts
computed by the writer), the failed-write rules through an injected AppendIo,
file and directory modes, and the directory-sync failure that removes the new
file. The feedback read file and the list/summary readers are not ported.

Red: ledger.test.ts fails because events.ts, appendFile.ts, read.ts and
ledger.ts do not exist yet.

Issue: #9
Ports softfac's find_project and status cases from src/t3.rs, adds launch
and thread cases, and a contract test over the real orchestration layers
with the Claude replay transcript. Red: paths.ts and sessions.ts do not
exist yet, so none of the three files compile.

Issue: #5
Red: the wrapper sh dies on SIGTERM and the group is SIGKILLed at once,
so the grandchild's marker never reaches the tail.

Issue: #8
rawphp and others added 27 commits October 16, 2026 13:26
A run whose last complete line is a finished event is skipped after one
bounded read from the end of its file. A cut-off or invalid last line,
or a finished line longer than the tail, falls back to the full read.

Issue: #49
…s, isolates failures

Red: the tests call unassignedClaims with a known set and a ClaimMemo and expect a
{ claims, unread } result, none of which exist yet (compile errors count as red).

Issue: #48
…eady explained

Fails before the fix: the labeledAt tracker method does not exist yet, and releaseClaim skips the comment on an old cut-short comment and never reads the label history.

Issue: #50
… issues

The scan took one serial GraphQL history call per unassigned triage/building issue on
every pass, including issues the pass then dropped for having a run, and one failed
read failed the scan for the whole repo. It now takes the issue numbers the pass has
runs for and reads nothing for them, remembers issues judged not the factory's while
their updatedAt is unchanged (for the life of the process), and reports a failed read
for that issue only, which the pass prints as one error line.

Issue: #48
A run file holding only blank lines, or a large file whose last 8192 bytes are all blank, must be listed as unfinished exactly as readRun reads it (state null, or the state of its first line). Red: before the fix the run hung node (the vitest worker printed no result and had to be killed after the 60s alarm did not stop it).

Issue: #49
The blank-line step-back called lastIndexOf with a byteOffset of -1 once it reached the first byte, which Buffer reads as counting from the end, so the walk looped forever. With no earlier line it now returns false and the file is read in full.

Issue: #49
…short scan

The two new tests fail: such a failure is still swallowed into unread and the
scan goes on to the next issue.

Issue: #48
…scan

A rate limit, a timeout or a gh spawn failure would fail every remaining
history read, so it now ends the scan like a failing issue list does. Only
GitHubFailed, GitHubParse and GitHubOutputTooLarge are kept per issue.

Issue: #48
…c failures

Four cases a mutation review showed were missing. They pass against the
current code (test-only cycle):

- index: a judgement that an issue is not the factory's lasts across passes
  of one world (one history read over two scans)
- index: a failed history read reaches the pass as an unread entry with the
  issue and a non-empty message
- github: an unparseable history is that issue's GitHubParse error and the
  next issue is still read
- pass: a run in another repo is left out of the known set handed to the scan

Issue: #48
…shed are read in full

Both tests pin existing behavior, so there was no red phase. They kill two mutation survivors: a last line naming finished that is not a valid entry, and a cut-off fragment after a complete finished line, must each force a full read.

Issue: #49
A pass skips finished run files without parsing them
…se reads

Review round 1: f1, f2.
- labeledAt and unassignedClaims now pick the claim's label event through one
  newestLabelEvent helper.
- alreadySaid and releaseClaim share newestSays for 'the newest factory comment
  is this text, in GitHub's time form'.
- A release from building is tested against a history with both triage and
  building label events, so the released state must reach the label lookup;
  labeledAt is tested for building as well.
…ssue and label

The memo now keys by login, repo, issue and state label and stores the
updatedAt it judged at, so a re-authenticated gh judges again and a newer
judgement replaces the old one instead of piling up. Tests pin each part
of the key, a re-read after a failed read, an oversized history as that
issue's error, and a gh that cannot start ending the scan.
…on a set read

A run whose session reported its pull request is no longer limited until it
next enters building: the wait for the gate is not the build's time. A triage
run whose launch was sent with no session recorded is left to resume, which
names its session, so the limit never finishes a run whose session may be
running untracked.

The events wake checks an in-memory set of owned threads, read at start and
after every pass, instead of reading every run file per status change.

The time-limit comment no longer promises that the next attempt continues
from the old branch. sanitize is imported from text.ts everywhere, and
advance's reminder send and ended-badly statuses each have one home.

Tests: an unconfirmed run is never time-limited, watched or advanced; a
hostile branch name reaches the comment and ledger only as prepared text; a
run back in building from pr-open is limited again.
Until the gate is wired into the pass, a building run whose session reported
its pull request stays in building. It has no session at work, so like a
pr-open run it no longer takes one of max_concurrent_runs, and new claims are
not starved by finished builds. The listing carries prReported from the
ledger, cleared when the run enters building again.
…ad again

Once the session has reported its pull request, the build is done and the run
waits for the gate (#19). The pass no longer watches its text or calls advance
for it, so each pass makes no tracker or T3 call for a finished build and only
says it is ready for the gate.
Building: follow the session to its result line
Cut-short scan stops re-reading label history for issues it never acts on
# Conflicts:
#	apps/server/src/factory/index.test.ts
#	apps/server/src/factory/index.ts
#	apps/server/src/factory/tracker/github.ts
A cut-short release comments unless this claim was already explained
Fails until the scripts check health: the unhealthy install, closed port, old log line, no-previous, and both rollback-check cases. The healthy and truncated-log cases already pass.

Issue: #59
After the restart, install.sh and rollback.sh wait up to T3FAC_HEALTH_WAIT seconds (90) for the port and a fresh factory started or no factory config log line. A failed install runs rollback.sh; a failed rollback exits 1 naming both versions.

Issue: #59
…il is 50 lines

Review findings: nothing exercised the "no factory config" line, and the
tail assertion could not tell 50 lines from the whole log. Both already pass.

Issue: #59
@rawphp rawphp closed this Oct 16, 2026
@github-actions github-actions Bot added vouch:unvouched PR author is not yet trusted in the VOUCHED list. size:XXL 1,000+ changed lines (additions + deletions). labels Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XXL 1,000+ changed lines (additions + deletions). vouch:unvouched PR author is not yet trusted in the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants