Skip to content

fix(lastcode): stop hand-repairing the same checkpoint failures every cycle - #293

Merged
lastobelus merged 3 commits into
lastcode/mainfrom
lastcode/checkpoint-repeat-failures
Oct 7, 2026
Merged

lastobelus merged 3 commits into
lastcode/mainfrom
lastcode/checkpoint-repeat-failures

Conversation

@lastobelus

Copy link
Copy Markdown
Owner

Three checkpoint/build failures recur on most cycles and get repaired by hand each time. All three are automation bugs, not real conflicts.

Cleanup after publish leaves a half-deleted worktree. git worktree remove drops the worktree's metadata even when deleting its files fails partway. macOS writes .DS_Store into directories while Git empties them. In the 2689 and 2702 leftovers, node_modules/.pnpm contained only a .DS_Store written after the delete began. The fix, removeAutomationWorktree, keeps Git's dirty and lock checks. Once Git has already unregistered the worktree, it finishes the delete with fs.rmSync retries. All eight automation-owned removal sites use it.

Checkout refresh blocks on any submodule pointer change. Upstream regularly adds, moves, and removes vendored .repos/alchemy-effect/* submodules, which are never initialized in the primary checkout. The refresh already checks out with --no-recurse-submodules, so those pointer changes do nothing on disk. The guard now rejects only gitlinks whose submodule directory has content, ignoring .DS_Store. The runbook sentence is updated.

Wait for Checkpoint crashes on startup (#290). The waiter runs from the primary checkout the service is refreshing. Its reporter imported the shared protocol and contracts, so it failed with ERR_MODULE_NOT_FOUND when the source moved ahead of node_modules. The LastCode Action kit now encodes frames using only Node built-ins and type-only imports. The new test decodes them with the host's createActionProtocolDecoder, and a second test loads the waiter from a copy with no node_modules. Run against the previous kit, that second test fails with ERR_MODULE_NOT_FOUND. The waiter also allows a 30s grace when the launchd job is briefly unavailable right after a start request. An idle service still reports "not running" immediately.

Closes #290.

Validation: 223 focused tests across checkpoint, supervisor, waiter, Action kit, Wait for PR, build package, and recovery pass. Scoped format and lint are clean, and the scripts typecheck passes. Quick CI was skipped at the maintainer's request; GitHub CI is the gate.

Implemented with Claude Opus 5.5 in the Claude Code harness.

🤖 Generated with Claude Code

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-07T02:47:12.279850Z 98e9372 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

lastobelus and others added 3 commits October 6, 2026 19:40
git worktree remove drops a worktree's metadata even when deleting its files fails midway, which happens when macOS writes .DS_Store into a directory Git is emptying. Checkpoint publication then failed during cleanup and left an unregistered checkout behind. Keep Git's dirty and lock checks, and finish the delete with retries only once Git has already unregistered the worktree.

Carry-Group: tooling
Carry-Observation: Published 2689 and 2702 both failed cleanup with .DS_Store as the only entry left in a directory Git had emptied.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…odule moves

The primary checkout refresh rejected every promoted gitlink change, but upstream regularly adds, moves, and removes vendored .repos submodules that are never initialized locally. The refresh checks out without recursing into submodules, so those pointer changes are no-ops on disk. Reject only gitlinks whose submodule has local content, ignoring Finder's .DS_Store.

Carry-Group: tooling
Carry-Observation: Refreshes for 2667 and 2702 were blocked by empty .repos/alchemy-effect submodules and were advanced by hand.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The waiter runs from the primary checkout that the checkpoint service is refreshing. Its reporter loaded the shared protocol and contracts, so it crashed with ERR_MODULE_NOT_FOUND when the source moved ahead of node_modules. Encode Action frames in the kit with Node built-ins and type-only imports; a test decodes them with the host decoder. The waiter also allows a short grace when the launchd job is briefly unloaded by a just-requested start.

Carry-Group: resumable-actions
Carry-Fix: #290
Carry-Observation: Seen moving Effect 4.0.0-rc.115 to 4.0.1, and as a launchctl probe failure when armed right after starting the service.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@lastobelus
lastobelus force-pushed the lastcode/checkpoint-repeat-failures branch from 44a5f8f to 98e9372 Compare October 7, 2026 02:42
@lastobelus

Copy link
Copy Markdown
Owner Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: 98e9372ea0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@lastobelus
lastobelus merged commit 51db0b3 into lastcode/main Oct 7, 2026
17 checks passed
@lastobelus
lastobelus deleted the lastcode/checkpoint-repeat-failures branch October 7, 2026 02:50
lastobelus added a commit that referenced this pull request Oct 7, 2026
Merging a PR while a checkpoint was validating threw the run away. Repaired checkpoints published the tag and `lastcode/main` in one `--atomic` push, so a mid-run merge rejected the validated tag too. The repair then had to be reselected, with the merge folded in by hand, and fully revalidated. Ordinary checkpoints published their tag but failed the run at promotion, which alerted the maintenance thread. On 2026-10-05, #280 and #282 each cost a manual fold-in and a 10–15 minute revalidation.

The validated tag no longer depends on main:

- **Repaired checkpoints** publish the tag and immutable source ref atomically, then promote separately, the same as ordinary checkpoints.
- **Promotion defers instead of failing** when main has advanced to a descendant of the candidate's source or git's lease rejects the push because a descendant merge landed mid-push. A rewrite that drops the pinned source still fails promotion after tag publication, and the rejected-push path fetches the competing head before checking ancestry. The run succeeds and logs that promotion is left to the next run. Failure to acquire the promotion lock still fails the run after the tag publishes; the lock ref cannot prove its writer is active, and abandoned locks or authentication/transport errors must remain visible to maintenance.
- The guarded merge already requests a service run, and the supervisor runs it straight after the current run. That run publishes a revision replaying only the merged work onto the new tag, then promotes it. If main still equals the source, it promotes the published tag directly.
- **Merges after recovery selection no longer invalidate it.** Only a main that no longer descends from the selected source (for example, rewritten by another promotion) requires reselection.
- **Recovery cleanup follows promotion validation.** A main rewrite or promotion failure keeps the repaired worktree and selection for inspection. Retry promotes the existing immutable tag without republishing; successful promotion or validated descendant deferral releases the repair.

You can merge at any time. The build can start from the checkpoint tag immediately, and merged work follows minutes later as a revision.

Runbook text in `fork-conventions.md`, `release.md`, `nightly-workflow.md` and the `lastcode-pr` skill is rewritten to match.

Validation: focused checkpoint and recovery tests, selected-recovery lifecycle tests, and carry lifecycle tests pass. Lifecycle regressions cover descendant merges after selection, during validation, and during the promotion push; unrelated remote-only rewrites retain the selected repair and fail visibly while preserving the published tag and rewritten main. They also verify blocked retries preserve that repair, restored-source retries promote the existing tag, and promotion-lock failures retain recovery until a successful retry. The scripts typecheck and scoped format/lint checks pass. GitHub CI and Codex review are required on the final head and base. Quick CI was skipped at the maintainer's request; GitHub CI is the gate.

Includes merged #293 as its current base.

Implemented with Claude Opus 5.5 in the Claude Code harness. Refreshed and repaired with GPT-6.1-Sol in the Codex harness.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Carry-Source-Ref: refs/lastcode/carry-sources/pr-294/e70ba03bd32f5e16161d968d3da613b6bd5b0d9b
Carry-Source-Base: 00eaf51
Carry-Source-Head: e70ba03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Wait for Checkpoint can fail during primary dependency refresh

1 participant