Skip to content

fix(control): recover a damaged control journal without duplicate effects (#1240) - #1254

Merged
Juliusolsson05 merged 9 commits into
mainfrom
fix/control-history-quarantine
Sep 25, 2026
Merged

Juliusolsson05 merged 9 commits into
mainfrom
fix/control-history-quarantine

Conversation

@Juliusolsson05

Copy link
Copy Markdown
Owner

Closes #1240.

Problem

A damaged events.jsonl (a crash mid-append, a newer build's event after a downgrade, or missing rows) made every control call fail with history_unavailable: renderer feed paging, terminals, workflows, External Control. It stayed that way across restarts until someone edited the file by hand. A failed append also blocked the rest of the process.

Why it isn't "move the file aside and start fresh"

The journal is also the executor's dedupe ledger. A keyed retry finds its earlier received row and replays the stored result, or answers interrupted. An empty ledger would turn that retry into a second real effect (steering q12). Plan: docs/plans/2026-09-25-control-history-recovery.md (plan-only commits 8be3658b and 4ea65889).

What changes

Damage Recovery
Torn tail only Preserve the original bytes and keep the verified prefix. Full service. This is safe because received and dispatched are durable before any effect, so the torn row is the last write and its call's earlier rows survive.
Any row valid except for an unknown kind (downgrade) Keep the valid rows and block only that row's exact (caller, requestKey).
Anything else (bad JSON, missing or odd fields, a sequence gap) Keep the valid rows and block all new keyed intents (q13).
  • In every case, reads, unkeyed calls and retries whose received survived keep working.
  • The block is derived from the preserved evidence on every load. A restart, deleting recovery.json, or a malformed marker (q14) cannot lift it. Lifting it takes an explicit acceptance: putting the evidence digest in recovery-accepted.json. The refusal message and a control.history_recovered incident name the file and the digest.
  • A failed append now re-reads the disk on the next append instead of poisoning the process.

Evidence

  • The fixture is real rows from the owner's journal (5,885 rows). The rows hold only ids and payload digests.
  • Seven new tests; all were red on main (history_unavailable) where applicable.
  • Mutation checks: each guard, the q13 rule, the kind-restriction (q14), marker validation, the reload-on-failure path and torn-tail truncation all fail a test when removed. trust-record-over-file is equivalent, because present files are always rescanned.
  • tsc -b is clean. The control, control-sdk and incident suites pass (80 tests).

Not in scope

  • A UI to inspect quarantined journals. The incident and the path are the surface for now.

🤖 Generated with Claude Code

Juliusolsson05 and others added 3 commits September 25, 2026 03:21
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lls; explicit acceptance (steering q13)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ects

A torn tail, a schema-rejected row or a sequence gap made
FileControlHistory.load() reject, cached, so every control call, reads
included, failed with history_unavailable across restarts until
events.jsonl was edited by hand (#1240). A failed append also poisoned
the process.

The journal is also the executor's dedupe ledger (steering q12), so it
is recovered, not reset:
- Torn tail only: the verified prefix is complete evidence (received
  and dispatched are durable before any effect), so full service
  resumes after the original bytes are preserved.
- Other damage: every valid row is kept. New keyed intents are refused
  wherever a lost row could have held their key. Only a row valid except
  for an unknown kind proves its exact pair; anything else blocks all
  new keyed intents (q13). Reads, unkeyed calls and retries whose
  received survived keep working.
- The block comes from the preserved evidence, re-analyzed on every
  load. A restart, deleting recovery.json, or a malformed marker (q14)
  cannot lift it; only naming the evidence digest in
  recovery-accepted.json does. The refusal message says so.
- A failed append drops the cached load, so the next append re-reads
  the disk instead of poisoning the process.
- Recovery is recorded as a control.history_recovered incident.

Tests use real journal rows and are red on main; each guard is
mutation-checked.

Closes #1240

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Juliusolsson05 added a commit that referenced this pull request Sep 25, 2026
…for #1240

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Juliusolsson05 and others added 5 commits September 25, 2026 04:04
…records never lifted

Review A (blocker): the executor stamps every row of a call with its
request key (all 1,846 real calls agree), but the dedupe lookup reads only
`received` rows. A downgrade-shaped `received` row with a corrupted key
string, or a schema-valid one with its key deleted, let the real key
dispatch again while its dispatched/result rows still named it. Keys named
by a damaged call's kept rows are now blocked, and a call whose rows
disagree about the key is damage in its own right.

Review A (major): truncating a quarantine inside its damaged line made
re-analysis call it a torn tail and replace the recorded block. Recorded
guards now always apply; re-analysis only adds.

Review B: a MISSING kind is corruption, not the downgrade shape; exact-pair
precision, gap salvage + renumber, preserved invalid markers and
load-failure un-caching are now pinned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… every launch (steering q17)

Rewriting a call whose valid rows disagree about its request key could
never make the ledger clean, so each launch re-quarantined the same bytes
and filed another incident. Its rows now leave the rewritten ledger; the
quarantine keeps the bytes and the recorded pairs keep its keys blocked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eyed call needs its intent

Both round-2 reviewers: the task writer (tasks.ts) appends `step` rows on
a call's id with no request key, so any keyed startControlTask call read
as inconsistent. Its rows (including the durable task result) were
dropped at the next load, and accepting that digest let the key run
twice. The consistency check now reads only the rows the executor stamps
(received/dispatched/result/duplicate); all 1,846 real calls agree there,
while 660 real step rows carry no key.

Review A: a keyed call left with dispatched/result rows but no received
or duplicate row lost its intent (for example, rewritten into another
valid call) and is now damage whose keys are blocked. There are 0 such
calls in the real journal.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Juliusolsson05

Copy link
Copy Markdown
Owner Author

Review disposition (2 rounds: Grok A, Pi B; steering q12–q17)

Finding Disposition
q12: "quarantine and start fresh" would re-run retried keys Design. A torn tail keeps the verified prefix (received/dispatched are durable before any effect); any other damage blocks new keyed intents wherever a lost row could hold their key.
q13: an ambiguous damaged row, and how clearing works Only a row valid except for an unknown kind proves its pair; everything else blocks globally. Clearing requires naming the evidence digest in recovery-accepted.json.
q14: duplicate rows / malformed marker The refusal applies only to a new received row. The marker is schema-validated; a malformed marker is preserved and blocks globally.
A1 r1: a corrupted key string, or a deleted key, while siblings keep the real key Fixed. Keys named by a damaged call's kept siblings are blocked; a call whose executor rows disagree is itself damage.
A2 r1: truncating the quarantine lifted a block Fixed. Records always apply; re-analysis only adds.
B r1: exact-pair precision, missing kind, gap salvage, invalid markers, load un-caching Fixed or pinned.
q17: an inconsistent call re-quarantined on every launch Fixed. Its rows leave the active ledger once; the keys stay blocked.
Round 2, A and B: keyed startControlTask step rows (no key) read as damage, and accepting then ran the key twice Fixed. Only executor-stamped kinds are compared (0 disagreements among 1,846 real calls; 660 real step rows carry no key). tasks.test.ts is green again.
Round 2, A: a received row rewritten into another call Fixed. A keyed call without a received/duplicate row is damage (0 in the real journal).
B4/B5 (incident wiring test; relative filename) Accepted.

CI is green on cca694f0, which contains origin/main. An earlier CI failure was the unrelated intermittent extension Electron test, filed as #1261.

@Juliusolsson05
Juliusolsson05 merged commit 9c1de6e into main Sep 25, 2026
2 checks passed
@Juliusolsson05
Juliusolsson05 deleted the fix/control-history-quarantine branch September 25, 2026 12:05
Juliusolsson05 added a commit that referenced this pull request Sep 25, 2026
…nce rule

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(control): one torn line in control-history blocks every control call until hand-repaired

1 participant