Skip to content

fix(tldr): one invalid TLDR/Goal record no longer breaks TLDR and Goal for every agent (#1247) - #1257

Merged
Juliusolsson05 merged 16 commits into
mainfrom
fix/tldr-store-invalid-record
Sep 25, 2026
Merged

Juliusolsson05 merged 16 commits into
mainfrom
fix/tldr-store-invalid-record

Conversation

@Juliusolsson05

@Juliusolsson05 Juliusolsson05 commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

Closes #1247.

Verified failure

On the owner's real tldr.json (328 records, one exactly at the 400-char limit), one over-limit record made every store call throw TLDR must contain 1–400 characters. The validator threw instead of returning false. The result:

  • every tldr_update and goal_set failed;
  • the peeks and Agent Activity were empty;
  • the enforcement hooks threw.

A half-written goal completion did the same. Plan: docs/plans/2026-09-25-tldr-store-invalid-record.md (plan-only commit first).

Fix

  • Invalid record: set aside per identity, never guessed or repaired. That identity has no current record and can report again.
  • Preserved bytes: the original file is kept byte for byte at <file>.invalid-<digest>.json before any write can drop the record; one copy per content.
  • Malformed container: still refuses, since a write through it would destroy every record.
  • Validation: stored text uses a non-throwing check, including completion notes and history entries.

Review follow-ups (in this PR)

  • Reads survive a failed copy: if the evidence copy cannot be made, reads still work, and writes are refused until the copy exists.
  • Revisions continue above a set-aside record, so live readers (TLDR overlay, Close Completed) never discard a fresh record as stale.
  • History salvage:
    • keeps every valid entry;
    • writes an evidence copy with no .json extension, so history eviction never counts it;
    • moves an oversized history aside whole;
    • leaves an unreadable history untouched rather than replacing it.
  • An unchanged re-post repairs a salvaged history (review C). The file on disk is rewritten, so the history view stops failing.
  • Salvage takes nothing from a damaged history of a different identity.

Evidence

  • Real records, with texts redacted to same-length markers: testing/fixtures/tldr-store/real-records-2026-09-25.json.
  • Four tests are red on main (one rewrites the old "refuse the whole file" pin); the container test already passed.
  • Mutants killed: refuse-again, no-preserve, random copy name, throwing validator.
  • tsc -b is clean; src/main/tldr passes (65 tests).
  • Every surviving mutation from reviewers B and C is pinned (C: continuing at a revision-32 record, the salvage identity check, a set-aside revision below 1, the salvaged re-post).

🤖 Generated with Claude Code

Juliusolsson05 and others added 4 commits September 25, 2026 04:12
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he whole store

One record that fails validation, such as an over-limit text from a newer
build before a downgrade (the owner's real file already has one AT the
400-char limit) or a half-written completion, made every read and write
of the store throw. Every agent's tldr_update and goal_set failed, the
peeks and Agent Activity went empty, and the enforcement hooks threw.
The validator itself threw instead of returning false. Invalid records
are now set aside per identity, the original bytes are preserved beside
the file (named by digest, one copy), and a malformed container still
refuses.

Closes #1247

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y, carry revisions

- A crash mid-copy left a partial file under the final name, and its
  existence was taken as proof the evidence was safe. Copies are now
  published with temp + rename, and an existing copy counts only if its
  bytes are identical.
- One unreadable history entry read as an empty history, so the next
  report replaced that identity's whole timeline. The raw file is now
  preserved (without a .json name, so eviction never counts or deletes it)
  and every entry that still validates is kept.
- An identity reporting again after its record was set aside restarted
  at revision 1, which every reader discards as stale. It now continues
  above the set-aside revision.
- The copy's 0600 mode is pinned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Juliusolsson05 and others added 2 commits September 25, 2026 04:46
…ait for it

Review B: preservation failing (a read-only directory, a full disk) made
every READ of a store with one set-aside record throw again, which is
#1247's symptom. Reads now succeed; the copy is owed before the first
write, which retries it and is refused if it still cannot be made, so
nothing is dropped unpreserved. Pinned: an over-limit completion note, an
over-limit history entry, and reads never answering an inherited property
name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Juliusolsson05 added a commit that referenced this pull request Sep 25, 2026
…ptional fields, owed copy

Both reviewers: an object without a `workspaces` list (a typo'd key, a
top-level list) loaded as zero workspaces, and the next save erased it
with no copy. Every file this store writes carries the list, so its
absence now refuses like any malformed container.

Review A: mistyped optional fields passed through (a `description`
object threw in the command palette's text helper, taking the picker
down). They are now dropped per field; a mistyped title becomes the
file's basename, and a mistyped attachedAt becomes the workspace's
createdAt.

A copy that cannot be made no longer fails the load (same rule as
#1257): it is owed before the next save, which retries it and is refused
while it fails.

Review B: preserveInvalidBytes gets its own tests (exact bytes once,
0600, digest name, forks a fresh name over foreign bytes, no temp
leftovers on failure). The status repair is asserted as stale, and the
remaining row guards are pinned.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Juliusolsson05 and others added 8 commits September 25, 2026 05:15
…versized ones move aside unread

Review A (round 2): when even the raw read failed, the history fell back
to empty and the next report replaced the file uncopied; it now throws
(the report still succeeds, and history failures stay non-fatal) and
leaves the file untouched. Salvage also bypassed the reader's 512 KiB cap
by loading the whole file; an oversized history is now renamed aside
without being read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Juliusolsson05 and others added 2 commits September 25, 2026 09:53
…istory; C's surviving mutants pinned

After a salvage the file on disk was still damaged, and an unchanged
re-post returned early, so the history view kept failing. It now rewrites
the salvaged rows. Tests pin continuing above a revision-32 set-aside
record, the identity check in salvage, and a set-aside revision below 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Juliusolsson05

Copy link
Copy Markdown
Owner Author

Final disposition (pre-merge), head 5362ee6. Three reviewers: A and B (two rounds each) and C (third reviewer per steering q37, on the final head).

Reviewer Round 1 Round 2
A (correctness, consumers) FIX-BEFORE-MERGE FIX-BEFORE-MERGE
B (tests, real data) FIX-BEFORE-MERGE MERGE

Findings and dispositions:

Round 2

  • A, major: if the raw read of a history file also failed (for example chmod 000), the history was treated as empty and the next report replaced it with no copy. Fixed in 97d6ce0. The raw read now throws, the file is left alone, and the report still succeeds because both appendHistory callers catch history failures. A test checks the bytes are unchanged.
  • A, minor: salvage skipped the 512 KiB history size cap and loaded the whole file. Fixed in 97d6ce0. A history file over the cap is renamed to .invalid-oversize-<uuid> without being read. A test covers this.
  • B, pinning gap (mutation survived): no test catches setting the carried-over revision to 1, because the test record is already revision 1. Not fixed. The code is correct (B's probe went from revision 12 to 13). Not blocking.
  • B, pinning gap (mutation survived): no test catches removing unpreserved = null. Without it, the warning would repeat on every write. Not fixed. Only affects log output. Not blocking.

Round 1

  • A1, major: an interrupted preserved-copy write was trusted as complete. Fixed in 02a0fb9 (write to a temp file, then atomic rename; an existing copy is accepted only if its bytes match).
  • A2, major: one invalid history entry erased that identity's history. Fixed in 02a0fb9 (raw bytes preserved, valid entries kept). The unreadable-file remainder was fixed in 97d6ce0.
  • A3, major: one bad entry still made the Goal Loop store fail as a whole. Out of scope here. Fixed by fix(goal-loop): one unreadable loop no longer loses every goal loop (#1248) #1258 (merged).
  • A4, major: re-reporting a set-aside identity reset its revision to 1. Fixed in 02a0fb9 (numbering continues above the set-aside revision).
  • A5, minor: a set-aside record reads as "never reported". Accepted as the plan's policy.
  • A6, minor: distinct invalid versions pile up copies with no limit. Accepted. It needs repeated outside edits, and a normal write drops the record.
  • B1, minor: a failed copy failed every read. Fixed in 0a0622f (reads succeed; the first write retries the copy and is refused until it succeeds).
  • B2, B3, B4, minor: tests did not cover over-limit completion notes, over-limit history entries, or the commit null-prototype guard. Fixed in 02a0fb9 and 0a0622f. B confirmed in round 2 that all three mutations are now caught.

Caveats / residuals:

  • A carried-over revision lasts only while the app runs. Once the dropped record is gone from disk and the app restarts, that identity starts again at revision 1.
  • Copy names, warning text, and the two pinning gaps above have no tests.

Reviewer C (q37), on the final head: MERGE-READY with four minors. All four are fixed in this head's parent commit:

  • C1 (test strength): the revision-continuation test used the at-limit record, which is revision 1, so "always continue at 2" passed. It now uses the highest real revision (32). The mutant dies.
  • C2 (behaviour): an unchanged re-post after a salvage returned early, leaving the damaged file on disk, so the history view kept failing. It now rewrites the salvaged rows. A new test covers it and the mutant dies.
  • C3 (test strength): salvage's identity check was untested. A damaged history from another identity now donates nothing; the mutant dies.
  • C4 (test strength): the set-aside revision ≥ 1 guard was untested. A −5 revision now continues at 1; the mutant dies.

The body now describes every review follow-up. src/main/tldr passes (65 tests), and `npx tsc -b` is clean. Merges on a green gate that contains current main.

🤖 Generated with Claude Code

@Juliusolsson05
Juliusolsson05 merged commit f619149 into main Sep 25, 2026
1 of 2 checks passed
@Juliusolsson05
Juliusolsson05 deleted the fix/tldr-store-invalid-record branch September 25, 2026 18:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(tldr): one invalid tldr.json or goal.json record breaks TLDR and Goal for every agent

1 participant