fix(cli): a seat log recorded four failures and not one time - #1002
Conversation
The fleet is launched as `nohup commonly agent run <name> > <log> 2>&1`, so every line a seat emits lands in one file. None of them carried a timestamp. That stopped being cosmetic on 2026-08-18, when a fleet-wide quota stall destroyed 38 queued events (#993) and the seat log was the ONLY surviving trace — the kernel rows were deleted, so nothing else recorded that those turns had ever been attempted. The log says `(4 consecutive)` and `next probe in 2.0m` and gives no way to place either on a clock. The investigation had to date the events by decoding ObjectId prefixes, because the file that named them could not say when. Stamps every sink in `agent run`: the bootstrap logger, its error path, the run logger, and the run error channel. ISO-8601, so they sort lexically and line up with kernel timestamps without conversion. Both run sinks, deliberately. A spawn failure is emitted through `log` AND `onError` — they serve different contracts, the narrative line carrying the event-type prefix and the error channel being what an embedder hooks — so stamping only the first would leave every FAILURE line undated, which is the exact class the stamps exist for. It also makes that pair legible rather than misleading: two identical messages at the same millisecond read as one event, where `grep -c 'session limit'` previously returned 8 for 4 failures and made the delivery cap look twice as close as it was. Not unit-tested, and stating why rather than adding a vacuous one: these sinks are injected inside `.action()`, which no test invokes — the existing registration smoke proves `registerAgent` builds, not that a callback runs. Verified instead by building the command tree with the helper in scope, and by the full suite (21 suites, 298 tests). The honest check is a seat restart, which belongs to whoever fast-forwards the worktree. Reported-by: sprint-review Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The comment said stamping only `log:` would "leave every FAILURE line undated". It wouldn't. The failure path calls `log(...)` and then `onError?.(wrapped)`, so each failure already prints twice into one merged stream — patching one sink dates one copy and leaves its twin bare. Half-dated is worse to read than uniformly undated: a stamped line beside an identical unstamped one looks like two events at two times, so the artifact that already inflates the count would start inflating the timeline too. Same conclusion, both sinks stamped; the reason was wrong, and it was the reason a later reader would use to decide whether the second stamp mattered. Reported-by: sprint-review Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Correcting the rationale in this PR's body and in the code comment — @sprint-review caught that I described the wrong failure mode. I wrote that stamping only Their framing is sharper than mine: half-dated is worse to read than uniformly undated. A stamped line beside an identical unstamped one looks like two events at two times — so the artifact that already inflates the count ( The change is unaltered — both sinks stamped, which is what makes the pair collapse to "same message, same millisecond, one event." Only the justification was wrong, and it was the justification a later reader would use to decide whether the second stamp was worth keeping. Fixed in the code comment. |
The retry path called `log(...)` and then `onError?.(wrapped)` with the same text. Under `nohup … > log 2>&1` both land in one file, so every spawn failure appeared twice: `grep -c 'session limit'` returned 8 for 4 failures, and counting hits per event id returned 2, which reads as two deliveries of one event. That number was load-bearing on 2026-08-18. Two deliveries per event puts `attempts` at 2 and the requeue cap one step away, which is exactly the theory the evidence had to rule out before #993's real cause could be named — the log corroborated the wrong diagnosis, and the duplication had to be discovered by hand first. Routed through the error channel when a caller provides one, and to the log when it doesn't. Both contracts survive: an embedder passing no `onError` still sees the failure, and `agent run` — which always passes one — stops printing it twice. No text is lost either way, because `wrapped.message` already opens with the event type; the log copy was adding a second prefix to a line that named itself. Only this site. The other `log`-then-`onError` pair (the no-prompt skip) carries DIFFERENT text in each — a terse log line and a detailed diagnostic — so it is two messages, not one message twice. Both branches tested and both control-run: restoring the double emission fails the single-emission assertion, and dropping the log fallback fails the no-onError assertion. 51 in the run-loop suite, 299 across the cli. Reported-by: sprint-review Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lilyshen0722
left a comment
There was a problem hiding this comment.
Both sinks stamped — that was my finding and it's covered. Ran the suite on the head: 51/51 passing. One finding, and it is the defect class this PR's sibling entry is about.
Verified
log:andonError:are both stamped at the registration site. Patching one and not the other was the specific hazard and it did not happen.- The bootstrap path picked up something I had not flagged:
console.error(err.message)was previously unprefixed as well as unstamped, and now carries both. Good catch. - Every run-loop line inherits the stamp through the injected callback, so this covers the whole seat rather than the failure path.
- ISO-8601 is the right choice and the rationale is correct — lexical sort is what makes
sort/grep-based forensics work at all, and it lines up with the kernel's own stamps without conversion.
Finding: the onError: comment justifies the change with behaviour this same commit removes
// Both sinks are stamped, and both must be. A spawn failure is emitted
// through `log` AND `onError` … so each failure already prints twice
// into this one merged stream.
//
// Stamping only `log:` would therefore date ONE COPY of each failure and
// leave its twin bare.That was true an hour ago. It is not true of the tree this comment lands in, because the production change forty lines up removes the duplication:
if (onError) onError(wrapped);
else log(`[${event.type}] ${wrapped.message}`);and the new test pins it:
expect(logs.filter((l) => /claude process died/.test(l))).toHaveLength(0);So after this PR a spawn failure prints exactly once. A reader arriving at that comment will look for a duplication that no longer exists, and the closing line — "Stamped on both, the pair collapses … where grep -c 'session limit' previously returned 8 for 4 failures" — describes an outcome this commit achieves by a completely different mechanism.
The conclusion is still right and the code is still right. Both sinks do need stamping, just for the surviving reason rather than the removed one: they carry different content now. log gets every narrative line plus failures for an embedder with no error channel; onError gets failures when one is provided. Neither is a subset of the other, so an unstamped sink leaves a real gap either way.
Suggested: rewrite that comment in the post-change tense — "each sink carries lines the other does not, so both need stamps" — and move the 8-for-4 history to the collapse comment upstream, which is where it now belongs and where it already reads correctly.
Worth fixing rather than waving through, because it is precisely the shape of #997's entry: a surface stating a premise that is false in the tree containing it, written by the person who knew better, in the commit that made it false. I filed the same class of finding on #1001 an hour ago (OnboardingSilenceEpisode.ts:18–19), so this is three in one session — which is itself an argument that "does this comment survive my own diff?" belongs on the review checklist.
Not verified
The stamping itself has no test, and I agree with your reasoning for not adding one — a test asserting stamp() returns an ISO string pins the helper and leaves the wiring exactly as unverified, which is the #997 shape again. The real check is a seat restart, and that is Sam's. Worth noting explicitly in the PR body so "no test" reads as a decision rather than an omission.
The comment said the doubled count "was load-bearing for ruling out the requeue cap". It wasn't. What ruled the cap out was the database: a capped event retires to 'failed', 'failed' survives 168h, and no such row existed for any of the events in question. That leg stands on its own and never touches the log. The doubling did the opposite of load-bearing — it corroborated the wrong cause, and had to be spotted by hand before the count could be discounted. Worth stating precisely in the file that produced it: a log that inflates is worse than one that is silent, because it argues. Reported-by: sprint-review Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
#1002 stamped both log sinks and justified it from a double emission — the retry path called `log` and `onError` with identical text, so every failure printed twice into one merged stream. True when the stamps landed. False eleven minutes later, when 13ad436 in the same PR collapsed that duplication, and false on main ever since. The conclusion survives: both sinks still must be stamped, for the reason the comment originally claimed and then abandoned. A spawn failure now routes to `onError` when a caller provides one and to `log` only as the fallback, so in `agent run` every failure line comes out of the error channel — stamping only `log:` would leave the entire failure class undated. The no-prompt skip is a second case: two sinks, genuinely different text, half a pair dated. Kept the history in the comment rather than overwriting it, because the shape is the lesson: a diff shows changed code and never the prose it just invalidated, and here the prose was five lines from the change, added by the same PR, and survived three rounds of review including two that rewrote this very block. Reported-by: sprint-review Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Three times on 2026-08-18, in careful PRs by the author who knew the subject best, a change invalidated a comment it did not touch: #1001 removed the 30-minute pending delete, while models/OnboardingSilenceEpisode.ts:18-19 went on citing "pending rows are deleted at AGENT_EVENT_STALE_PENDING_MINUTES (default 30)" as the reason its event snapshot is NOT OPTIONAL. #1002 collapsed a double log emission, while the comment it ADDED in the same diff argued "each failure already prints twice" as the reason to stamp both sinks -- a behaviour its own new test now forbids. #997 the AX entry drafted the same hour, cataloguing this identical defect on two product surfaces. The third is the tell. A room that could name the failure in the product could not see it in its own diffs, because the author holds the pre-change model in their head as the thing being fixed. That makes it a reviewer's rule rather than an author's, which is why it lands here rather than in REVIEW.md. The check is mechanical and cheap: for each behaviour a PR alters, git grep the constant, env var, status value, threshold or tool name it touches, and read every prose hit. Two riders included because both changed what the fix should be -- the stale sentence is usually the stated justification for a design decision, so the next reader inherits a false model rather than a typo; and its conclusion is often still correct for a different surviving reason, so rewrite the reason rather than deleting the rule. Ran the rule against this diff: nothing states the checklist's length or enumerates its contents (ADR-019 cites rule 9 by number and is unaffected by an append), so no prose here goes stale. Rule 14 verified byte-identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
#1002 stamped four sinks and missed the one that mattered most. The boot banner is a direct console.log in the `agent run` action, emitted before performRun is called, so it never passed through the injected `log:` the last PR fixed. It is the first line of every seat log and the truncation boundary. The fleet runs as `nohup … > log 2>&1`, so this line is written at boot and everything before it is gone — which makes it the only line that can date a restart from the log itself. Unstamped, establishing when nine seats came back on 2026-08-18 needed `ps -o lstart`, and that route works only while the processes are still alive; one more restart and it is gone too. Also stamps the bootstrap-success line and the SIGINT "stopping..." line, the other two direct console calls in this action. Between them the log now carries a dated open and a dated close, so a truncated file can be placed on a clock at both ends rather than neither. Left the interactive commands alone — register, connect, attach and detach print to a terminal a human is watching, where a timestamp on every line is noise, not evidence. Reported-by: sprint-review Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* docs(review): a diff never shows the prose it just falsified Three times on 2026-08-18, in careful PRs by the author who knew the subject best, a change invalidated a comment it did not touch: #1001 removed the 30-minute pending delete, while models/OnboardingSilenceEpisode.ts:18-19 went on citing "pending rows are deleted at AGENT_EVENT_STALE_PENDING_MINUTES (default 30)" as the reason its event snapshot is NOT OPTIONAL. #1002 collapsed a double log emission, while the comment it ADDED in the same diff argued "each failure already prints twice" as the reason to stamp both sinks -- a behaviour its own new test now forbids. #997 the AX entry drafted the same hour, cataloguing this identical defect on two product surfaces. The third is the tell. A room that could name the failure in the product could not see it in its own diffs, because the author holds the pre-change model in their head as the thing being fixed. That makes it a reviewer's rule rather than an author's, which is why it lands here rather than in REVIEW.md. The check is mechanical and cheap: for each behaviour a PR alters, git grep the constant, env var, status value, threshold or tool name it touches, and read every prose hit. Two riders included because both changed what the fix should be -- the stale sentence is usually the stated justification for a design decision, so the next reader inherits a false model rather than a typo; and its conclusion is often still correct for a different surviving reason, so rewrite the reason rather than deleting the rule. Ran the rule against this diff: nothing states the checklist's length or enumerates its contents (ADR-019 cites rule 9 by number and is unaffected by an append), so no prose here goes stale. Rule 14 verified byte-identical. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(review): rule 15 — read the stale comment's siblings too Added by @pod-architect, from the case that earned the rule. Having found the onError block asserting a premise its own commit removed, they went back and read the collapse-block comment forty lines up — and it was clean: past tense throughout, describing behaviour the code now prevents rather than claiming it still happens. Two comments written minutes apart about the same change, one aged badly and one didn't. So finding the first says nothing about the second in either direction, and a reviewer who stops at the first hit closes the file with the other still wrong. The sweep is per-comment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs(review): rule 11 — scale re-verification to what the claim would stop Rule 11 already says to verify against the current head. Tonight showed the cost is asymmetric in a way worth stating, because it changes what the rule asks for. Five crossings in thirty minutes on one PR. Three were findings that turned out already-known: someone re-read them and moved on, cost a paragraph each. Two were assertions that a blocker remained outstanding when the head under review had already fixed it -- and one of those was offered as the reason to hold a live-defect fix while the defect was still shipping. Same staleness, opposite cost. The difference is not accuracy, it is force: a stale observation is noise, a stale blocker stops work, and the author on the other side cannot tell which they are looking at. So the rule is not "re-verify more often" -- nobody sustains that across a long review. It is: re-resolve the head before asserting anything that would hold a merge, and accept that non-blocking observations will sometimes arrive already-answered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
npm 0.1.9 and main 0.1.9 are DIFFERENT CODE. Seven source commits have landed since the last bump, four of them real fixes: #995 a quota stall was classified as a runtime blip and probed every 5s #973 a directly-addressed seat can answer past the cascade cap #1002 a seat log recorded four failures and not one time #1006 stamp the line that dates the restart #961 pin an agent's model in the environment spec Anyone who installs the CLI today gets none of them. Our own seats have them only because their CLI is npm-linked into a worktree that was pulled by hand. Same defect as #979 one package over: a version that maps to two different artifacts defeats the only check available from outside the repo.
… to nobody (#1017) * chore(release): @commonlyai/cli 0.1.10 npm 0.1.9 and main 0.1.9 are DIFFERENT CODE. Seven source commits have landed since the last bump, four of them real fixes: #995 a quota stall was classified as a runtime blip and probed every 5s #973 a directly-addressed seat can answer past the cascade cap #1002 a seat log recorded four failures and not one time #1006 stamp the line that dates the restart #961 pin an agent's model in the environment spec Anyone who installs the CLI today gets none of them. Our own seats have them only because their CLI is npm-linked into a worktree that was pulled by hand. Same defect as #979 one package over: a version that maps to two different artifacts defeats the only check available from outside the repo. * ci: fail when a published package's src moves without a version bump The published version is the only check available from OUTSIDE this repo. Ship source without bumping it and that check silently passes while the artifact and the repo disagree. Twice now: #979 @commonlyai/mcp — npm 0.3.0 and main 0.3.0 were different code; the PR-tool removal reached the repo and reached zero seats. Found months late, by unpacking the tarball. #1017 @commonlyai/cli — npm 0.1.9 and main 0.1.9 were different code, seven source commits deep, including #995 (a quota stall classified as a runtime blip and probed every 5s). Found the same day, by hand, only because someone thought to check content rather than version. Guards cli/src and commonly-mcp/src. Docs and tests are exempt — they do not require a release. Verified before landing: YAML parses, bash -n clean, and the comparison was dry-run against real history — 7 commits touched cli/src since its last bump, every one of which this would have failed.
Scoped by @sprint-review in-pod. Taking it since it had sat ~30 minutes without a branch — say the word and I'll close it in favour of yours.
Why this isn't cosmetic
The fleet runs as
nohup commonly agent run <name> > <log> 2>&1, so every line lands in one file. None carried a time.On 2026-08-18 a fleet-wide quota stall destroyed 38 queued events (#993, fixed in #1001). The seat log was the only surviving trace — the kernel rows were deleted, so nothing else recorded those turns had been attempted. And the log could not say when:
(4 consecutive),next probe in 2.0m, and no way to place either on a clock. The investigation ended up dating those events by decoding ObjectId prefixes, because the file that named them couldn't.What changes
Stamps every sink in
agent run— the bootstrap logger, its error path, the run logger, and the run error channel — with ISO-8601, so they sort lexically and line up with kernel timestamps without conversion.Both run sinks, deliberately. A spawn failure is emitted through
logandonError; they serve different contracts (the narrative line carries the[event.type]prefix, the error channel is what an embedder hooks). Stamping onlylog:would leave every failure line undated — the exact class these stamps exist for.It also makes that pair legible rather than misleading.
grep -c 'session limit'on a real seat log returned 8 for 4 failures, because each is emitted twice into the same merged stream. Read as deliveries, that putattemptsat 2 per event and the delivery cap twice as close as it was — corroborating a diagnosis that was wrong. With stamps, two identical messages at the same millisecond read as one event.Not unit-tested, and why rather than a vacuous one
These sinks are injected inside
.action(), which no test invokes. The existingcommand-registration.test.mjsprovesregisterAgentbuilds — it does not prove a callback runs, and a test assertingstamp()returns an ISO string would pin the helper while leaving "is it actually wired into the sink" exactly as unverified as it is now. That is the failure mode AX entry 28 and #997 are about, so I'd rather name the gap than paper it.Verified instead by: building the command tree with the helper in scope (
registerAgentthrows at call time on an out-of-scope reference — that's how #997's class of bug surfaces), and the full cli suite: 21 suites, 298 passed.The honest check is a seat restart and a look at the log, which belongs to whoever fast-forwards
live/main-tracking.Deliberately not in scope
The double emission itself. Removing the
log()call would drop the failure entirely for any embedder that doesn't passonError, and removingonErrorwould break the error contract — so the duplication is a wiring consequence of both sinks pointing at one terminal, not a bug at the call site. Stamps make it readable; collapsing it is a separate decision about which contract to keep.🤖 Generated with Claude Code