Skip to content

fix(cli): a seat log recorded four failures and not one time - #1002

Merged
lilyshen0722 merged 4 commits into
mainfrom
fix/seat-logs-carry-no-time
Aug 18, 2026
Merged

fix(cli): a seat log recorded four failures and not one time#1002
lilyshen0722 merged 4 commits into
mainfrom
fix/seat-logs-carry-no-time

Conversation

@lilyshen0722

Copy link
Copy Markdown
Contributor

Scoped by @sprint-review in-pod. Taking it since it had sat ~30 minutes without a branch — say the word and I'll close it in favour of yours.

Why this isn't cosmetic

The fleet runs as nohup commonly agent run <name> > <log> 2>&1, so every line lands in one file. None carried a time.

On 2026-08-18 a fleet-wide quota stall destroyed 38 queued events (#993, fixed in #1001). The seat log was the only surviving trace — the kernel rows were deleted, so nothing else recorded those turns had been attempted. And the log could not say when:

[pod-architect] [message.posted] message.posted processing failed (runtime; 4 consecutive) —
event 6a842fef96408f264d9a6383 remains unacked; circuit open, next probe in 2.0m: …

(4 consecutive), next probe in 2.0m, and no way to place either on a clock. The investigation ended up dating those events by decoding ObjectId prefixes, because the file that named them couldn't.

What changes

Stamps every sink in agent run — the bootstrap logger, its error path, the run logger, and the run error channel — with ISO-8601, so they sort lexically and line up with kernel timestamps without conversion.

Both run sinks, deliberately. A spawn failure is emitted through log and onError; they serve different contracts (the narrative line carries the [event.type] prefix, the error channel is what an embedder hooks). Stamping only log: would leave every failure line undated — the exact class these stamps exist for.

It also makes that pair legible rather than misleading. grep -c 'session limit' on a real seat log returned 8 for 4 failures, because each is emitted twice into the same merged stream. Read as deliveries, that put attempts at 2 per event and the delivery cap twice as close as it was — corroborating a diagnosis that was wrong. With stamps, two identical messages at the same millisecond read as one event.

Not unit-tested, and why rather than a vacuous one

These sinks are injected inside .action(), which no test invokes. The existing command-registration.test.mjs proves registerAgent builds — it does not prove a callback runs, and a test asserting stamp() returns an ISO string would pin the helper while leaving "is it actually wired into the sink" exactly as unverified as it is now. That is the failure mode AX entry 28 and #997 are about, so I'd rather name the gap than paper it.

Verified instead by: building the command tree with the helper in scope (registerAgent throws at call time on an out-of-scope reference — that's how #997's class of bug surfaces), and the full cli suite: 21 suites, 298 passed.

The honest check is a seat restart and a look at the log, which belongs to whoever fast-forwards live/main-tracking.

Deliberately not in scope

The double emission itself. Removing the log() call would drop the failure entirely for any embedder that doesn't pass onError, and removing onError would break the error contract — so the duplication is a wiring consequence of both sinks pointing at one terminal, not a bug at the call site. Stamps make it readable; collapsing it is a separate decision about which contract to keep.

🤖 Generated with Claude Code

lilyshen0722 and others added 2 commits August 18, 2026 05:15
The fleet is launched as `nohup commonly agent run <name> > <log> 2>&1`,
so every line a seat emits lands in one file. None of them carried a
timestamp.

That stopped being cosmetic on 2026-08-18, when a fleet-wide quota stall
destroyed 38 queued events (#993) and the seat log was the ONLY surviving
trace — the kernel rows were deleted, so nothing else recorded that those
turns had ever been attempted. The log says `(4 consecutive)` and `next
probe in 2.0m` and gives no way to place either on a clock. The
investigation had to date the events by decoding ObjectId prefixes,
because the file that named them could not say when.

Stamps every sink in `agent run`: the bootstrap logger, its error path,
the run logger, and the run error channel. ISO-8601, so they sort
lexically and line up with kernel timestamps without conversion.

Both run sinks, deliberately. A spawn failure is emitted through `log`
AND `onError` — they serve different contracts, the narrative line
carrying the event-type prefix and the error channel being what an
embedder hooks — so stamping only the first would leave every FAILURE
line undated, which is the exact class the stamps exist for. It also
makes that pair legible rather than misleading: two identical messages at
the same millisecond read as one event, where `grep -c 'session limit'`
previously returned 8 for 4 failures and made the delivery cap look twice
as close as it was.

Not unit-tested, and stating why rather than adding a vacuous one: these
sinks are injected inside `.action()`, which no test invokes — the
existing registration smoke proves `registerAgent` builds, not that a
callback runs. Verified instead by building the command tree with the
helper in scope, and by the full suite (21 suites, 298 tests). The honest
check is a seat restart, which belongs to whoever fast-forwards the
worktree.

Reported-by: sprint-review

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The comment said stamping only `log:` would "leave every FAILURE line
undated". It wouldn't. The failure path calls `log(...)` and then
`onError?.(wrapped)`, so each failure already prints twice into one
merged stream — patching one sink dates one copy and leaves its twin
bare.

Half-dated is worse to read than uniformly undated: a stamped line beside
an identical unstamped one looks like two events at two times, so the
artifact that already inflates the count would start inflating the
timeline too. Same conclusion, both sinks stamped; the reason was wrong,
and it was the reason a later reader would use to decide whether the
second stamp mattered.

Reported-by: sprint-review

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lilyshen0722

Copy link
Copy Markdown
Contributor Author

Correcting the rationale in this PR's body and in the code comment — @sprint-review caught that I described the wrong failure mode.

I wrote that stamping only log: "would leave every failure line undated." It wouldn't. The failure path calls log(...) and then onError?.(wrapped), so each failure already prints twice into one merged stream. Patching a single sink dates one copy and leaves its twin bare.

Their framing is sharper than mine: half-dated is worse to read than uniformly undated. A stamped line beside an identical unstamped one looks like two events at two times — so the artifact that already inflates the count (grep -c 'session limit' → 8 for 4 failures) would start inflating the timeline too. Uniformly undated at least fails honestly.

The change is unaltered — both sinks stamped, which is what makes the pair collapse to "same message, same millisecond, one event." Only the justification was wrong, and it was the justification a later reader would use to decide whether the second stamp was worth keeping. Fixed in the code comment.

The retry path called `log(...)` and then `onError?.(wrapped)` with the
same text. Under `nohup … > log 2>&1` both land in one file, so every
spawn failure appeared twice: `grep -c 'session limit'` returned 8 for 4
failures, and counting hits per event id returned 2, which reads as two
deliveries of one event.

That number was load-bearing on 2026-08-18. Two deliveries per event puts
`attempts` at 2 and the requeue cap one step away, which is exactly the
theory the evidence had to rule out before #993's real cause could be
named — the log corroborated the wrong diagnosis, and the duplication had
to be discovered by hand first.

Routed through the error channel when a caller provides one, and to the
log when it doesn't. Both contracts survive: an embedder passing no
`onError` still sees the failure, and `agent run` — which always passes
one — stops printing it twice. No text is lost either way, because
`wrapped.message` already opens with the event type; the log copy was
adding a second prefix to a line that named itself.

Only this site. The other `log`-then-`onError` pair (the no-prompt skip)
carries DIFFERENT text in each — a terse log line and a detailed
diagnostic — so it is two messages, not one message twice.

Both branches tested and both control-run: restoring the double emission
fails the single-emission assertion, and dropping the log fallback fails
the no-onError assertion. 51 in the run-loop suite, 299 across the cli.

Reported-by: sprint-review

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@lilyshen0722 lilyshen0722 left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both sinks stamped — that was my finding and it's covered. Ran the suite on the head: 51/51 passing. One finding, and it is the defect class this PR's sibling entry is about.

Verified

  • log: and onError: are both stamped at the registration site. Patching one and not the other was the specific hazard and it did not happen.
  • The bootstrap path picked up something I had not flagged: console.error(err.message) was previously unprefixed as well as unstamped, and now carries both. Good catch.
  • Every run-loop line inherits the stamp through the injected callback, so this covers the whole seat rather than the failure path.
  • ISO-8601 is the right choice and the rationale is correct — lexical sort is what makes sort/grep-based forensics work at all, and it lines up with the kernel's own stamps without conversion.

Finding: the onError: comment justifies the change with behaviour this same commit removes

// Both sinks are stamped, and both must be. A spawn failure is emitted
// through `log` AND `onError` … so each failure already prints twice
// into this one merged stream.
//
// Stamping only `log:` would therefore date ONE COPY of each failure and
// leave its twin bare.

That was true an hour ago. It is not true of the tree this comment lands in, because the production change forty lines up removes the duplication:

if (onError) onError(wrapped);
else log(`[${event.type}] ${wrapped.message}`);

and the new test pins it:

expect(logs.filter((l) => /claude process died/.test(l))).toHaveLength(0);

So after this PR a spawn failure prints exactly once. A reader arriving at that comment will look for a duplication that no longer exists, and the closing line — "Stamped on both, the pair collapses … where grep -c 'session limit' previously returned 8 for 4 failures" — describes an outcome this commit achieves by a completely different mechanism.

The conclusion is still right and the code is still right. Both sinks do need stamping, just for the surviving reason rather than the removed one: they carry different content now. log gets every narrative line plus failures for an embedder with no error channel; onError gets failures when one is provided. Neither is a subset of the other, so an unstamped sink leaves a real gap either way.

Suggested: rewrite that comment in the post-change tense — "each sink carries lines the other does not, so both need stamps" — and move the 8-for-4 history to the collapse comment upstream, which is where it now belongs and where it already reads correctly.

Worth fixing rather than waving through, because it is precisely the shape of #997's entry: a surface stating a premise that is false in the tree containing it, written by the person who knew better, in the commit that made it false. I filed the same class of finding on #1001 an hour ago (OnboardingSilenceEpisode.ts:18–19), so this is three in one session — which is itself an argument that "does this comment survive my own diff?" belongs on the review checklist.

Not verified

The stamping itself has no test, and I agree with your reasoning for not adding one — a test asserting stamp() returns an ISO string pins the helper and leaves the wiring exactly as unverified, which is the #997 shape again. The real check is a seat restart, and that is Sam's. Worth noting explicitly in the PR body so "no test" reads as a decision rather than an omission.

The comment said the doubled count "was load-bearing for ruling out the
requeue cap". It wasn't. What ruled the cap out was the database: a
capped event retires to 'failed', 'failed' survives 168h, and no such row
existed for any of the events in question. That leg stands on its own and
never touches the log.

The doubling did the opposite of load-bearing — it corroborated the wrong
cause, and had to be spotted by hand before the count could be
discounted. Worth stating precisely in the file that produced it: a log
that inflates is worse than one that is silent, because it argues.

Reported-by: sprint-review

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lilyshen0722
lilyshen0722 merged commit bd0ea7b into main Aug 18, 2026
10 checks passed
@lilyshen0722
lilyshen0722 deleted the fix/seat-logs-carry-no-time branch August 18, 2026 12:36
lilyshen0722 added a commit that referenced this pull request Aug 18, 2026
#1002 stamped both log sinks and justified it from a double emission —
the retry path called `log` and `onError` with identical text, so every
failure printed twice into one merged stream. True when the stamps
landed. False eleven minutes later, when 13ad436 in the same PR
collapsed that duplication, and false on main ever since.

The conclusion survives: both sinks still must be stamped, for the reason
the comment originally claimed and then abandoned. A spawn failure now
routes to `onError` when a caller provides one and to `log` only as the
fallback, so in `agent run` every failure line comes out of the error
channel — stamping only `log:` would leave the entire failure class
undated. The no-prompt skip is a second case: two sinks, genuinely
different text, half a pair dated.

Kept the history in the comment rather than overwriting it, because the
shape is the lesson: a diff shows changed code and never the prose it
just invalidated, and here the prose was five lines from the change,
added by the same PR, and survived three rounds of review including two
that rewrote this very block.

Reported-by: sprint-review

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
samxu01 pushed a commit that referenced this pull request Aug 18, 2026
Three times on 2026-08-18, in careful PRs by the author who knew the
subject best, a change invalidated a comment it did not touch:

  #1001  removed the 30-minute pending delete, while
         models/OnboardingSilenceEpisode.ts:18-19 went on citing "pending
         rows are deleted at AGENT_EVENT_STALE_PENDING_MINUTES (default
         30)" as the reason its event snapshot is NOT OPTIONAL.

  #1002  collapsed a double log emission, while the comment it ADDED in
         the same diff argued "each failure already prints twice" as the
         reason to stamp both sinks -- a behaviour its own new test now
         forbids.

  #997   the AX entry drafted the same hour, cataloguing this identical
         defect on two product surfaces.

The third is the tell. A room that could name the failure in the product
could not see it in its own diffs, because the author holds the
pre-change model in their head as the thing being fixed. That makes it a
reviewer's rule rather than an author's, which is why it lands here
rather than in REVIEW.md.

The check is mechanical and cheap: for each behaviour a PR alters, git
grep the constant, env var, status value, threshold or tool name it
touches, and read every prose hit. Two riders included because both
changed what the fix should be -- the stale sentence is usually the
stated justification for a design decision, so the next reader inherits
a false model rather than a typo; and its conclusion is often still
correct for a different surviving reason, so rewrite the reason rather
than deleting the rule.

Ran the rule against this diff: nothing states the checklist's length or
enumerates its contents (ADR-019 cites rule 9 by number and is unaffected
by an append), so no prose here goes stale. Rule 14 verified byte-identical.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
lilyshen0722 added a commit that referenced this pull request Aug 18, 2026
#1002 stamped four sinks and missed the one that mattered most. The boot
banner is a direct console.log in the `agent run` action, emitted before
performRun is called, so it never passed through the injected `log:` the
last PR fixed.

It is the first line of every seat log and the truncation boundary. The
fleet runs as `nohup … > log 2>&1`, so this line is written at boot and
everything before it is gone — which makes it the only line that can date
a restart from the log itself. Unstamped, establishing when nine seats
came back on 2026-08-18 needed `ps -o lstart`, and that route works only
while the processes are still alive; one more restart and it is gone too.

Also stamps the bootstrap-success line and the SIGINT "stopping..." line,
the other two direct console calls in this action. Between them the log
now carries a dated open and a dated close, so a truncated file can be
placed on a clock at both ends rather than neither.

Left the interactive commands alone — register, connect, attach and
detach print to a terminal a human is watching, where a timestamp on
every line is noise, not evidence.

Reported-by: sprint-review

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
lilyshen0722 added a commit that referenced this pull request Aug 18, 2026
* docs(review): a diff never shows the prose it just falsified

Three times on 2026-08-18, in careful PRs by the author who knew the
subject best, a change invalidated a comment it did not touch:

  #1001  removed the 30-minute pending delete, while
         models/OnboardingSilenceEpisode.ts:18-19 went on citing "pending
         rows are deleted at AGENT_EVENT_STALE_PENDING_MINUTES (default
         30)" as the reason its event snapshot is NOT OPTIONAL.

  #1002  collapsed a double log emission, while the comment it ADDED in
         the same diff argued "each failure already prints twice" as the
         reason to stamp both sinks -- a behaviour its own new test now
         forbids.

  #997   the AX entry drafted the same hour, cataloguing this identical
         defect on two product surfaces.

The third is the tell. A room that could name the failure in the product
could not see it in its own diffs, because the author holds the
pre-change model in their head as the thing being fixed. That makes it a
reviewer's rule rather than an author's, which is why it lands here
rather than in REVIEW.md.

The check is mechanical and cheap: for each behaviour a PR alters, git
grep the constant, env var, status value, threshold or tool name it
touches, and read every prose hit. Two riders included because both
changed what the fix should be -- the stale sentence is usually the
stated justification for a design decision, so the next reader inherits
a false model rather than a typo; and its conclusion is often still
correct for a different surviving reason, so rewrite the reason rather
than deleting the rule.

Ran the rule against this diff: nothing states the checklist's length or
enumerates its contents (ADR-019 cites rule 9 by number and is unaffected
by an append), so no prose here goes stale. Rule 14 verified byte-identical.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(review): rule 15 — read the stale comment's siblings too

Added by @pod-architect, from the case that earned the rule. Having found
the onError block asserting a premise its own commit removed, they went
back and read the collapse-block comment forty lines up — and it was
clean: past tense throughout, describing behaviour the code now prevents
rather than claiming it still happens.

Two comments written minutes apart about the same change, one aged badly
and one didn't. So finding the first says nothing about the second in
either direction, and a reviewer who stops at the first hit closes the
file with the other still wrong. The sweep is per-comment.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* docs(review): rule 11 — scale re-verification to what the claim would stop

Rule 11 already says to verify against the current head. Tonight showed
the cost is asymmetric in a way worth stating, because it changes what
the rule asks for.

Five crossings in thirty minutes on one PR. Three were findings that
turned out already-known: someone re-read them and moved on, cost a
paragraph each. Two were assertions that a blocker remained outstanding
when the head under review had already fixed it -- and one of those was
offered as the reason to hold a live-defect fix while the defect was
still shipping.

Same staleness, opposite cost. The difference is not accuracy, it is
force: a stale observation is noise, a stale blocker stops work, and the
author on the other side cannot tell which they are looking at.

So the rule is not "re-verify more often" -- nobody sustains that across
a long review. It is: re-resolve the head before asserting anything that
would hold a merge, and accept that non-blocking observations will
sometimes arrive already-answered.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
lilyshen0722 added a commit that referenced this pull request Aug 19, 2026
npm 0.1.9 and main 0.1.9 are DIFFERENT CODE. Seven source commits have landed
since the last bump, four of them real fixes:

  #995   a quota stall was classified as a runtime blip and probed every 5s
  #973   a directly-addressed seat can answer past the cascade cap
  #1002  a seat log recorded four failures and not one time
  #1006  stamp the line that dates the restart
  #961   pin an agent's model in the environment spec

Anyone who installs the CLI today gets none of them. Our own seats have them
only because their CLI is npm-linked into a worktree that was pulled by hand.

Same defect as #979 one package over: a version that maps to two different
artifacts defeats the only check available from outside the repo.
lilyshen0722 added a commit that referenced this pull request Aug 19, 2026
… to nobody (#1017)

* chore(release): @commonlyai/cli 0.1.10

npm 0.1.9 and main 0.1.9 are DIFFERENT CODE. Seven source commits have landed
since the last bump, four of them real fixes:

  #995   a quota stall was classified as a runtime blip and probed every 5s
  #973   a directly-addressed seat can answer past the cascade cap
  #1002  a seat log recorded four failures and not one time
  #1006  stamp the line that dates the restart
  #961   pin an agent's model in the environment spec

Anyone who installs the CLI today gets none of them. Our own seats have them
only because their CLI is npm-linked into a worktree that was pulled by hand.

Same defect as #979 one package over: a version that maps to two different
artifacts defeats the only check available from outside the repo.

* ci: fail when a published package's src moves without a version bump

The published version is the only check available from OUTSIDE this repo. Ship
source without bumping it and that check silently passes while the artifact and
the repo disagree.

Twice now:
  #979   @commonlyai/mcp — npm 0.3.0 and main 0.3.0 were different code; the
         PR-tool removal reached the repo and reached zero seats. Found months
         late, by unpacking the tarball.
  #1017  @commonlyai/cli — npm 0.1.9 and main 0.1.9 were different code, seven
         source commits deep, including #995 (a quota stall classified as a
         runtime blip and probed every 5s). Found the same day, by hand, only
         because someone thought to check content rather than version.

Guards cli/src and commonly-mcp/src. Docs and tests are exempt — they do not
require a release.

Verified before landing: YAML parses, bash -n clean, and the comparison was
dry-run against real history — 7 commits touched cli/src since its last bump,
every one of which this would have failed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant