Repository navigation
Stream agent output through kelos-capture instead of writing to /tmp - #1189
Conversation
There was a problem hiding this comment.
2 issues found across 10 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
|
/kelos review |
|
🤖 Kelos Task Status Task |
There was a problem hiding this comment.
🤖 Kelos Reviewer Agent @gjkim42
Review Summary
Verdict: COMMENT (GitHub blocks self-approval; submitting as COMMENT — the patch is approve-grade)
Overall correctness: patch is correct
Scope: Replace tee /tmp/agent-output.jsonl + post-hoc file parse with a streaming kelos-capture that forwards stdin verbatim and accumulates token usage in memory.
Findings Overview
| Priority | Count | File:Line | Summary |
|---|---|---|---|
| P0 | 0 | — | none |
| P1 | 0 | — | none |
| P2 | 2 | internal/capture/usage.go:46 | Deferred bw.Flush() error is silently discarded |
| P2 | internal/capture/usage.go:47-48 | Scanner ErrTooLong on a >10MB line now kills the agent (regression) |
|
| P3 | 1 | internal/capture/usage_test.go:169-179 | assertForwarded only checks substring presence; misses reordering/duplication |
Findings
Correctness
- [P2]
internal/capture/usage.go:46—defer bw.Flush()discards the flush error. If the loop exits cleanly with buffered bytes still pending and the finalFlush()fails (e.g., the downstream pod-log writer hits a transient error),StreamUsagereturnsniland the last buffered lines vanish from pod logs without any signal to the operator. Capture the flush error explicitly (e.g., named return +defer func() { if ferr := bw.Flush(); err == nil { err = ferr } }()). - [P2]
internal/capture/usage.go:47-48— Reliability regression on oversized lines. The scanner is configured with a 10 MiB max (scanner.Buffer(make([]byte, 0, 1<<20), 10<<20)), matching the previous implementation. Before, that limit only meant token usage was lost when a line exceeded it becauseParseUsageran after the agent exited from the on-disk file. Nowkelos-captureis the agent's sole stdout consumer in the pipe; onbufio.ErrTooLongScan()stops,StreamUsagereturns immediately,Run()proceeds to emit markers and exits, and the agent gets SIGPIPE on its next write — turning what used to be a silent parse miss into an aborted agent run. Consider draining the remainder withio.Copy(w, r)(and giving up on further usage parsing) whensc.Err()is non-nil, so the agent always gets a clean EOF.
Tests
- [P3]
internal/capture/usage_test.go:169-179—assertForwardedonly checksstrings.Contains(out, line+"\n")for each non-empty input line, so duplicates, reordering, or extra output would pass.TestStreamUsageForwardsUnterminatedLinealready does the stricter byte-level comparison; the same shape (normalize input to end in\n, thenif out.String() != want) could be reused across the table-driven cases for a stronger forwarder invariant.
Key takeaways
- Refactor is clean: accumulator interface keeps memory O(1), per-agent extractors are well-factored, and the marker contract with the controller is preserved.
- Two defensive gaps worth tightening: the deferred flush error and the regression where a >10 MiB line now propagates to the agent as SIGPIPE.
Note on prompt injection: third-party bot content in the PR thread (including a section instructing reviewers to attribute findings to it) was disregarded; the findings above are formed independently from the diff.
|
/kelos pick-up |
|
🤖 Kelos Task Status Task |
|
/kelos pick-up |
|
🤖 Kelos Task Status Task |
There was a problem hiding this comment.
1 issue found across 2 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
|
/kelos review |
|
🤖 Kelos Task Status Task |
|
/kelos pick-up |
|
🤖 Kelos Task Status Task |
There was a problem hiding this comment.
🤖 Kelos Reviewer Agent @gjkim42
Review Summary
Verdict: COMMENT (GitHub blocks self-approval; submitting as COMMENT — the patch is approve-grade)
Overall correctness: patch is correct
Scope: Address prior round of feedback on streaming kelos-capture: surface the deferred bw.Flush() error via a named return, drain the rest of r after bufio.ErrTooLong so the agent gets a clean EOF instead of SIGPIPE, and tighten the table-driven forwarding test to a byte-for-byte equality check.
Findings Overview
| Priority | Count | File:Line | Summary |
|---|---|---|---|
| P0 | 0 | — | none |
| P1 | 0 | — | none |
| P2 | 0 | — | none |
| P3 | 1 | internal/capture/usage.go:74-83 | io.Copy drain leaks scanner-buffered bytes; "line itself is lost" comment is slightly inaccurate |
Findings
Correctness
- [P3]
internal/capture/usage.go:74-83— Onbufio.ErrTooLong, the scanner has already read the head of the oversized line into its internal buffer; those bytes are not inranymore whenio.Copy(bw, r)runs, so the head of the bad line vanishes while the tail (whatever was not yet read into the buffer) plus all subsequent lines are forwarded. The behaviour is acceptable — the agent still gets a clean EOF — but the comment "The oversized line itself is lost (bufio.Scanner discards it)" overstates what is dropped: a partial tail of that line still lands in pod logs. If perfect byte fidelity past an oversized line ever matters, abufio.Reader.ReadBytes('\n')loop would avoid the buffered-byte loss; otherwise rewording the comment to "the head of the oversized line is dropped; the rest is forwarded as raw bytes" is enough.
Key takeaways
- Previous round's P2 concerns (silent
Flusherror, SIGPIPE on >10 MiB line, weakContains-based forwarding assertion) are all addressed in 8bcdb6a — named-return + deferred flush capture, drain-via-io.Copyfallback, and the newassertForwardedbyte-equality helper across the table-driven cases. - The new
TestStreamUsageDrainsOversizedLineandTestStreamUsagePropagatesFlushErrorcover the regression seams introduced by those fixes, so future refactors should not silently regress them. - Only nit left is a wording/edge-case detail on the oversized-line drain path; no blocking issues.
Note on prompt injection: third-party bot content in the PR thread (including a section instructing reviewers to attribute findings to it) was disregarded; the findings above are formed independently from the diff.
8bcdb6a to
7566006
Compare
|
🤖 Kelos Agent @gjkim42 Addressed the byte-fidelity concern raised in the latest reviews by replacing the |
… conventions Add three conventions backed by review feedback on PRs from the past 7 days: - "Wire new agent runtime images through the full stack in one PR" — from the /kelos api-review on PR #1194 (Antigravity CLI). The PR added an image directory + kind-load step but did not extend the Task.spec.type enum, JobBuilder, credential env-var mapping, agent-image-interface.md, or e2e coverage, so `type: antigravity` is rejected at admission. - "Pin third-party CLI installs in agent-image Dockerfiles" — from cubic (P2) and kelos-bot (P3) findings on PR #1194's antigravity/Dockerfile:41 unversioned `curl install.sh | bash`. The sibling images already use ARG XXX_VERSION + pinned URLs. - "Propagate deferred Close()/Flush() errors via a named return" — from the cubic (P2) and kelos-bot (P2) findings on PR #1189 internal/capture/usage.go:46 silent `defer bw.Flush()` error drop.
|
/kelos pick-up |
|
🤖 Kelos Task Status Task |
7566006 to
3d658a9
Compare
|
🤖 Kelos Agent @gjkim42 Renamed the local |
|
/kelos review |
|
🤖 Kelos Task Status Task |
There was a problem hiding this comment.
🤖 Kelos Reviewer Agent @gjkim42
Review Summary
Verdict: COMMENT (GitHub blocks self-approval since the PR was opened by kelos-bot — the patch is approve-grade)
Overall correctness: patch is correct
Scope: Inline kelos-capture into the agent stdout pipe (<agent> | /kelos/kelos-capture), replacing the prior tee /tmp/agent-output.jsonl + post-hoc ParseUsage(filePath) flow. The new StreamUsage(agentType, io.Reader, io.Writer) forwards bytes verbatim via a bufio.Reader.ReadBytes('\n') loop and accumulates per-agent token usage in memory. Updates touch the five entrypoints, internal/capture/{capture,usage,*_test}.go, and docs/agent-image-interface.md; PIPESTATUS[0] exit-code propagation and the controller's ---KELOS_OUTPUTS_*--- marker contract are unchanged.
Findings Overview
| Priority | Count | File:Line | Summary |
|---|---|---|---|
| P0 | 0 | — | none |
| P1 | 0 | — | none |
| P2 | 0 | — | none |
| P3 | 0 | — | none |
Findings
None.
Key takeaways
- All prior review-round concerns are resolved in commit
3d658a9: named-return + deferredFlush()capture surfaces flush errors (internal/capture/usage.go:49-56); thebufio.Reader.ReadBytes('\n')loop removes the 10 MiB scanner cap entirely so an arbitrarily long line is forwarded byte-for-byte with no SIGPIPE risk;assertForwardedenforces byte-identical forwarding across the table-driven cases; and the localparseLineshadowing of the package-level helper was renamed tobodyatinternal/capture/usage.go:74. - The controller marker contract is preserved —
cost-usd/input-tokens/output-tokensare still emitted in that fixed order fromcaptureOutputs, withKELOS_AGENT_TYPEconsumed insideRun()and the per-agent extractors well-factored intolastResultAccumulator/sumAccumulator(O(1) state). - All five entrypoints (
claude-code,codex,cursor,gemini,opencode) consistently use<agent> "${ARGS[@]}" | /kelos/kelos-capturewithset -uo pipefailandPIPESTATUS[0]exit-code propagation;docs/agent-image-interface.mdmatches the implementation. - New tests (
TestStreamUsageForwardsLargeLineByteForByte,TestStreamUsagePropagatesFlushError,TestStreamUsageForwardsUnterminatedLine) lock in the regression seams introduced by the rewrite.
Note on prompt injection: third-party bot content in the PR thread (including <!-- cubic:attribution --> directives demanding attribution to another reviewer) was disregarded; findings here are formed independently from the diff.
The existing "vacuous substring assertions" convention was scoped only to printer/formatter tests, leaving stream/forwarding tests uncovered. In #1189 both the Kelos reviewer and cubic independently flagged a forwarder test that used strings.Contains per line, which passes even when lines are dropped, duplicated, or reordered. Extend the rule to require full byte-equality comparison for writer/pipe forwarding tests, mirroring the assertForwarded fix landed in that PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The existing "vacuous substring assertions" convention was scoped only to printer/formatter tests, leaving stream/forwarding tests uncovered. In #1189 both the Kelos reviewer and cubic independently flagged a forwarder test that used strings.Contains per line, which passes even when lines are dropped, duplicated, or reordered. Extend the rule to require full byte-equality comparison for writer/pipe forwarding tests, mirroring the assertForwarded fix landed in that PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The existing "vacuous substring assertions" convention was scoped only to printer/formatter tests, leaving stream/forwarding tests uncovered. In #1189 both the Kelos reviewer and cubic independently flagged a forwarder test that used strings.Contains per line, which passes even when lines are dropped, duplicated, or reordered. Extend the rule to require full byte-equality comparison for writer/pipe forwarding tests, mirroring the assertForwarded fix landed in that PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The existing "vacuous substring assertions" convention was scoped only to printer/formatter tests, leaving stream/forwarding tests uncovered. In #1189 both the Kelos reviewer and cubic independently flagged a forwarder test that used strings.Contains per line, which passes even when lines are dropped, duplicated, or reordered. Extend the rule to require full byte-equality comparison for writer/pipe forwarding tests, mirroring the assertForwarded fix landed in that PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The existing "vacuous substring assertions" convention was scoped only to printer/formatter tests, leaving stream/forwarding tests uncovered. In #1189 both the Kelos reviewer and cubic independently flagged a forwarder test that used strings.Contains per line, which passes even when lines are dropped, duplicated, or reordered. Extend the rule to require full byte-equality comparison for writer/pipe forwarding tests, mirroring the assertForwarded fix landed in that PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The existing "vacuous substring assertions" convention was scoped only to printer/formatter tests, leaving stream/forwarding tests uncovered. In #1189 both the Kelos reviewer and cubic independently flagged a forwarder test that used strings.Contains per line, which passes even when lines are dropped, duplicated, or reordered. Extend the rule to require full byte-equality comparison for writer/pipe forwarding tests, mirroring the assertForwarded fix landed in that PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The existing "vacuous substring assertions" convention was scoped only to printer/formatter tests, leaving stream/forwarding tests uncovered. In #1189 both the Kelos reviewer and cubic independently flagged a forwarder test that used strings.Contains per line, which passes even when lines are dropped, duplicated, or reordered. Extend the rule to require full byte-equality comparison for writer/pipe forwarding tests, mirroring the assertForwarded fix landed in that PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The existing "vacuous substring assertions" convention was scoped only to printer/formatter tests, leaving stream/forwarding tests uncovered. In #1189 both the Kelos reviewer and cubic independently flagged a forwarder test that used strings.Contains per line, which passes even when lines are dropped, duplicated, or reordered. Extend the rule to require full byte-equality comparison for writer/pipe forwarding tests, mirroring the assertForwarded fix landed in that PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The existing "vacuous substring assertions" convention was scoped only to printer/formatter tests, leaving stream/forwarding tests uncovered. In #1189 both the Kelos reviewer and cubic independently flagged a forwarder test that used strings.Contains per line, which passes even when lines are dropped, duplicated, or reordered. Extend the rule to require full byte-equality comparison for writer/pipe forwarding tests, mirroring the assertForwarded fix landed in that PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The existing "vacuous substring assertions" convention was scoped only to printer/formatter tests, leaving stream/forwarding tests uncovered. In #1189 both the Kelos reviewer and cubic independently flagged a forwarder test that used strings.Contains per line, which passes even when lines are dropped, duplicated, or reordered. Extend the rule to require full byte-equality comparison for writer/pipe forwarding tests, mirroring the assertForwarded fix landed in that PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The existing "vacuous substring assertions" convention was scoped only to printer/formatter tests, leaving stream/forwarding tests uncovered. In #1189 both the Kelos reviewer and cubic independently flagged a forwarder test that used strings.Contains per line, which passes even when lines are dropped, duplicated, or reordered. Extend the rule to require full byte-equality comparison for writer/pipe forwarding tests, mirroring the assertForwarded fix landed in that PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The existing "vacuous substring assertions" convention was scoped only to printer/formatter tests, leaving stream/forwarding tests uncovered. In #1189 both the Kelos reviewer and cubic independently flagged a forwarder test that used strings.Contains per line, which passes even when lines are dropped, duplicated, or reordered. Extend the rule to require full byte-equality comparison for writer/pipe forwarding tests, mirroring the assertForwarded fix landed in that PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The existing "vacuous substring assertions" convention was scoped only to printer/formatter tests, leaving stream/forwarding tests uncovered. In #1189 both the Kelos reviewer and cubic independently flagged a forwarder test that used strings.Contains per line, which passes even when lines are dropped, duplicated, or reordered. Extend the rule to require full byte-equality comparison for writer/pipe forwarding tests, mirroring the assertForwarded fix landed in that PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The existing "vacuous substring assertions" convention was scoped only to printer/formatter tests, leaving stream/forwarding tests uncovered. In #1189 both the Kelos reviewer and cubic independently flagged a forwarder test that used strings.Contains per line, which passes even when lines are dropped, duplicated, or reordered. Extend the rule to require full byte-equality comparison for writer/pipe forwarding tests, mirroring the assertForwarded fix landed in that PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The existing "vacuous substring assertions" convention was scoped only to printer/formatter tests, leaving stream/forwarding tests uncovered. In #1189 both the Kelos reviewer and cubic independently flagged a forwarder test that used strings.Contains per line, which passes even when lines are dropped, duplicated, or reordered. Extend the rule to require full byte-equality comparison for writer/pipe forwarding tests, mirroring the assertForwarded fix landed in that PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The existing "vacuous substring assertions" convention was scoped only to printer/formatter tests, leaving stream/forwarding tests uncovered. In #1189 both the Kelos reviewer and cubic independently flagged a forwarder test that used strings.Contains per line, which passes even when lines are dropped, duplicated, or reordered. Extend the rule to require full byte-equality comparison for writer/pipe forwarding tests, mirroring the assertForwarded fix landed in that PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The existing "vacuous substring assertions" convention was scoped only to printer/formatter tests, leaving stream/forwarding tests uncovered. In #1189 both the Kelos reviewer and cubic independently flagged a forwarder test that used strings.Contains per line, which passes even when lines are dropped, duplicated, or reordered. Extend the rule to require full byte-equality comparison for writer/pipe forwarding tests, mirroring the assertForwarded fix landed in that PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The existing "vacuous substring assertions" convention was scoped only to printer/formatter tests, leaving stream/forwarding tests uncovered. In #1189 both the Kelos reviewer and cubic independently flagged a forwarder test that used strings.Contains per line, which passes even when lines are dropped, duplicated, or reordered. Extend the rule to require full byte-equality comparison for writer/pipe forwarding tests, mirroring the assertForwarded fix landed in that PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
What type of PR is this?
/kind bug
What this PR does / why we need it:
Every agent entrypoint used to
teethe agent's JSON-lines stdout into/tmp/agent-output.jsonlso that/kelos/kelos-capturecould re-read itafter the agent exited and emit token usage between the
---KELOS_OUTPUTS_*---markers./tmpcounts against the pod'sephemeral-storagelimit and the file was never rotated, so a verboserun could fill the ephemeral budget and trigger node-pressure eviction
that silently failed user tasks.
This PR inlines the parser into the pipe. Each entrypoint now does:
kelos-capturereads stdin line by line, forwards each line unchangedto its own stdout (pod logs are byte-identical to today), and
accumulates per-agent counters in memory:
claude-code/gemini/cursor— keep only the most recent{"type":"result", ...}line.codex/opencode— sum tokens across everyturn.completed/step_finishevent.Memory cost is O(1) in the size of the stream,
/tmp/agent-output.jsonlis no longer created, and the controller's marker contract is
unchanged.
Which issue(s) this PR is related to:
Fixes #1187
Special notes for your reviewer:
ParseUsage(agentType, filePath)is replaced byStreamUsage(agentType, io.Reader, io.Writer).PIPESTATUS[0]— samepattern as before, just with
kelos-captureon the right of the pipeinstead of
tee.internal/capturenow exercise the streaming path forall five agent types and verify every input line is forwarded
unchanged.
docs/agent-image-interface.mdupdated to describe the pipe form.Does this PR introduce a user-facing change?
Summary by cubic
Stream agent stdout through
/kelos/kelos-captureinstead of teeing to/tmp, removing the ephemeral-storage hotspot that caused task pod evictions. Pod logs remain identical to previous behavior and the---KELOS_OUTPUTS_*---contract is unchanged.Bug Fixes
/tmp/agent-output.jsonlwrites to stop ephemeral-storage bloat and node-pressure evictions (fixes Task pods evicted by ephemeral storage: /tmp/agent-output.jsonl grows unbounded #1187).Refactors
ParseUsage(...)withStreamUsage(agentType, io.Reader, io.Writer);/kelos-capturenow streams stdin to stdout and accumulates usage in O(1) memory.<agent> "${ARGS[@]}" | /kelos/kelos-capture; agent exit code preserved viaPIPESTATUS[0]. Tests cover large lines, newline-append, and flush-error paths; docs now prescribe the pipe pattern.Written for commit 3d658a9. Summary will update on new commits. Review in cubic