Skip to content

feat(#7414): pass pi tool call ids, arguments and results through - #9147

Merged
waynesun09 merged 5 commits into
fullsend-ai:mainfrom
AshwinUgale:feat/7414-pi-tool-ids-results
Nov 6, 2026
Merged

waynesun09 merged 5 commits into
fullsend-ai:mainfrom
AshwinUgale:feat/7414-pi-tool-ids-results

Conversation

@AshwinUgale

@AshwinUgale AshwinUgale commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

Summary

pi runs emit tool calls with no id and no result, so they have no execute_tool spans and no tool I/O at Level 3. EM-002 also skips them: review runs on pi, and all 405 of its jobs were skipped in the sample on #8495. This passes the ids, arguments and results through. This is the pi half; codex follows in a separate PR, as agreed on the issue.

Related Issue

Part of #7414

Changes

  • Parser (pi_progress.go): at tool_execution_end, emit a ToolUseEvent with the call's id and arguments, then a ToolResultEvent with the result text and isError. One use event per call, so fullsend.tool_calls is unchanged. Successful output still stays out of Summary.
  • Span timing: the pi parser emits a call's event at tool_execution_end, because a failed call's summary needs the result, so ToolUseEvent.StartedAt records when tool_execution_start was read and the tracker starts the span there. Parsers that don't set it behave as before.
  • End with no start: the use event carries no id, so the result becomes a fullsend.tool.unmatched span, as on Claude. A run whose start line exceeded 1 MiB will fail EM-002, as on Claude.
  • Recorded arguments: recordedArguments is keyed by Claude's tool names, so pi's arguments would have been dropped. This adds bash, read, write, edit, grep, find, ls and Task, checked against pi 1.5.0's schemas. write and edit bodies and Agent and Task prompts are not recorded.
  • Docs: runtime matrix, tracing guides, eval-measurements.md, architecture.md, the event.go, tool_spans.go and runhealth.go comments, and new dated annotations in ADR 0050 and ADR 0108.

Fix found along the way

The extended redaction test found that a failed call's summary fell back to the raw result JSON when the result had an undocumented shape, and JSON escapes can hide a secret from the redactor. This was already on main. A result object with no content list is now empty and unmarked, matching pi's content ?? []. Any other value that won't decode, such as an array, is dropped and marked truncated, as on Claude. piResultSummary and the new result text share one flatten. A failed call whose result is empty or only an image now gets no text appended, where it used to show raw JSON.

For reviewers

  • runtime.ParsePiStream is a thin export so the cli redaction test can run the real parser in front of the collector. Simulating its events would be weaker, but I can change it if you prefer.
  • Task in recordedArguments applies to any tool with that name, including on Claude. It keeps the same members as Agent.

Acceptance items

  • toolCallId and result shape: confirmed against a live pi output.jsonl from the review of feat(#6842): record forge and CI pointers on the root run span #9011 (run 41606947989). testdata/pi/tool_calls.ndjson is cut down from it.
  • tool_result hook: in pi 1.5.0's agent-loop.ts, tool_execution_end is emitted after afterToolCall, so the result is what a hook left. This comes from reading the source, not a live run.

Not covered

  • Lines over 1 MiB are still skipped, so a call whose end line is that long gets no span.
  • A pi call cut off by a kill or timeout: the parser holds a call's event until tool_execution_end, so it has no span and doesn't count as unanswered. Emitting these at end of stream would change fullsend.tool_calls, so that's left for a follow-up.
  • Tool calls inside pi sub-agents: separate processes, not on the parent's stream.
  • codex.

Testing

  • TestRedaction_NoSecretSurvivesAnyWrapping extended with 9 surfaces that run pi wire lines through ParsePiStream, 8 into the collector and 1 into the span tracker
  • New parser tests: the live capture, no start seen, no call id, interleaved calls, and result flattening
  • New span timing and recordedArguments tests, plus two that run the pi parser into the span tracker (no start gives an unmatched span; start and end give a normal one)
  • internal/runtime and internal/evalmeasure pass; go vet and gofmt are clean
  • internal/cli passes except 4 vendor tests that fail the same way on main in my local setup (a cross-compiled test binary cannot locate the source checkout)
  • No live pi run yet with this change. After merge, EM-002 will start scoring pi runs, which could surface failures not seen on pi so far.

Checklist

  • PR title follows Conventional Commits (correct type, ! for breaking changes)
  • Commits are signed off (DCO) — human and human-directed agent sessions only

…through

The pi parser emitted id-less tool calls with no results, so pi runs had
no execute_tool spans and no tool I/O at Level 3. Emit the call id,
arguments and result, and start the span when tool_execution_start was
read, since pi reports a call once it has finished. List pi's tools in
recordedArguments, and cover its parser in the redaction invariant test.

That test also found the failed-call summary falling back to raw JSON,
whose escapes can hide a secret from the redactor. Results in a shape pi
doesn't document are now dropped and marked.

Signed-off-by: Ashwin Govind Ugale <ugaleashwin@gmail.com>
@AshwinUgale
AshwinUgale requested a review from a team as a code owner November 6, 2026 10:07
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown

E2E tests did not run

E2E tests run automatically for pull requests authored by users with write access or higher.

For other contributors, someone with write access must add the ok-to-test label after the latest push.

Reason: unauthorized

See E2E testing guide for details.

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Capture pi tool call IDs, arguments, results, and spans

✨ Enhancement 🐞 Bug fix 🧪 Tests 📝 Documentation 🕐 40+ Minutes

Grey Divider

AI Description

• Pass pi tool IDs, arguments, and results into tracing and Level 3 content capture.
• Start pi tool spans when calls begin, enabling run-health scoring for observable pi calls.
• Drop undocumented result shapes instead of exposing raw JSON that can evade redaction.
Diagram

graph TD
  P["pi stream"] --> R["pi parser"] --> E["Tool events"] --> T["Span tracker"] --> S["Tool spans"] --> H["Run-health scorer"]
  E --> C["Content collector"] --> M["Output messages"]
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Emit tool use at execution start
  • ➕ Spans would open as soon as pi reports a call.
  • ➖ Changes the existing end-of-call reporting flow and requires separate handling to append failure text to the call summary.

Recommendation: Keep end-of-call emission with a captured start timestamp. It preserves one tool-use event per completed call and existing failure summaries while recovering the call's span timing; emitting at start would require a broader reporting change.

Files changed (17) +396 / -55

Enhancement (4) +116 / -39
content_collector.goAllowlist pi tool argument members +11/-0

Allowlist pi tool argument members

• Adds pi built-in tool names and Task to the recorded-arguments map. Body, edit, and prompt fields remain outside the captured argument set.

internal/cli/content_collector.go

tool_spans.goHonor parser-provided tool start times +12/-7

Honor parser-provided tool start times

• Starts a span at ToolUseEvent.StartedAt when supplied, preserving receipt-time behavior for other parsers and unmatched results.

internal/cli/tool_spans.go

event.goAdd optional tool-call start timestamp +12/-5

Add optional tool-call start timestamp

• Adds StartedAt to ToolUseEvent and updates event contracts to reflect pi's IDs, arguments, and results.

internal/runtime/event.go

pi_progress.goEmit correlated pi tool calls and safe results +81/-27

Emit correlated pi tool calls and safe results

• Retains arguments and start time until each call ends, then emits one tool-use event followed by its correlated result. Flattens documented text results for both capture and summaries, dropping and marking undocumented shapes rather than falling back to raw JSON.

internal/runtime/pi_progress.go

Tests (5) +248 / -0
content_collector_test.goTest pi argument capture boundaries +32/-0

Test pi argument capture boundaries

• Checks retained arguments across pi tools and verifies that write bodies, edits, prompts, and unlisted extension-tool inputs are not recorded.

internal/cli/content_collector_test.go

redaction_invariant_test.goExercise pi parsing in redaction invariants +57/-0

Exercise pi parsing in redaction invariants

• Feeds pi wire records through the real parser and collector to test argument, result, malformed-shape, and missing-start surfaces. Also checks redaction of pi call IDs in tool spans.

internal/cli/redaction_invariant_test.go

tool_spans_test.goTest explicit and fallback span timing +23/-0

Test explicit and fallback span timing

• Verifies that a supplied start timestamp sets span start time and that zero timestamps continue to use receipt time.

internal/cli/tool_spans_test.go

pi_progress_test.goCover pi tool calls and result edge cases +125/-0

Cover pi tool calls and result edge cases

• Tests the live-capture fixture, missing starts and IDs, interleaved calls, and result flattening. Checks that successful output stays out of call summaries and unsupported result content is marked or dropped.

internal/runtime/pi_progress_test.go

tool_calls.ndjsonAdd representative pi tool-call fixture +11/-0

Add representative pi tool-call fixture

• Provides shortened tool execution records derived from a live pi run, including successful calls, a failure, and an Agent call.

internal/runtime/testdata/pi/tool_calls.ndjson

Documentation (8) +32 / -16
0050-distributed-tracing-instrumentation.mdAnnotate pi tool I/O support +5/-0

Annotate pi tool I/O support

• Adds a dated note that pi tool calls now carry IDs and arguments alongside tool responses, with pi-specific recorded-argument entries.

docs/ADRs/0050-distributed-tracing-instrumentation.md

0108-tool-call-span-topology.mdDocument pi tool-span timing +7/-0

Document pi tool-span timing

• Records that pi now produces execute-tool spans beginning when the parser reads each call's start. Clarifies that parent streams do not contain sub-agent calls.

docs/ADRs/0108-tool-call-span-topology.md

architecture.mdInclude pi in tool-span topology +1/-1

Include pi in tool-span topology

• Updates the architecture overview to identify pi, alongside Claude Code, as a source of ID-bearing tool calls.

docs/architecture.md

tracing.mdUpdate tracing guidance for pi events +4/-4

Update tracing guidance for pi events

• Clarifies that pi now supplies call IDs, arguments, and result events to the span tracker and content collector; codex remains unsupported.

docs/guides/dev/tracing.md

distributed-tracing.mdDescribe pi capture and argument filtering +11/-7

Describe pi capture and argument filtering

• Extends Level 3 and tool-span documentation to pi and lists the pi argument members eligible for capture. Keeps codex limitations explicit.

docs/guides/infrastructure/distributed-tracing.md

eval-measurements.mdUpdate tool-I/O scoring guidance +1/-1

Update tool-I/O scoring guidance

• States that tool-I/O-based measurements can cover Claude and pi, while scorers should still skip codex when that data is required.

docs/guides/infrastructure/eval-measurements.md

runtimes.mdRevise pi runtime capability matrix +2/-2

Revise pi runtime capability matrix

• Lists correlated pi tool content and parent-process tool spans, including their start timing and sub-agent boundary.

docs/runtimes.md

runhealth.goClarify pi run-health eligibility +1/-1

Clarify pi run-health eligibility

• Updates the scorer comment to reflect that pi can now emit execute-tool spans; codex still cannot.

internal/evalmeasure/runhealth.go

@fullsend-ai-review

fullsend-ai-review Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 10:09 AM UTC · Completed 10:20 AM UTC

Commit: d1fad50 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $3.06

@qodo-code-review

qodo-code-review Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

Grey Divider


Remediation recommended

1. Some pi results lose their truncation marker ✗ Dismissed
Description
piResultText can return empty text with partial=true, but the collector treats that partial-only
response as having no content and drops it. A successful call returning only an image block or an
undocumented result shape therefore has a tool call in Level 3 with neither a response part nor an
indication that its result was discarded.

2. Pi hook result ordering was never tested live ✗ Dismissed
Description
The new comment in parsePiStream states that pi emits tool_execution_end after its
afterToolCall hooks, so result is what a tool_result hook left, but the PR description says
this comes from reading agent-loop.ts, not from a live run. If a hook rewrites output after this
event is emitted, the Level 3 tool_call_response text and failed-call summaries would carry the
wrong content, and the comment would mislead later maintainers.

Grey Divider

Context sources
✅ Linked PRs:
  🌐 PR #9011
✅ Compliance rules (platform): 96 rules
✅ Skills: 5 invoked
  writing-user-docs
  writing-adrs
  check-patch-coverage
  cutting-releases
  adr-corner
✅ Cross-repo context — repo relationships
  Explored: repo: fullsend-ai/agents (sha: 7dbe6b7e) — View relationship
  Explored: repo: fullsend-ai/mlflow-assessments (sha: 7056b1d1) — View relationship
✅ REVIEW.md
Review mode: Auto: ⚖️ Balanced: Security-sensitive redaction and telemetry content capture across several logic paths; not clearly extended-dense.

Grey Divider

Tip of the day
💡 Did you know, you can switch off images and animations for a plain-text comment

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread internal/runtime/pi_progress.go Resolved
Comment thread internal/runtime/pi_progress.go Resolved
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Site preview

Preview: https://03bc1a61-site.fullsend-ai.workers.dev

Commit: ab374d1414e5193012a1ff99941b82634d50cc16

@fullsend-ai-review fullsend-ai-review Bot added the risk/moderate PR risk: moderate label Nov 6, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Risk Assessment: moderate (2/5)

Details

Moderate 19-file telemetry change with no protected paths or dependency updates, accompanied by comprehensive tests and aligned with agreed issue scope, preserving the moderate score of 2 following an incremental commit to existing files despite elevated historical churn in touched telemetry and documentation files.

Previous run

Risk Assessment: moderate (2/5)

Details

Moderate 19-file telemetry change with no protected paths or dependency updates, accompanied by comprehensive tests and aligned with agreed issue scope, preserving a moderate score of 2 despite elevated historical churn in touched telemetry and documentation files.

Previous run (2)

Risk Assessment: moderate (2/5)

Details

Moderate 19-file telemetry change with no protected paths or dependency updates, accompanied by comprehensive tests and aligned with agreed issue scope, preserving a moderate score of 2 despite elevated historical churn in touched telemetry and documentation files.

Previous run (3)

Risk Assessment: moderate (2/5)

Details

Moderate 19-file change (increased from 17 via low-risk comments and documentation edits within the same 11–25 file bucket) with no protected paths or dependency updates, accompanied by comprehensive tests and aligned with agreed issue scope, preserving the moderate score of 2 despite elevated historical churn in telemetry and documentation files.

Previous run (4)

Risk Assessment: moderate (2/5)

Details

Moderate 17-file change with no protected paths or dependency updates, accompanied by comprehensive tests and aligned with agreed issue scope, despite elevated historical churn in telemetry and documentation files.

@fullsend-ai-review

fullsend-ai-review Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Review

Findings

Low

  • [api-contract] internal/runtime/pi_progress.go:613 — Earlier low-severity observation in the tool_execution_end handling, carried forward. This push did not touch that code, so it stays open.
  • [api-contract] internal/runtime/pi_progress.go:610 — Earlier low-severity observation in the tool_execution_end handling, carried forward. This push did not touch that code, so it stays open.
  • [api-contract] internal/runtime/pi_progress.go:597 — Earlier low-severity observation in the tool_execution_end handling, carried forward. This push did not touch that code, so it stays open.

The latest commit changed piResultText so a pi result object with no content list is empty and unmarked. It added matching tests, redaction-test cases and a docs update. The review found no new issues in it.

Previous run

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJhcHByb3ZlIiwiZmluZGluZ3MiOlt7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJhcGktY29udHJhY3QiLCJmaWxlIjoiaW50ZXJuYWwvcnVudGltZS9waV9wcm9ncmVzcy5nbyIsImlkIjoiZl85MTg3ZTFlZWU2YzA2Yzg1IiwibGluZSI6NjExLCJhY3Rpb25hYmxlIjpmYWxzZSwiZGltZW5zaW9uIjoiY29ycmVjdG5lc3MifSx7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJhcGktY29udHJhY3QiLCJmaWxlIjoiaW50ZXJuYWwvcnVudGltZS9waV9wcm9ncmVzcy5nbyIsImlkIjoiZl84Y2E2ZWM0MjNhNDRkNGFkIiwibGluZSI6NjA4LCJhY3Rpb25hYmxlIjpmYWxzZSwiZGltZW5zaW9uIjoiY29ycmVjdG5lc3MifSx7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJhcGktY29udHJhY3QiLCJmaWxlIjoiaW50ZXJuYWwvcnVudGltZS9waV9wcm9ncmVzcy5nbyIsImlkIjoiZl9lZWVkNGYyMzkyMTQwNTQyIiwibGluZSI6NTk1LCJhY3Rpb25hYmxlIjpmYWxzZSwiZGltZW5zaW9uIjoiY29ycmVjdG5lc3MifSx7InNldmVyaXR5IjoibWVkaXVtIiwiY2F0ZWdvcnkiOiJhcGktY29udHJhY3QiLCJmaWxlIjoiaW50ZXJuYWwvcnVudGltZS9waV9wcm9ncmVzcy5nbyIsImlkIjoiZl83ZjkzMDNjZWU2MmRmNTVlIiwibGluZSI6NjAyLCJhY3Rpb25hYmxlIjp0cnVlLCJkaW1lbnNpb24iOiJjb3JyZWN0bmVzcyJ9LHsic2V2ZXJpdHkiOiJtZWRpdW0iLCJjYXRlZ29yeSI6InN0YWxlLWRvYyIsImZpbGUiOiJkb2NzL2d1aWRlcy91c2VyL2hvdy10by1lbWl0LXRyYWNlcy5tZCIsImlkIjoiZl8xMGRkZWRjMDg2ODliZTI5IiwibGluZSI6MTU0LCJhY3Rpb25hYmxlIjp0cnVlLCJkaW1lbnNpb24iOiJkb2NzLWN1cnJlbmN5In0seyJzZXZlcml0eSI6Im1lZGl1bSIsImNhdGVnb3J5Ijoic3RhbGUtZG9jIiwiZmlsZSI6ImRvY3MvcHJvYmxlbXMvb3BlcmF0aW9uYWwtb2JzZXJ2YWJpbGl0eS5tZCIsImlkIjoiZl81MTJiMzg3NzVhYmMxYWQ2IiwibGluZSI6MjE1LCJhY3Rpb25hYmxlIjp0cnVlLCJkaW1lbnNpb24iOiJkb2NzLWN1cnJlbmN5In0seyJzZXZlcml0eSI6Im1lZGl1bSIsImNhdGVnb3J5Ijoic3RhbGUtZG9jIiwiZmlsZSI6ImRvY3MvZ3VpZGVzL2luZnJhc3RydWN0dXJlL2V2YWwtbWVhc3VyZW1lbnRzLm1kIiwiaWQiOiJmXzJkNjcxMjAwY2NkZTNiNWQiLCJsaW5lIjoxMTgsImFjdGlvbmFibGUiOnRydWUsImRpbWVuc2lvbiI6ImRvY3MtY3VycmVuY3kifSx7InNldmVyaXR5IjoibWVkaXVtIiwiY2F0ZWdvcnkiOiJzdGFsZS1kb2MiLCJmaWxlIjoiZG9jcy9ndWlkZXMvZGV2L3RyYWNpbmcubWQiLCJpZCI6ImZfYmVjOWEwYzg1NjY4NzU4YSIsImxpbmUiOjE2NCwiYWN0aW9uYWJsZSI6dHJ1ZSwiZGltZW5zaW9uIjoiZG9jcy1jdXJyZW5jeSJ9LHsic2V2ZXJpdHkiOiJsb3ciLCJjYXRlZ29yeSI6InN0YWxlLWRvYyIsImZpbGUiOiJkb2NzL2d1aWRlcy9pbmZyYXN0cnVjdHVyZS9kaXN0cmlidXRlZC10cmFjaW5nLm1kIiwiaWQiOiJmX2VlNDYwZDg2YWUyOThjMjEiLCJsaW5lIjoxMDMsImFjdGlvbmFibGUiOmZhbHNlLCJkaW1lbnNpb24iOiJkb2NzLWN1cnJlbmN5In1dLCJkaXNwb3NpdGlvbnMiOlt7ImlkIjoiZl9lZWVkNGYyMzkyMTQwNTQyIiwic3RhdHVzIjoib3BlbiJ9LHsiaWQiOiJmXzdmOTMwM2NlZTYyZGY1NWUiLCJzdGF0dXMiOiJyZXNvbHZlZF9ieV9jaGFuZ2UifSx7ImlkIjoiZl8xMGRkZWRjMDg2ODliZTI5Iiwic3RhdHVzIjoicmVzb2x2ZWRfYnlfY2hhbmdlIn0seyJpZCI6ImZfNTEyYjM4Nzc1YWJjMWFkNiIsInN0YXR1cyI6InJlc29sdmVkX2J5X2NoYW5nZSJ9LHsiaWQiOiJmXzJkNjcxMjAwY2NkZTNiNWQiLCJzdGF0dXMiOiJyZXNvbHZlZF9ieV9jaGFuZ2UifSx7ImlkIjoiZl9iZWM5YTBjODU2Njg3NThhIiwic3RhdHVzIjoicmVzb2x2ZWRfYnlfY2hhbmdlIn0seyJpZCI6ImZfZWU0NjBkODZhZTI5OGMyMSIsInN0YXR1cyI6InJlc29sdmVkX2J5X2NoYW5nZSJ9XX0= -->

Review

Findings

Low

  • [api-contract] internal/runtime/pi_progress.go:611 — The parser still emits ToolUseEvent only at tool_execution_end. If a call starts but the process is killed or times out before its end arrives, the tracker never opens its span, Finish cannot mark it unanswered, and EM-002 omits it from its unanswered signal. The documentation now states this limitation accurately but the behavior is unchanged.
    Remediation: Track as a follow-up: represent started-but-unfinished calls without double-counting fullsend.tool_calls or losing failed-call summaries.
  • [api-contract] internal/runtime/pi_progress.go:608 — Leaving the ID off the use event when no start was seen makes the result read as unmatched, but it also removes Level 3 correlation for that path: the collector emits a tool_call part with no id and a tool_call_response with id=c1 and does not reconnect them. The trigger is narrow (the start line must exceed streamBufSize) and the behavior is intentional and tested, so this is advisory.
    Remediation: Consider representing the missing-start condition separately from the correlation ID, and add a parser-to-collector assertion for the end-only case.
Previous run (2)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJhcHByb3ZlIiwiZmluZGluZ3MiOlt7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJhcGktY29udHJhY3QiLCJmaWxlIjoiaW50ZXJuYWwvcnVudGltZS9waV9wcm9ncmVzcy5nbyIsImlkIjoiZl9lZWVkNGYyMzkyMTQwNTQyIiwibGluZSI6NTk1LCJhY3Rpb25hYmxlIjpmYWxzZSwiZGltZW5zaW9uIjoiY29ycmVjdG5lc3MifSx7InNldmVyaXR5IjoibWVkaXVtIiwiY2F0ZWdvcnkiOiJhcGktY29udHJhY3QiLCJmaWxlIjoiaW50ZXJuYWwvcnVudGltZS9waV9wcm9ncmVzcy5nbyIsImlkIjoiZl83ZjkzMDNjZWU2MmRmNTVlIiwibGluZSI6NjAyLCJhY3Rpb25hYmxlIjp0cnVlLCJkaW1lbnNpb24iOiJjb3JyZWN0bmVzcyJ9LHsic2V2ZXJpdHkiOiJtZWRpdW0iLCJjYXRlZ29yeSI6InN0YWxlLWRvYyIsImZpbGUiOiJkb2NzL2d1aWRlcy91c2VyL2hvdy10by1lbWl0LXRyYWNlcy5tZCIsImlkIjoiZl8xMGRkZWRjMDg2ODliZTI5IiwibGluZSI6MTU0LCJhY3Rpb25hYmxlIjp0cnVlLCJkaW1lbnNpb24iOiJkb2NzLWN1cnJlbmN5In0seyJzZXZlcml0eSI6Im1lZGl1bSIsImNhdGVnb3J5Ijoic3RhbGUtZG9jIiwiZmlsZSI6ImRvY3MvcHJvYmxlbXMvb3BlcmF0aW9uYWwtb2JzZXJ2YWJpbGl0eS5tZCIsImlkIjoiZl81MTJiMzg3NzVhYmMxYWQ2IiwibGluZSI6MjE1LCJhY3Rpb25hYmxlIjp0cnVlLCJkaW1lbnNpb24iOiJkb2NzLWN1cnJlbmN5In0seyJzZXZlcml0eSI6Im1lZGl1bSIsImNhdGVnb3J5Ijoic3RhbGUtZG9jIiwiZmlsZSI6ImRvY3MvZ3VpZGVzL2luZnJhc3RydWN0dXJlL2V2YWwtbWVhc3VyZW1lbnRzLm1kIiwiaWQiOiJmXzJkNjcxMjAwY2NkZTNiNWQiLCJsaW5lIjoxMTgsImFjdGlvbmFibGUiOnRydWUsImRpbWVuc2lvbiI6ImRvY3MtY3VycmVuY3kifSx7InNldmVyaXR5IjoibWVkaXVtIiwiY2F0ZWdvcnkiOiJzdGFsZS1kb2MiLCJmaWxlIjoiZG9jcy9ndWlkZXMvZGV2L3RyYWNpbmcubWQiLCJpZCI6ImZfYmVjOWEwYzg1NjY4NzU4YSIsImxpbmUiOjE2NCwiYWN0aW9uYWJsZSI6dHJ1ZSwiZGltZW5zaW9uIjoiZG9jcy1jdXJyZW5jeSJ9LHsic2V2ZXJpdHkiOiJsb3ciLCJjYXRlZ29yeSI6InN0YWxlLWRvYyIsImZpbGUiOiJkb2NzL2d1aWRlcy9pbmZyYXN0cnVjdHVyZS9kaXN0cmlidXRlZC10cmFjaW5nLm1kIiwiaWQiOiJmX2VlNDYwZDg2YWUyOThjMjEiLCJsaW5lIjoxMDMsImFjdGlvbmFibGUiOmZhbHNlLCJkaW1lbnNpb24iOiJkb2NzLWN1cnJlbmN5In1dLCJkaXNwb3NpdGlvbnMiOlt7ImlkIjoiZl8yZDY3MTIwMGNjZGUzYjVkIiwic3RhdHVzIjoicmVzb2x2ZWRfYnlfY2hhbmdlIn0seyJpZCI6ImZfYmVjOWEwYzg1NjY4NzU4YSIsInN0YXR1cyI6InJlc29sdmVkX2J5X2NoYW5nZSJ9LHsiaWQiOiJmX2VlNDYwZDg2YWUyOThjMjEiLCJzdGF0dXMiOiJyZXNvbHZlZF9ieV9jaGFuZ2UifSx7ImlkIjoiZl9lZWVkNGYyMzkyMTQwNTQyIiwic3RhdHVzIjoib3BlbiJ9LHsiaWQiOiJmXzdmOTMwM2NlZTYyZGY1NWUiLCJzdGF0dXMiOiJyZXNvbHZlZF9ieV9jaGFuZ2UifSx7ImlkIjoiZl8xMGRkZWRjMDg2ODliZTI5Iiwic3RhdHVzIjoicmVzb2x2ZWRfYnlfY2hhbmdlIn0seyJpZCI6ImZfNTEyYjM4Nzc1YWJjMWFkNiIsInN0YXR1cyI6InJlc29sdmVkX2J5X2NoYW5nZSJ9XX0= -->

Review

Findings

Low

  • [api-contract] internal/runtime/pi_progress.go:595 — When no matching tool_execution_start was seen, the parser still emits an id-bearing ToolUseEvent immediately before the result. This fabricates a matching call: toolSpanTracker opens and closes a normal span rather than marking fullsend.tool.unmatched, so EM-002 counts an orphan result as a real tool call. It occurs, for example, when the start line exceeded the parser's size limit but the end line did not.
    Remediation: Avoid fabricating correlation for a missing start, or propagate the missing-start condition to the tracker, and add a parser-to-tracker test asserting that an end without a start produces an unmatched-result span.

Earlier findings

  • docs/guides/infrastructure/eval-measurements.md:118, stale-doc — resolved; the EM-002 skip list now names only codex and states that an interrupted pi call has no span.
  • docs/guides/dev/tracing.md:164, stale-doc — resolved; the execute_tool span description now reads "Claude Code and pi".
  • docs/guides/infrastructure/distributed-tracing.md:103, stale-doc — resolved; pi is removed from the no-arguments runtime list and the redaction note covers pi's failed-call result text.
Previous run (3)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJyZXF1ZXN0LWNoYW5nZXMiLCJmaW5kaW5ncyI6W3sic2V2ZXJpdHkiOiJtZWRpdW0iLCJjYXRlZ29yeSI6InN0YWxlLWRvYyIsImZpbGUiOiJkb2NzL2d1aWRlcy9pbmZyYXN0cnVjdHVyZS9ldmFsLW1lYXN1cmVtZW50cy5tZCIsImlkIjoiZl8yZDY3MTIwMGNjZGUzYjVkIiwibGluZSI6MTE4LCJhY3Rpb25hYmxlIjp0cnVlLCJkaW1lbnNpb24iOiJkb2NzLWN1cnJlbmN5In0seyJzZXZlcml0eSI6Im1lZGl1bSIsImNhdGVnb3J5Ijoic3RhbGUtZG9jIiwiZmlsZSI6ImRvY3MvZ3VpZGVzL2Rldi90cmFjaW5nLm1kIiwiaWQiOiJmX2JlYzlhMGM4NTY2ODc1OGEiLCJsaW5lIjoxNjQsImFjdGlvbmFibGUiOnRydWUsImRpbWVuc2lvbiI6ImRvY3MtY3VycmVuY3kifSx7InNldmVyaXR5IjoibG93IiwiY2F0ZWdvcnkiOiJzdGFsZS1kb2MiLCJmaWxlIjoiZG9jcy9ndWlkZXMvaW5mcmFzdHJ1Y3R1cmUvZGlzdHJpYnV0ZWQtdHJhY2luZy5tZCIsImlkIjoiZl9lZTQ2MGQ4NmFlMjk4YzIxIiwibGluZSI6MTAzLCJhY3Rpb25hYmxlIjpmYWxzZSwiZGltZW5zaW9uIjoiZG9jcy1jdXJyZW5jeSJ9LHsic2V2ZXJpdHkiOiJsb3ciLCJjYXRlZ29yeSI6ImFwaS1jb250cmFjdCIsImZpbGUiOiJpbnRlcm5hbC9ydW50aW1lL3BpX3Byb2dyZXNzLmdvIiwiaWQiOiJmX2VlZWQ0ZjIzOTIxNDA1NDIiLCJsaW5lIjo1OTUsImFjdGlvbmFibGUiOmZhbHNlLCJkaW1lbnNpb24iOiJjb3JyZWN0bmVzcyJ9LHsic2V2ZXJpdHkiOiJtZWRpdW0iLCJjYXRlZ29yeSI6ImFwaS1jb250cmFjdCIsImZpbGUiOiJpbnRlcm5hbC9ydW50aW1lL3BpX3Byb2dyZXNzLmdvIiwiaWQiOiJmXzdmOTMwM2NlZTYyZGY1NWUiLCJsaW5lIjo2MDIsImFjdGlvbmFibGUiOnRydWUsImRpbWVuc2lvbiI6ImNvcnJlY3RuZXNzIn0seyJzZXZlcml0eSI6Im1lZGl1bSIsImNhdGVnb3J5Ijoic3RhbGUtZG9jIiwiZmlsZSI6ImRvY3MvZ3VpZGVzL3VzZXIvaG93LXRvLWVtaXQtdHJhY2VzLm1kIiwiaWQiOiJmXzEwZGRlZGMwODY4OWJlMjkiLCJsaW5lIjoxNTQsImFjdGlvbmFibGUiOnRydWUsImRpbWVuc2lvbiI6ImRvY3MtY3VycmVuY3kifSx7InNldmVyaXR5IjoibWVkaXVtIiwiY2F0ZWdvcnkiOiJzdGFsZS1kb2MiLCJmaWxlIjoiZG9jcy9wcm9ibGVtcy9vcGVyYXRpb25hbC1vYnNlcnZhYmlsaXR5Lm1kIiwiaWQiOiJmXzUxMmIzODc3NWFiYzFhZDYiLCJsaW5lIjoyMTUsImFjdGlvbmFibGUiOnRydWUsImRpbWVuc2lvbiI6ImRvY3MtY3VycmVuY3kifV0sImRpc3Bvc2l0aW9ucyI6W3siaWQiOiJmXzdmOTMwM2NlZTYyZGY1NWUiLCJzdGF0dXMiOiJyZXNvbHZlZF9ieV9jaGFuZ2UifSx7ImlkIjoiZl8xMGRkZWRjMDg2ODliZTI5Iiwic3RhdHVzIjoicmVzb2x2ZWRfYnlfY2hhbmdlIn0seyJpZCI6ImZfNTEyYjM4Nzc1YWJjMWFkNiIsInN0YXR1cyI6InJlc29sdmVkX2J5X2NoYW5nZSJ9XX0= -->

Review

Findings

Medium

Low

  • [stale-doc] docs/guides/infrastructure/distributed-tracing.md:103 — The text still lists pi among runtimes that report "a summary and no arguments (pi, codex, OpenCode)", but pi now passes ToolUseEvent.Arguments and the same guide lists pi's recorded argument members at lines 136-140. Line 692 also still says "Claude Code today" for id-bearing calls.
    Remediation: Remove pi from the no-arguments list at line 103, rework the "for pi and codex, cut it" clause, and update line 692 to include pi.
  • [api-contract] internal/runtime/pi_progress.go:595 — ToolUseEvent is emitted only at tool_execution_end. tool_execution_start only records the call in a local map, so a call that started but was interrupted (pi killed or stream cut) never reaches toolSpanTracker and Finish cannot close it as error.type=unanswered. This is not a regression and the issue's acceptance criteria do not require it. The generic claim in the tool_spans.go comment and in distributed-tracing.md that unanswered calls are closed by Finish does not hold for pi.
    Remediation: Optionally flush start-seen calls with no end as id-bearing ToolUseEvents when the parser finishes, with a start-then-EOF test. Otherwise qualify the tool_spans.go comment and the doc text for pi.

Earlier findings

  • internal/runtime/pi_progress.go:602, api-contract — resolved. The piResultText and ParsePiStream comments now state that an undocumented-shape result's text is dropped with partial set, and that the collector keeps a part only for a failed call. distributed-tracing.md now says the same.
  • docs/guides/user/how-to-emit-traces.md:154, stale-doc — resolved. Lines 151-158 now say Claude and pi record tool arguments and results and emit execute_tool spans.
  • docs/problems/operational-observability.md:215, stale-doc — resolved. The span granularity and tool-result sentences now say "Claude Code and pi".

Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR
Previous run (4)

<!-- fullsend:review-findings-v2:eyJ2ZXJzaW9uIjoyLCJhY3Rpb24iOiJyZXF1ZXN0LWNoYW5nZXMiLCJmaW5kaW5ncyI6W3sic2V2ZXJpdHkiOiJtZWRpdW0iLCJjYXRlZ29yeSI6ImFwaS1jb250cmFjdCIsImZpbGUiOiJpbnRlcm5hbC9ydW50aW1lL3BpX3Byb2dyZXNzLmdvIiwiaWQiOiJmXzdmOTMwM2NlZTYyZGY1NWUiLCJsaW5lIjo2MDIsImFjdGlvbmFibGUiOnRydWUsImRpbWVuc2lvbiI6ImNvcnJlY3RuZXNzIn0seyJzZXZlcml0eSI6Im1lZGl1bSIsImNhdGVnb3J5Ijoic3RhbGUtZG9jIiwiZmlsZSI6ImRvY3MvZ3VpZGVzL3VzZXIvaG93LXRvLWVtaXQtdHJhY2VzLm1kIiwiaWQiOiJmXzEwZGRlZGMwODY4OWJlMjkiLCJsaW5lIjoxNTQsImFjdGlvbmFibGUiOnRydWUsImRpbWVuc2lvbiI6ImRvY3MtY3VycmVuY3kifSx7InNldmVyaXR5IjoibWVkaXVtIiwiY2F0ZWdvcnkiOiJzdGFsZS1kb2MiLCJmaWxlIjoiZG9jcy9wcm9ibGVtcy9vcGVyYXRpb25hbC1vYnNlcnZhYmlsaXR5Lm1kIiwiaWQiOiJmXzUxMmIzODc3NWFiYzFhZDYiLCJsaW5lIjoyMTUsImFjdGlvbmFibGUiOnRydWUsImRpbWVuc2lvbiI6ImRvY3MtY3VycmVuY3kifV19 -->

Review

Findings

Medium

  • [api-contract] internal/runtime/pi_progress.go:602 — A pi tool result that piResultText drops (image-only content or an undocumented shape) is emitted as ToolResultEvent{Result:"", IsError:false, Partial:true}, but the truncation marker is lost in Level 3 telemetry. In content_collector.go the collector sets Truncated = e.Partial || e.Oversized, but contentBytes only counts a footprint for errors and oversized results, so appendPart discards the empty part. The tool_call_response and its marker both disappear and fullsend.content.truncated stays false. This contradicts piResultText's own "dropped and marked" comment, and incomplete content looks complete.
    Remediation: Make the collector retain and account for empty Partial results, as it already does for empty Oversized results. Add an end-to-end parser-to-collector test asserting that image-only and undocumented successful results keep a marked tool_call_response and set the span-level truncation marker.

  • [stale-doc] docs/guides/user/how-to-emit-traces.md:154 — The "Capture conversation content" section still says "Claude runs include tool arguments and results; the pi and codex runtimes do not yet" and that execute_tool child spans appear only with the claude runtime. This PR makes pi emit tool call ids, arguments and results and start spans from tool_execution_start, so the passage is now incorrect for pi. The file is not changed in the PR.
    Remediation: Update lines 154-158 to say that Claude and pi runs include tool arguments and results and emit execute_tool child spans, and that codex is the only runtime that does not yet.

  • [stale-doc] docs/problems/operational-observability.md:215 — The open-questions text (lines 210 and 215) says execute_tool spans exist only for id-bearing calls, "Claude Code today; pi and codex emit no call ids (telemetry: pi and codex parsers drop tool call ids and results — Level 3 tool I/O and execute_tool spans are Claude-only #7414)", and that Level 3 result content is recorded "on Claude runs only". This PR resolves the pi half of telemetry: pi and codex parsers drop tool call ids and results — Level 3 tool I/O and execute_tool spans are Claude-only #7414, so the pi claims are stale. The file is not changed in the PR.
    Remediation: Update lines 210 and 215 to say that Claude Code and pi now emit call ids, tool results and execute_tool spans, leaving codex as the remaining runtime without call ids.


Labels: PR changes runner telemetry parsing and its docs.


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review fullsend-ai-review Bot added component/runner Agent runner behavior and lifecycle component/docs User-facing documentation labels Nov 6, 2026
@codecov

codecov Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.48936% with 4 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/runtime/pi_progress.go 90.24% 3 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

…sult is marked

The user guide and the observability problem doc still said pi has no
tool I/O or spans. A dropped result is only marked when the call failed,
so say that in the piResultText comment and the tracing reference, and
name pi 1.5.0's source as the basis for the hook ordering.

Signed-off-by: Ashwin Govind Ugale <ugaleashwin@gmail.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 10:36 AM UTC · Completed 10:47 AM UTC

Commit: d990df1 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $1.98

fullsend-ai-review[bot]

This comment was marked as outdated.

fullsend-ai-review[bot]

This comment was marked as outdated.

…pted-call limit

EM-002 and the tracing guides still treated pi as having no spans or
arguments. pi reports a call only when it ends, so a call interrupted
before then has no span and never counts as unanswered; say so where
the unanswered rule is described.

Signed-off-by: Ashwin Govind Ugale <ugaleashwin@gmail.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 11:23 AM UTC · Completed 11:34 AM UTC

Commit: 1040ab6 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $1.63

@fullsend-ai-review
fullsend-ai-review Bot dismissed stale reviews from themself November 6, 2026 11:34

Superseded by updated review

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review fullsend-ai-review Bot added the ready-for-merge All reviewers approved — ready to merge label Nov 6, 2026

@waynesun09 waynesun09 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two doc findings on how pi tool spans are described. Both are about wording; the code behaviour stays as the PR has it.

Comment thread docs/guides/dev/tracing.md Resolved
…orrect the docs

An end with no start seen emitted an id-bearing use event, so the tracker
opened a normal span and EM-002 could not see the orphan result. Leave the
id off that event. The docs also blamed pi for the missing span on a cut-off
call, but pi does report the start; the parser holds the event until the end.
Say that, and separate pi's span start from Claude's receipt-time start.

Signed-off-by: Ashwin Govind Ugale <ugaleashwin@gmail.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 2:55 PM UTC · Completed 3:07 PM UTC

Commit: fddeded · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $2.62

@fullsend-ai-review fullsend-ai-review Bot removed the ready-for-merge All reviewers approved — ready to merge label Nov 6, 2026
fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review
fullsend-ai-review Bot dismissed their stale review November 6, 2026 15:07

Superseded by updated review

@fullsend-ai-review fullsend-ai-review Bot added the ready-for-merge All reviewers approved — ready to merge label Nov 6, 2026

@waynesun09 waynesun09 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One finding on how piResultText handles a pi result with no content list.

Comment thread internal/runtime/pi_progress.go Resolved
pi 1.5.0 normalizes a missing content list with content ?? [], and an
untyped extension tool can return one, so it is a normal result and not a
cut. Only a value that won't decode as a string or result object is now
dropped and marked.

Signed-off-by: Ashwin Govind Ugale <ugaleashwin@gmail.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Nov 6, 2026 •

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 3:52 PM UTC · Completed 4:04 PM UTC

Commit: ab374d1 · View workflow run →

Runtime: pi · Model: sonnet → claude-sonnet-5-5 · Effort: high · Cost: $1.90

@fullsend-ai-review fullsend-ai-review Bot removed the ready-for-merge All reviewers approved — ready to merge label Nov 6, 2026
Comment thread internal/runtime/pi_progress.go Resolved
Comment thread internal/runtime/pi_progress.go Resolved
Comment thread internal/runtime/pi_progress.go Resolved
@fullsend-ai-review
fullsend-ai-review Bot dismissed their stale review November 6, 2026 16:04

Superseded by updated review

@fullsend-ai-review fullsend-ai-review Bot added the ready-for-merge All reviewers approved — ready to merge label Nov 6, 2026

@waynesun09 waynesun09 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at ab374d1. Validated with a live pi 1.5.0 run: a rewriting tool_result hook's output reaches tool_execution_end and the Level 3 record, call ids and arguments pass through, and image-only and content-less results follow distributed-tracing.md. Pi parser and cli redaction, collector and span tests pass.

@waynesun09
waynesun09 added this pull request to the merge queue Nov 6, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Nov 6, 2026
@waynesun09
waynesun09 added this pull request to the merge queue Nov 6, 2026
Merged via the queue into fullsend-ai:main with commit 7f1f2a3 Nov 6, 2026
133 of 134 checks passed

This branch was successfully deployed

1 active deployment
site-preview — ab374d14 Deployed Nov 6, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/docs User-facing documentation component/runner Agent runner behavior and lifecycle ready-for-merge All reviewers approved — ready to merge risk/moderate PR risk: moderate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants