Skip to content

docs(adr): ADR 0107 — capture tool-call results in the run trace - #7419

Closed
AshwinUgale wants to merge 1 commit into
fullsend-ai:mainfrom
AshwinUgale:docs/adr-capture-tool-results
Closed

AshwinUgale wants to merge 1 commit into
fullsend-ai:mainfrom
AshwinUgale:docs/adr-capture-tool-results

Conversation

@AshwinUgale

Copy link
Copy Markdown
Member

Follows the merged problem-doc addition in #7386, which recorded the open question in operational-observability.md of whether trace scoring should cover run health, not just trace fitness (ADR 0087), and the earlier discussion in #7246.

This ADR takes the one prerequisite that gates all of it: capturing tool-call results on the trace. Today Level 3 content capture records tool-call names and counts but explicitly defers results (internal/cli/content_collector.go: "not captured yet"), so the structurally-decidable failure classes — a tool error the run ignored, an errored result reused by a later side-effecting call — aren't visible to any deterministic post-run analysis.

I'm suggesting this and asking for your approval on the direction — not asserting it's already settled. There's no "Proposed" ADR status in this repo, so per convention it's written as Accepted, but the intent is maintainer sign-off at review; if the direction isn't wanted, that's useful to know too. The decision is scoped to the capture-contract change only — a run-health scorer (EM-002) would be a separate follow-up if you want it.

Notes:

  • Number 0107 may collide with in-flight ADRs — happy to renumber at merge.
  • Includes the living-doc updates the writing-adrs skill requires for an Accepted ADR: an architecture.md Observability "Decided" bullet, and an annotation on the operational-observability.md open question (capture prerequisite marked decided; the scorer question left open).
  • Docs-only; no code or behaviour change.

Refs: #7246, #7386.

Proposes capturing each tool call's result on the trace (bounded, redacted,
gated), the prerequisite for deterministic run-health analysis raised as an
open question in operational-observability.md (fullsend-ai#7386). Scoped to the
capture-contract change only; a run-health scorer is a separate follow-up.
Includes the required architecture.md and problem-doc updates.

Refs: fullsend-ai#7246, fullsend-ai#7386.
Signed-off-by: Ashwin Govind Ugale <ugaleashwin@gmail.com>
@AshwinUgale
AshwinUgale requested a review from a team September 17, 2026 18:51
@github-actions

Copy link
Copy Markdown

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

@qodo-code-review

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

Grey Divider

Great, no issues found!

Qodo reviewed your code and found no material issues that require review

Grey Divider

Tip of the day
💡 Did you know, you can enable the Remediation agent and Qodo fixes findings in a dedicated fix PR

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Adopt bounded tool-result capture for run traces

📝 Documentation 🕐 10-20 Minutes

Grey Divider

AI Description

• Documents bounded, redacted, opt-in tool-result capture in Level 3 run traces.
• Establishes capture as prerequisite for deterministic run-health analysis and richer replay.
• Updates architecture and observability docs while leaving scorer design open.
Diagram

graph TD
  A["Tool result"] --> B{"Capture enabled?"} -->|Yes| C["Size bound"] --> D["Redaction pipeline"] --> E["Agent span"] --> F["Health analysis"]
  B -->|No| G["No capture"]
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Record dedicated tool spans or events
  • ➕ Provides explicit per-call structure, timing, status, and result association.
  • ➕ Avoids concentrating multiple potentially large results in one agent-span attribute.
  • ➖ Requires a broader telemetry-contract and instrumentation change.
  • ➖ May complicate compatibility with existing trace consumers and GenAI conventions.
2. Capture error metadata only
  • ➕ Reduces trace volume and sensitive-data exposure.
  • ➕ Supports basic ignored-error detection with a smaller schema.
  • ➖ Cannot support richer replay or analysis of successful results.
  • ➖ May not provide enough context to detect errored-result reuse reliably.

Recommendation: The proposed gated, bounded, and redacted agent-span capture is the best incremental approach because it reuses established Level 3 controls while preserving enough evidence for replay and future health scoring. Dedicated tool spans may be preferable in a broader telemetry redesign, but they exceed this ADR’s focused prerequisite scope.

Files changed (3) +37 / -1

Documentation (3) +37 / -1
0107-capture-tool-call-results-in-run-trace.mdDefine the tool-result trace capture decision +35/-0

Define the tool-result trace capture decision

• Adds ADR 0107 establishing bounded, redacted, opt-in capture of tool-call results on agent spans. It documents security, volume, compatibility, and no-backfill consequences while explicitly excluding run-health scoring.

docs/ADRs/0107-capture-tool-call-results-in-run-trace.md

architecture.mdRecord tool-result capture in observability decisions +1/-0

Record tool-result capture in observability decisions

• Adds the accepted capture contract to the architecture document’s Observability section. It links the decision to ignored-error detection, errored-result reuse analysis, and replay.

docs/architecture.md

operational-observability.mdMark the trace-capture prerequisite as decided +1/-1

Mark the trace-capture prerequisite as decided

• Annotates the run-health scoring question with ADR 0107 as the resolution of its capture prerequisite. The separate decision about adding a behavioral health scorer remains open.

docs/problems/operational-observability.md

@codecov

codecov Bot commented Sep 17, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@github-actions

Copy link
Copy Markdown

Site preview

Preview: https://89efde52-site.fullsend-ai.workers.dev

Commit: 8052c7e793b809117d614f822d95006dc9f52d09

@AshwinUgale

Copy link
Copy Markdown
Member Author

Closing , I hadn't seen #6603, which already lands this capture decision (ADR 0108, plus the execute_tool spans and tool-result content). No sense in a second ADR on the same thing. Thanks @dhshah13; I'll build the run-health scorer (fullsend-ai/fullsend#7246) on top once it merges.

@fullsend-ai-retro

fullsend-ai-retro Bot commented Sep 17, 2026 •

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 9:56 PM UTC · Completed 10:07 PM UTC

Commit: 8052c7e · View workflow run →

Runtime: claude · Model: sonnet → claude-sonnet-5 · Effort: high · Cost: $2.07

@fullsend-ai-retro

Copy link
Copy Markdown

PR #7419 was a human-authored, docs-only ADR proposal ("ADR 0107 — capture tool-call results in the run trace") that its own author closed ~3 hours after opening because it duplicated the decision already made in the still-open PR #6603 (ADR 0108), which has covered the same capture prerequisite since 2026-08-25. No fullsend review agent ever looked at PR #7419's content: the author holds only read permission on the repo, so the pull_request_target opened routing rule (which requires the PR author to have at least triage permission, or be a bot) intentionally fell through to "no stage matched," and no maintainer added a ready-for-review label during the PR's lifetime. That is the working-as-designed external-contributor trust gate from ADR 0054/#5223, and is already an active area of tracked work (#2552, #4374, #6803) — no new proposal filed there. The only automated commentary came from the third-party Qodo bot, which correctly found zero code bugs (there weren't any) but has no way to detect cross-PR ADR/decision duplication. The duplicate was actually surfaced indirectly: maintainer ralphbean ran /fs-triage on the related issue #7246, and fullsend's triage agent correctly identified that #6603/ADR 0108 already satisfies the prerequisite decision (see #7246). The author closed #7419 minutes later, explicitly crediting that triage output. Root cause, confirmed by reading skills/writing-adrs/SKILL.md directly: its only pre-authoring duplicate-avoidance step (checklist step 1) is scoped to ADR numbering collisions ("scan open pull requests for new docs/ADRs/NNNN-*.md files so you do not collide with in-flight ADRs... pick the lowest unused four-digit NNNN") and says nothing about checking whether an open or recent PR already covers the same decision. The author was demonstrably following this exact checklist item (his PR body flags "Number 0107 may collide with in-flight ADRs"), so the numbering-only framing is a plausible explanation for why the content duplication with #6603 wasn't caught before ~35 lines of ADR text, an architecture.md edit, a problem-doc annotation, and a Qodo/codecov/site-preview CI cycle were spent on it. Checked for existing coverage before proposing: #2571 (code agent should consult ADRs before writing unrelated architecture docs), #1313 (review agent duplicate-PR detection, different repo/context), #3030 (ADR review design-depth checklist for the review agent), and #1546 (triage should flag proposals that contradict existing decisions) are all open but none proposes a pre-authoring content/topic-overlap check inside the ADR-writing skill itself — this looks like a genuinely open, narrowly-scoped gap.

Proposals filed

This branch was successfully deployed

1 active deployment
site-preview — 8052c7e7 Deployed Sep 17, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants