Repository navigation
docs(observability): raise run-health trace scoring as an open question - #7386
waynesun09 merged 1 commit into
Conversation
Adds an open question to operational-observability.md: trace_fitness (ADR 0087) scores whether a run's telemetry is well-formed, but nothing scores whether the run behaved correctly. Structural tool-call defects (ignored tool error, errored-result reuse into a side-effecting call, no-progress loops, success-on-failure like fullsend-ai#6621) are decidable from the trace with no model. Frames two open decisions -- a second eval-measure scorer vs. leaving behavioral correctness to review/retro, and the tool-result-capture prerequisite -- rather than prescribing. Graduates fullsend-ai#7246 into the problem-doc layer. Signed-off-by: Ashwin Govind Ugale <ugaleashwin@gmail.com>
Code Review by Qodo🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)
Great, no issues found!Qodo reviewed your code and found no material issues that require reviewTip of the day💡 Did you know, you can group findings by type and pick your Finding display, from Minimal to Full |
E2E tests did not runE2E tests run automatically for org/repo members and collaborators on pull requests. For other contributors, a maintainer must add the See E2E testing guide for details. |
PR Summary by QodoDocument open questions for trace-based run-health scoring
AI Description
Diagram
High-Level Assessment
Files changed (1)
|
Site previewPreview: https://2cb5aa53-site.fullsend-ai.workers.dev Commit: |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
waynesun09
left a comment
There was a problem hiding this comment.
Docs-only problem-doc addition, reviewed at 2e85bb8: no findings, no open threads.
Adds an open question to
docs/problems/operational-observability.md.eval-measure'strace_fitnessscorer (ADR 0087) checks that a run's telemetry is well-formed — span tree, identity, model, usage, cost, exit — but nothing scores whether the run behaved correctly. There's a class of failure that's structurally decidable straight from the trace, with no model in the loop: a tool error the run ignored, an errored result reused by a later side-effecting call, no-progress loops, or a job reporting success despite an in-trace failure (e.g. #6621).The new entry frames this as two open decisions rather than prescribing a fix:
testing-agents.md— or does behavioral correctness belong with the review/retro agents?Graduates the discussion in #7246 into the problem-doc layer, per CONTRIBUTING ("issues can graduate into … problem docs").
Docs-only; no code or behaviour change.