Skip to content

docs(observability): raise run-health trace scoring as an open question - #7386

Merged
waynesun09 merged 1 commit into
fullsend-ai:mainfrom
AshwinUgale:docs/observability-run-health-scoring
Sep 17, 2026
Merged

waynesun09 merged 1 commit into
fullsend-ai:mainfrom
AshwinUgale:docs/observability-run-health-scoring

Conversation

@AshwinUgale

Copy link
Copy Markdown
Member

Adds an open question to docs/problems/operational-observability.md.

eval-measure's trace_fitness scorer (ADR 0087) checks that a run's telemetry is well-formed — span tree, identity, model, usage, cost, exit — but nothing scores whether the run behaved correctly. There's a class of failure that's structurally decidable straight from the trace, with no model in the loop: a tool error the run ignored, an errored result reused by a later side-effecting call, no-progress loops, or a job reporting success despite an in-trace failure (e.g. #6621).

The new entry frames this as two open decisions rather than prescribing a fix:

  1. Does eval-measure grow a second scorer for run-health defects — the deterministic complement to the statistical evals in testing-agents.md — or does behavioral correctness belong with the review/retro agents?
  2. Prerequisite either way: tool results aren't on the trace yet, so the high-value rules depend on first extending capture — a telemetry-contract decision.

Graduates the discussion in #7246 into the problem-doc layer, per CONTRIBUTING ("issues can graduate into … problem docs").

Docs-only; no code or behaviour change.

Adds an open question to operational-observability.md: trace_fitness (ADR
0087) scores whether a run's telemetry is well-formed, but nothing scores
whether the run behaved correctly. Structural tool-call defects (ignored
tool error, errored-result reuse into a side-effecting call, no-progress
loops, success-on-failure like fullsend-ai#6621) are decidable from the trace with
no model. Frames two open decisions -- a second eval-measure scorer vs.
leaving behavioral correctness to review/retro, and the tool-result-capture
prerequisite -- rather than prescribing. Graduates fullsend-ai#7246 into the
problem-doc layer.

Signed-off-by: Ashwin Govind Ugale <ugaleashwin@gmail.com>
@AshwinUgale
AshwinUgale requested a review from a team September 16, 2026 20:56
@qodo-code-review

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)

Grey Divider

Great, no issues found!

Qodo reviewed your code and found no material issues that require review

Grey Divider

Tip of the day
💡 Did you know, you can group findings by type and pick your Finding display, from Minimal to Full

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

@github-actions

Copy link
Copy Markdown

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Document open questions for trace-based run-health scoring

📝 Documentation 🕐 Less than 5 minutes

Grey Divider

AI Description

• Documents deterministic run-health defects that trace-fitness scoring does not detect.
• Frames scorer ownership and tool-result capture as unresolved observability decisions.
• Preserves current behavior while promoting issue discussion into the problem documentation.
Diagram

graph TD
  A["Run Trace"] --> B["Trace Fitness"]
  A --> C{"Tool results present?"}
  C -- "No" --> D["Extend Capture"] --> E{"Health scoring owner?"}
  C -- "Yes" --> E
  E -- "Eval layer" --> F["Run Health Scorer"]
  E -- "Agent layer" --> G["Review and Retro"]
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Decide through an ADR now
  • ➕ Would establish ownership and telemetry requirements immediately
  • ➕ Could accelerate implementation planning
  • ➖ Prematurely commits to a design before tool-result capture is resolved
  • ➖ Would bypass broader exploration expected in problem documentation
2. Keep discussion only in the issue
  • ➕ Avoids expanding the observability problem document
  • ➕ Keeps early-stage discussion lightweight
  • ➖ Makes the architectural gap less discoverable
  • ➖ Separates the question from related observability decisions and ADR context

Recommendation: The PR's problem-document approach is appropriate: it makes the trace-health versus run-health gap discoverable while avoiding a premature implementation decision. A follow-up ADR should be created only after ownership and the tool-result telemetry contract are resolved.

Files changed (1) +1 / -0

Documentation (1) +1 / -0
operational-observability.mdAdd run-health trace scoring as an open observability question +1/-0

Add run-health trace scoring as an open observability question

• Adds an open question distinguishing telemetry fitness from deterministic behavioral health checks. It records the ownership choice between eval-measure and review/retro agents, plus structured tool-result capture as a prerequisite.

docs/problems/operational-observability.md

@github-actions

Copy link
Copy Markdown

Site preview

Preview: https://2cb5aa53-site.fullsend-ai.workers.dev

Commit: 2e85bb802e219acadaeb09c53d8fdefeeefd8691

@codecov

codecov Bot commented Sep 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@waynesun09 waynesun09 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Docs-only problem-doc addition, reviewed at 2e85bb8: no findings, no open threads.

@waynesun09
waynesun09 added this pull request to the merge queue Sep 17, 2026
Merged via the queue into fullsend-ai:main with commit bbfb3e6 Sep 17, 2026
32 checks passed

This branch was successfully deployed

1 active deployment
site-preview — 2e85bb80 Deployed Sep 16, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants