Skip to content

Playbook reflector silently no-ops on a claude-opus-5 root — recorded as a zero-ms cross-provider call that never happened #388

Description

@justrach

Found by the same 16-run eval study. Under a --model claude-opus-5 (codegraff gateway) root, the #383 learned-playbook reflector never mints anything: 4/4 fixtures ended with no playbook file, and the seeded fixture gained 0 learned items — while identical tasks under the sol default minted 1–3 items every run.

The trace explains it and raises a second problem: the reflect agent is recorded as model=gpt-5.6-sol, ms=0 in all inspected opus-5 runs — attributed to a DIFFERENT provider than the root, with zero elapsed time (no HTTP call happened). Under the sol root the same agent shows model=gpt-5.6-luna, ms=4471 (a real call). So either:

  1. the reflector resolves its seat via the worker-tier path, lands cross-provider without --allow-cross-provider-subagents consent, gets (correctly) refused somewhere silent, and the failure is swallowed — leaving the learning loop dead under any gateway root; and/or
  2. the trace row records the intended seat rather than what ran — a routing-telemetry mis-attribution that would mislead any audit built on agent_route/trace rows (which after Ultracode: learn orchestration and tier policy, not only prompt variants #372 is supposed to be the ground truth).

Fix should make the reflector's seat resolution follow the same consent-aware chain as any worker (a gateway root with no sub should still be able to reflect ON the gateway), and a refused/failed reflection must leave a visible trace note, never a fabricated zero-ms success row.

Evidence: eval-study runs/-C/.graff/traces/.jsonl vs runs/*-A equivalents.

🤖 Generated with Claude Code

https://claude.ai/code/session_01QbsV84fdmf39Bh2RF8LdPs

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions