Repository navigation
resolve_query's guidance and required context are gated behind Graph Editing's routing score; fail-open delivers the tool without either #1840
Description
Overview
Split from #1835. resolve_query's only usage guidance, and the RELEVANT ENTITY TYPES context block its required parameter depends on, are both gated behind a single skill (Graph Editing) clearing a 0.30 confidence bar in Stage-2 routing — and the fail-open path hands over the tool without either.
Problem Statement
Offered tools are scoped to the routed skill's whitelist by routing::stage2_tools (routing.rs:332-354). resolve_query appears in exactly one seeded skill's whitelist — Graph Editing (skill_pipeline.rs:347). stage2_tools fails open (routing.rs:339-341, 350-352), producing this matrix:
| Routing outcome | resolve_query offered? |
INDIRECT TARGET guidance? | RELEVANT ENTITY TYPES block? |
|---|---|---|---|
| Graph Editing clears gate (>=0.30) | yes | yes | yes |
| No candidate clears gate | yes (fail-open) | no | no |
| A different skill alone clears gate | no | no | no |
On the fail-open path, the tool is present but stripped of the only guidance instructing its use. This is concretely broken: node_type is a required parameter (tools.rs:403) whose description tells the model to "copy the id exactly from the RELEVANT ENTITY TYPES block" (tools.rs:400) — a block that does not exist on an unrouted turn. The model is directed to copy from something it cannot see.
This reconciles two measurements that looked contradictory. #1637 (f82b7fe5, PR #1825) tuned resolve_query's description and guidance assuming reliable reachability, and measured 3/3 invocation. The very next commit (e4ff4890, PR #1826, ADR-038) made Graph Editing's whitelist conditional on MUTATING_SKILL_SCORE_BAR = 0.30 (vs 0.15 for read-only skills) — a change that postdates #1637's measurement. A Graph Editing candidate scoring 0.15-0.29 is silently dropped from both the tool set and the prompt block, which #1637's evidence never exercised.
Evidence
packages/agent/src/local_agent/routing.rs:332-354(stage2_tools, fail-open behavior)packages/agent/src/skill_pipeline.rs:347(Graph Editing's whitelist, sole owner ofresolve_query)packages/agent/src/local_agent/routing.rs:63(MUTATING_SKILL_SCORE_BAR = 0.30)packages/agent/src/local_agent/tools.rs:400,403(node_typerequired, description depends onRELEVANT ENTITY TYPES)- Commits
f82b7fe5(resolve_query must return a resolved node, not query fragments #1637/resolve_query must return a resolved node, not query fragments #1825) ande4ff4890(Implement ADR-038 Stage-1 and Stage-2 gates; replace the rejected single-turn pull #1826/ADR-038), in that order - Surfaced while investigating Update scenario asks the user for a node ID instead of calling resolve_query, despite guidance and tool availability #1835's original (mis)diagnosis; live reproduction deferred to Fixture: agent-matrix scenarios 3->4->6 key on a value the chain never persists; trace doesn't record offered tools/routed skill #1839 (fixture must be fixed first, so a clean run actually exercises this path)
Acceptance Criteria
- Decide the fail-open contract: either the fail-open tool set excludes tools whose guidance/context-block dependency isn't satisfied (drop
resolve_queryon fail-open), or fail-open includes a minimal fallback guidance/entity-types block sufficient for tools that need it -
node_type's required-parameter description should not assumeRELEVANT ENTITY TYPESis always present in context — either guarantee the block is always available when the parameter is required, or make the tool description self-sufficient - Regression test: a Graph Editing candidate scoring in the 0.15-0.29 range (below the mutating bar, above read-only) is exercised explicitly, not just the clears/fails-entirely cases
- Live reproduction against the fixed Fixture: agent-matrix scenarios 3->4->6 key on a value the chain never persists; trace doesn't record offered tools/routed skill #1839 fixture, confirming the mechanism and the fix
Architecture Impact
A gap in ADR-038's Stage-2 confidence-bar implementation: the fail-open path was designed for graceful degradation but doesn't account for tools whose usability depends on guidance/context that only arrives when routing succeeds. Relevant to #1815's Stage-2 design and worth checking whether other skills' required parameters have the same latent dependency.
Activity
malibio commentedon Jul 31, 2026
Frontend baseline: 171 test files passed, 4112 tests passed.