Repository navigation
Update scenario asks the user for a node ID instead of calling resolve_query, despite guidance and tool availability #1835
Description
Overview
The baseline run recorded on #1764 (commit 57bf1088, post-ADR-064/#1800) found the model does not call resolve_query for an indirect-reference update, contrary to what #1637's own closing evidence claims (3/3 invocation with the guidance rule present).
Problem Statement
Scenario 6 of scripts/eval/fixtures/agent-matrix.ts — "The 2400 one came back — set it to returned" — is the exact phrasing #1637 used to validate the resolve_query return-contract fix (its evidence table cites 100% success on "the 2400 one" with the {id, title} return shape). That fix landed (f82b7fe5, PR #1825) and is present in the code this baseline ran against.
In 2 of 3 fresh runs against the current guidance and code, the model called no tool at all and instead asked the user for a node ID or URI:
- Run 2: "I need a specific node ID to mark the laser cutter as returned. Could you provide me with its unique identifier?"
- Run 3: "I need the ID of the laser cutter record you are referring to in order to update its status. Could you provide that node's URI?"
(Run 1's failure is a cascade from scenario 4 never creating the referenced node — a different, reasonable clarifying question, not this pattern — so this is 2/3 clean reproductions, not 3/3.)
resolve_query is confirmed present in the offered tool list on every one of these turns (verified via the daemon log's "system prompt and tools prepared" line, stage2_candidates_injected=true), and the guidance rule instructing its use for indirect references is present verbatim in skill_pipeline.rs:127 ("INDIRECT TARGET: ... call resolve_query(request=, node_type) FIRST"). The model simply does not reach for it — this is not a routing failure or a tool-availability gap.
Why this is not #1637
#1637 was about resolve_query's return contract once called — the model invoked it reliably (3/3) but couldn't act on the query-fragment result it got back. That's fixed and confirmed in code. This issue is about the model not invoking the tool at all on a subset of runs, which is a different failure mode #1637's evidence did not observe under its own measurement conditions.
Evidence
- Baseline: Make the agent eval diagnostic before recording a baseline; revisit ADR-056's Scenario 5 claim #1764, 3 runs, commit
57bf1088 - Raw traces:
scripts/results-e4b.trace.jsonl(committed run) and the daemon logs from the other two runs (not committed — available on request) - 2/3 runs: no tool called, model asks for an ID/URI
- 1/3 runs (run 1): cascading failure from scenario 4, not this pattern
Suggested Investigation
- Reproduce in isolation (without the scenario 3→4→5→6 chain) to rule out any cross-turn context effect from earlier failures in the same chat
- Check whether resolve_query's
node_typeparameter — which the model must supply itself — is where comprehension breaks down for this phrasing specifically ("the 2400 one" implies a value/amount but not an explicit type name in the same sentence) - Compare against resolve_query must return a resolved node, not query fragments #1637's own stubbed-resolver measurement, which this issue's acceptance criteria flagged as unverified ("the real scoped inference call is exercised — the measurements above used a stubbed resolver, so its live accuracy is unverified")
Correction
The baseline comment on #1764 and scripts/results-e4b.json's caveats incorrectly attributed this failure to #1637 (already closed). This issue supersedes that attribution.
Activity
Correct #1637 misattribution in the baseline; fix a stale doc-comment…
Make the agent eval diagnostic before recording a baseline; revisit A…
malibio commentedon Jul 31, 2026
Investigation: the premise doesn't hold — scenario 6 is unwinnable by construction
Static investigation against the committed baseline (
scripts/results-e4b.json,scripts/results-e4b.trace.jsonl, commit57bf1088) plus a read of the routing/tool-offering path. No live reproduction has been run yet — the diagnosis changes what should be built, so the run is deliberately deferred.Scenario 6 cannot be satisfied: "2400" is never persisted
Scenario 6 keys on "The 2400 one came back". That value is never stored:
status,checkout_date,due_date.By scenario 6 no node anywhere carries 2400. The model answering "I need a specific node ID to mark the laser cutter as returned" is correct behavior for an unresolvable reference — a perfect
resolve_querycall would returnresolved:false, reason:"no_match".The fixture comment at
scripts/eval/fixtures/agent-matrix.ts:224-226shows the gap was anticipated for status but not for cost: it reasons carefully that scenario 3 "plausibly carries a status field" so scenario 6 tests "resolve_query routing, not whether the model guessed a status value" — while scenario 6's actual discriminator is the amount, which nothing in scenario 3's prompt asks the schema to carry.This makes the "2/3 clean reproductions" figure a measurement of the fixture, not the model.
The evidentiary claim about tool availability needs correcting
The issue states
resolve_query"is confirmed present in the offered tool list on every one of these turns," citingstage2_candidates_injected=true. That flag doesn't support the claim — it only reports that some candidate block was rendered (agent_loop.rs:669), naming no skill and no tool.Offered tools are scoped to the routed skill's whitelist by
routing::stage2_tools(routing.rs:332-354), andresolve_queryappears in exactly one seeded skill's whitelist — Graph Editing (skill_pipeline.rs:347). Becausestage2_toolsfails open (routing.rs:339-341,350-352), the actual matrix is:resolve_queryoffered?RELEVANT ENTITY TYPESblock?So the tool may indeed have been present — but on the fail-open path it arrives stripped of the only guidance instructing its use. That matters concretely:
node_typeis a required parameter (tools.rs:403) whose description tells the model to "copy the id exactly from the RELEVANT ENTITY TYPES block" (tools.rs:400) — a block that does not exist on an unrouted turn. The model is directed to copy from something it cannot see.The committed trace records neither the offered tool list nor the routed skill (
toolsOfferedis an empty string on every scored turn), so none of this was checkable without reading source.Why #1637's evidence and this baseline can both be honest
f82b7fe5(#1825) tunedresolve_query's description and skill guidance assuming the tool is reliably reachable. The immediately following commite4ff4890(#1826, ADR-038) made Graph Editing's entire whitelist conditional on clearing the 0.30 mutating score bar (MUTATING_SKILL_SCORE_BAR,routing.rs:63) — vs 0.15 for read-only skills. A Graph Editing candidate scoring 0.15–0.29 is silently dropped from both the tool set and the prompt block.#1637's 3/3 invocation evidence predates that gate. That reconciles the two measurements without either being wrong, and supports this issue's separate observation that #1637's stubbed-resolver measurement left live behavior unverified.
Two independent defects — the fixture one masks the routing one
resolve_queryand its only instructions are gated behind a single skill clearing 0.30, and the fail-open path delivers the tool without the guidance or the entity-types block its required parameter depends on.Defect 2 is not what this issue describes ("the model simply does not reach for it — this is not a routing failure or a tool-availability gap"). On this evidence it is precisely a routing/guidance-delivery gap, closer to #1832's territory.
Also worth fixing regardless: record the offered tool list and routed skill per turn in the trace, so an availability claim is checkable from results rather than from source.
Recommendation
resolve_queryavailability/guidance coupling — or fold (b) into OpenAI-compat path has no routing-reliability testing; Stage-2 candidate injection can silently suppress tool-calling on served models #1832.Open questions for @malibio: repurpose vs. close-and-split; whether (b) belongs in #1832; and whether to run the live reproduction after the fixture fix.
Proceeding on this recommendation unless redirected.
malibio commentedon Jul 31, 2026
Excellent work — this is the strongest piece of investigation in this whole thread. The re-derivation of the 3→4→6 chain from the actual fixture text (not the comment's stated intent) and tracing
stage2_candidates_injecteddown to what it actually reports versus what the issue claimed are exactly the right instincts. Answers below.#1832 is closed (merged PR, not a live issue) — so defect (b) can't fold into it. Needs its own fresh issue.
Decisions
Close Update scenario asks the user for a node ID instead of calling resolve_query, despite guidance and tool availability #1835 as superseded/misdiagnosed. Agreed. The original claim ("model simply doesn't reach for it, not a routing failure") is contradicted by your own finding — it's precisely a routing/guidance-delivery gap. Don't leave it open with a wrong diagnosis sitting at the top; close it with this investigation as the resolution, same discipline as the Scenario 6 → resolve_query must return a resolved node, not query fragments #1637 correction earlier.
Split into two, not folded together:
agent-guidancerename bug.resolve_query's only guidance and theRELEVANT ENTITY TYPESblock it depends on are coupled to a single skill clearingMUTATING_SKILL_SCORE_BAR(0.30), with a fail-open path that delivers the tool without its guidance or the block its requirednode_typeparameter needs. This is the one that actually matters — file it with the full mechanism from your comment (the routing.rs line references, the fail-open table, the required-parameter dependency on an absent context block), not just a link back to this closed issue.Defer the live reproduction until the fixture is fixed. Agreed — running it now measures a fixture bug, which is exactly the kind of wasted-inference-on-a-broken-harness this workstream has hit repeatedly (the spike's six harness bugs, Measure whether context-block enrichment (types, relationships, related schemas) helps #1817's arm-entanglement, Decide the runtime response to routing-suppressed OpenAI-compat models (matrix now exists) #1830's ornith contamination). Fix the fixture first, then a live run on (b) actually tells you something.
One addition to (a): while fixing the value-persistence gap, also fix scenario 3's prompt to imply a cost/value field OR change scenario 6's key to something scenario 4 actually persists — your writeup frames it as either fix being acceptable, make sure whichever is chosen doesn't just relocate the same "prompt implies X but schema doesn't capture X" gap to a different field.
Proceed as recommended: close #1835, file (a) and (b) as new issues.
malibio commentedon Jul 31, 2026
Closing as misdiagnosed, per the investigation above. Split into:
Live reproduction deferred to #1840, after #1839's fixture fix lands.