Repository navigation
Scenarios 6, 8e and 11d fail for every model measured — audit whether they are winnable #2242
Description
Summary
Three agent-matrix scenarios — 6, 8e, 11d — fail for every model measured, including DeepSeek V4 Pro, which passes 17/20 overall and clears six scenarios that Gemma 4 E4B fails consistently. When a model that capable still cannot pass them, the scenario itself is the suspect.
Measured in #2239 across five models (Gemma 4 E4B native, Ornith 1.5 9B served, Laguna-XS 2.1 33B served, DeepSeek V4 Pro served, plus a discarded Gemma 4 12B run):
| scenario | assertion | E4B | Ornith | Laguna-XS | DeepSeek V4 Pro |
|---|---|---|---|---|---|
| 6 | [resolve_query, update_node] subsequence |
fail / excluded | fail | fail | fail |
| 8e | search_nodes exactly once |
fail | fail | fail | fail |
| 11d | get_related_nodes ≥ 1 |
fail | fail (1 of 2 reps passed) | fail | fail |
Why each is suspect, specifically
11d — the strongest candidate for a harness bug. The prompt is "What did we settle on that the rebuild has to respect?", and answering requires traversing a relationship created two turns earlier in the same chat node. Every model returns tools: []. There is a known defect that would explain this exactly: the prompt template silently drops all role="tool" history (recorded in the project's own notes on invisible prompt inputs). If prior turns' tool results never reach the model, 11d is unwinnable by construction — no model can traverse a link it cannot see.
First check: dump the assembled prompt for 11d via NODESPACE_PROMPT_DUMP and confirm whether the earlier create_relationship result is present. Prose tuning provably cannot fix a missing fact, so this needs verifying before anything else is attempted.
6 — the assertion may be wrong rather than the scenario. DeepSeek called update_node and produced the correct outcome, but failed because it did not call resolve_query first. It reached the right end state by a different path. This is trajectory-scoring punishing a correct result, and it is the clearest live argument for outcome-based assertions (see #2241's sibling discussion). Separately, E4B's run on the current fixture had scenario 6 excluded by the new guard — Stage-2 offered neither resolve_query nor update_node — so this scenario also intersects #2240.
8e — under-calling on a cross-type read. Prompt is "Run through those calls for me". Laguna's reply on this turn described creating a Planning Cycle, i.e. it answered the previous scenario's prompt — another symptom consistent with the history problem above. Worth checking whether "those calls" resolves at all given what the model can actually see.
What to do
For each of the three:
- Dump the assembled prompt and confirm every fact the scenario depends on is actually present — especially prior-turn tool results for 6 and 11d.
- Hand-execute the ideal tool call against a live backend with the same seeded state, and confirm it is accepted. A rejected well-formed call proves the scenario is unwinnable without needing a model to demonstrate it.
- Then classify: harness bug (fix the plumbing), fixture bug (fix the prompt or the assertion), or genuine capability gap (leave it, and record that it is a real bar no local model currently clears).
Scope note
This list was originally 8-9 scenarios. DeepSeek V4 Pro passing six of them narrowed it to three — those six were genuine capability gaps, not harness defects, and an earlier reading of the shared-failure pattern as "our bug" was wrong. The remaining three are the ones the evidence still supports investigating.
Related: #2240 (Stage-2 withholding asserted tools — separately causes unwinnable turns), #2241 (no repetition support, so single-run failure lists are unreliable).
Activity
malibio commentedon Aug 22, 2026
Correction: the
role="tool"history hypothesis in this issue is wrongI filed this issue naming a dropped-tool-history bug as the leading explanation for 11d (and partly 8e). That was based on a stale note and it does not hold. Correcting it here rather than leaving it to mislead whoever picks this up.
Tool history is serialized correctly.
chat_message_to_oai_value(packages/nlp-engine/src/chat/mod.rs:978) emitsrole: "tool"withtool_call_id,content, andname, and serializes assistanttool_callsturns. The gap that once existed on the OpenAI-compat path was closed by #2236 (fe63b7d9, currently HEAD ofmain) — "Serialize assistant tool_calls and tool-result name on the OpenAI-compat path".Confirmed empirically, not just from the commit message. DeepSeek V4 Pro on 11d:
That is the correct answer, recalled from a decision node created two turns earlier. The model could see the history. It simply answered from context instead of calling
get_related_nodes.What 11d actually is
Not a harness bug, and not a capability gap — a scoring problem. The model reached the right outcome without the asserted tool call, and
noRetry/minCalls: 1reds it out for that.This makes 11d the same class as scenario 6, where DeepSeek called
update_node, produced the correct end state, and failed only for skippingresolve_query. Both belong to #2243 (trajectory vs outcome scoring), not to a winnability audit.There is a legitimate product question buried in it — whether we want the agent to answer relationship questions from conversational memory rather than traversing the graph, since memory does not scale past the context window and will silently start being wrong on a longer session. But that is a design decision to make deliberately, not a model failure to fix.
Revised scope of this issue
resolve_queryandupdate_nodeon both E4B new-fixture reps, reproducibly). What remains after that is also a Agent matrix scores trajectory where it should score outcome — correct results fail, severity is flattened #2243 case.So the winnable-scenario audit is down to one scenario, not three. The other two are scoring-model problems that a winnability audit would not have found.
Process note
The original hypothesis came from a note written when the bug was real; the fix landed afterwards and the note was never updated. The lesson worth keeping: verify a named defect still exists in the current tree before building an argument on it — the same discipline this evaluation applied to ADR-046's Ornith verdict, and did not apply here.
malibio commentedon Aug 22, 2026
Baseline (worktree issue-2242-scenario-audit): Frontend: 227 test files passed, 4794 tests passed.
Address review: rescope scenario 12 to what it actually measures
Add a comparative-reference scenario to the agent matrix (#2250)
Measure indirect-reference decomposition with an out-of-band seeded r…