Skip to content

resolve_query's guidance and required context are gated behind Graph Editing's routing score; fail-open delivers the tool without either #1840

Closed

Description

@malibio

Overview

Split from #1835. resolve_query's only usage guidance, and the RELEVANT ENTITY TYPES context block its required parameter depends on, are both gated behind a single skill (Graph Editing) clearing a 0.30 confidence bar in Stage-2 routing — and the fail-open path hands over the tool without either.

Problem Statement

Offered tools are scoped to the routed skill's whitelist by routing::stage2_tools (routing.rs:332-354). resolve_query appears in exactly one seeded skill's whitelist — Graph Editing (skill_pipeline.rs:347). stage2_tools fails open (routing.rs:339-341, 350-352), producing this matrix:

Routing outcome resolve_query offered? INDIRECT TARGET guidance? RELEVANT ENTITY TYPES block?
Graph Editing clears gate (>=0.30) yes yes yes
No candidate clears gate yes (fail-open) no no
A different skill alone clears gate no no no

On the fail-open path, the tool is present but stripped of the only guidance instructing its use. This is concretely broken: node_type is a required parameter (tools.rs:403) whose description tells the model to "copy the id exactly from the RELEVANT ENTITY TYPES block" (tools.rs:400) — a block that does not exist on an unrouted turn. The model is directed to copy from something it cannot see.

This reconciles two measurements that looked contradictory. #1637 (f82b7fe5, PR #1825) tuned resolve_query's description and guidance assuming reliable reachability, and measured 3/3 invocation. The very next commit (e4ff4890, PR #1826, ADR-038) made Graph Editing's whitelist conditional on MUTATING_SKILL_SCORE_BAR = 0.30 (vs 0.15 for read-only skills) — a change that postdates #1637's measurement. A Graph Editing candidate scoring 0.15-0.29 is silently dropped from both the tool set and the prompt block, which #1637's evidence never exercised.

Evidence

Acceptance Criteria

  • Decide the fail-open contract: either the fail-open tool set excludes tools whose guidance/context-block dependency isn't satisfied (drop resolve_query on fail-open), or fail-open includes a minimal fallback guidance/entity-types block sufficient for tools that need it
  • node_type's required-parameter description should not assume RELEVANT ENTITY TYPES is always present in context — either guarantee the block is always available when the parameter is required, or make the tool description self-sufficient
  • Regression test: a Graph Editing candidate scoring in the 0.15-0.29 range (below the mutating bar, above read-only) is exercised explicitly, not just the clears/fails-entirely cases
  • Live reproduction against the fixed Fixture: agent-matrix scenarios 3->4->6 key on a value the chain never persists; trace doesn't record offered tools/routed skill #1839 fixture, confirming the mechanism and the fix

Architecture Impact

A gap in ADR-038's Stage-2 confidence-bar implementation: the fail-open path was designed for graceful degradation but doesn't account for tools whose usability depends on guidance/context that only arrives when routing succeeds. Relevant to #1815's Stage-2 design and worth checking whether other skills' required parameters have the same latent dependency.

Activity

  1. malibio commented on Jul 31, 2026

    @malibio
    CollaboratorAuthor

    Frontend baseline: 171 test files passed, 4112 tests passed.

  2. self-assigned this
    on Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions