Skip to content

TOOL_STRATEGY_RULES conflates safety policy with tool routing; split it and delete routing in favor of skill/schema-carried routing #1820

Closed

Description

@malibio

Overview

TOOL_STRATEGY_RULES (packages/agent/src/agent_guidance.rs:35-58) states its purpose as "the search-first mandate, tool selection hints, and canonical create/update/connect patterns." Read line by line, it is doing four unrelated jobs, and only one of them belongs in resident prose under ADR-064.

Surfaced while investigating #1814: the core-side confirmation found update_node never reaches search_skills at all, because line 48 hardcodes its routing in prose — and the Graph Editing skill already carries the correct "find then update" instruction, unreachable behind the prose that preempts it.

The four categories, and what each should be

1. Safety invariants (lines 37, 38, 39, 40, 44) — belongs here, keep. No tools for chitchat; don't fabricate tool-call history; don't assert unverified schema facts; don't repeat identical calls; don't claim an action without a result. Genuine policy with no other home. This is the entire legitimate scope of the block under ADR-064 rule 5.

2. Argument-shape rule (line 36) — wrong channel. NODE_TYPE COMES FROM RELEVANT ENTITY TYPES governs argument construction. ADR-064 rule 1 assigns this to tool schemas.

3. Tool-usage reference (lines 50, 51, 52, 53, 56, 57) — wrong channel, and duplicate. How search_nodes filters work, when to prefer search_semantic, how to read a search_semantic result, how get_node/get_related_nodes work. This is API documentation in prose for tools that already carry their own descriptions — the same duplication class as #1809's coreValues bug, spread across more lines.

4. Per-operation routing (lines 41, 42, 48, 54, 55) — wrong channel, confirmed harmful. Each line tells the model which sequence of tool calls to make for a given request shape:

  • Line 41: for create_node, call search_skills then create_node
  • Line 42: for create_schema, skip search_skills
  • Line 48: always call search_nodes before update_node — measured to make the model bypass search_skills entirely for this operation, in every trial of a controlled confirmation run
  • Line 54: for indirect updates, call resolve_query then search_nodes then update_node
  • Line 55: for create_node, call search_skills first for schema_metadata

Routing is exactly what ADR-038 assigns to skill-mediated discovery, not resident prose. Line 48 is now measured to actively defeat it: Graph Editing's own skill instructions say "FIND THEN UPDATE: call search_semantic or search_nodes first" — the correct routing, sitting dead behind the prose line that preempts it before the model ever reaches search_skills.

Evidence

  • packages/agent/src/agent_guidance.rs:31-34 — the block's own stated purpose, which does not match categories 2–4
  • Core-side confirmation (Resolve whether skill instructions delivered as a tool result suppress action #1814): with real LocalAgentLoop, real seeded skill content, and real tool schemas, update_node called search_nodes directly in 100% of trials across both arms of a controlled test — search_skills was never invoked, confirmed via full tool-call dump on sampled trials, not a scoring artifact
  • Graph Editing's seeded guidance already contains the correct routing instruction for this operation
  • ADR-064 rule 1 (tool schemas own argument shape), rule 2 (tool descriptions own tool selection), rule 5 (resident prose owns identity and policy only)

Scope of this issue

Not "delete the routing lines and see what breaks." Each deleted line needs its routing decision confirmed present elsewhere before removal:

  • Line 48 (update_node search-first): confirmed present in Graph Editing's skill instructions — verify it's sufficient once line 48 no longer preempts it
  • Line 41 / 55 (create_node → search_skills sequencing): verify the equivalent skill's instructions carry this, or that it's redundant with the skill match itself
  • Line 42 (create_schema skips search_skills): verify this is expressed as a tool description or is structurally true (no skill exists to route through)
  • Line 54 (resolve_query → search_nodes → update_node for indirect updates): dense, multi-step, not yet confirmed to have skill or schema coverage — check before deleting
  • Line 36 (node_type argument shape): move to the relevant tool schemas' parameter descriptions
  • Lines 50-53, 56-57 (tool-usage reference): confirm each fact is already present in search_nodes/search_semantic/get_node/get_related_nodes/create_relationship's own tool descriptions; add what's missing there, then delete from prose
  • Lines 37-40, 44 retained as the block's entire remaining content
  • Regression coverage for each deleted routing line, exercised through the real skill-routing path, not the old hardcoded path
  • Assembled prompt character count recorded before/after

Acceptance Criteria

  • Every routing/reference line removed only after its equivalent is confirmed to exist in a tool schema, tool description, or skill's instructions — not removed speculatively
  • TOOL_STRATEGY_RULES contains only safety-invariant policy when done
  • update_node and create_node both route through search_skills when it is expected to fire, verified with the real LocalAgentLoop
  • No regression in the agent test suite

Architecture Impact

Implements ADR-064 rules 1, 2, and 5 against a single concrete file. Likely a prerequisite for #1815 (ADR-038 Stage-1/Stage-2 gates) — building model-judged routing on top of prose that already hardcodes routing for some operations means the two systems will disagree, with prose currently winning per the measured pattern.

Related: #1809 (the same channel-duplication pattern — resident prose contradicting a correct authoritative source — found in a different file). #1814 (the confirmation run that surfaced line 48's effect; that issue's scope is retrieval mismatch, not this routing-duplication problem, so it is filed separately rather than folded in).

Activity

  1. malibio commented on Jul 30, 2026

    @malibio
    CollaboratorAuthor

    Frontend baseline: 171 test files passed, 4112 tests passed (46.20s)

  2. self-assigned this
    on Jul 30, 2026
  3. malibio commented on Jul 30, 2026

    @malibio
    CollaboratorAuthor

    Caveat from #1814's create_node confirmation, worth weighing here

    #1814's core-side confirmation for create_node found a 100-point separation (control 38/38, mismatch 0/30) between a correctly-matched skill and a mismatched one — but the failure mode is relevant to this issue's scope, not just #1814's.

    All 30 mismatch trials: the model called search_skills, received Bulk Import's instructions (document-import guidance — "pass the markdown content directly," nothing about a properties map), then called create_node with the right node_type but content as a freeform sentence and properties: {} — every field the user supplied silently dropped, turn reported as success. Not narration, not refusal — silent data loss with no signal anything went wrong.

    The open question, not yet isolated: whether this specifically requires a mismatched skill, or whether any skill whose instructions omit field-shape guidance produces the same result once it happens to be the one returned. #1814's run didn't run a third arm to separate those two.

    That distinction matters for this issue's fix, not just #1814's:

    • If it's mismatch-specific → the fix is retrieval precision (ADR-064 rule 3), which is Resolve whether skill instructions delivered as a tool result suppress action #1814's conclusion and outside this issue's scope.
    • If it's missing-field-shape-generally → it's a property of skill completeness, independent of whether retrieval picked the right one. That would mean this issue's audit (confirming each deleted TOOL_STRATEGY_RULES line has equivalent coverage in a skill/schema before removal) needs to check not just that a skill exists for an operation, but that its instructions actually specify the properties/field contract — not just "call this tool" routing. A skill that only says "call create_node" without Node Creation's "REQUIRED FIELDS... MUST be in the properties map" language would reproduce this failure even with perfect retrieval.

    Worth keeping in mind when auditing lines 41/55 (create_node routing) specifically: confirming the model reaches search_skills isn't sufficient on its own — the skill it lands on also needs to carry the field-shape contract, or removing the prose routing line succeeds at routing while leaving this silent-drop failure mode intact.

    Not proposing new scope here — flagging it so it's not lost, and so "the skill already covers this" isn't accepted at face value during the audit without checking what the skill's instructions actually specify.

  4. added a commit that references this issue on Jul 30, 2026
    dc8e748
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions