Resolve whether skill instructions delivered as a tool result suppress action #1814

Activity

1. malibio

   added

   foundation

   backend

   on Jul 30, 2026
2. malibio commented on Jul 30, 2026

   

   malibio

   on Jul 30, 2026

   CollaboratorAuthor

   More actions

   Stage 1 pre-registration — recorded before any arm was run

   Harness: nodespace-nlp-experiment, new instruction-shape experiment (src/instruction_shape.rs). Model: Gemma 4 E4B Q4_K_M (ADR-056 locked native model), temp 0.1 / top_p 0.9 / min_p 0.05, fresh LlamaContext per generation so no KV state carries between trials. Reuses prompt_arch_bench's existing render/inference/parse path, so numbers are comparable with the probe D and E measurements cited in this issue.

   Scenario

   Status change on a node that already exists. search_skills has already been called; its result is the variable under test. Six request phrasings, each with its own resolved node.

   Schema creation — where the 100%→22% collapse was originally measured — was rejected as the Stage 1 scenario. A declarative framing has to be expressible for the contrast to exist at all: "the record is node X, its status is Y" is a fact, but "create a schema for a new type" has no resolved state to hand back, only a procedure. Probe D therefore could not have separated shape from size, whatever it measured.

   Conditions (fixed payload length, ~1500 chars, mid-range of the non-monotonic sweep)

   condition	role	payload
   compact	positive control	name/description/confidence, no instructions field
   procedural	arm	the facts as numbered steps — "STEP 1 Identify… STEP 3 Call update_node…"
   declarative	arm	the same facts as resolved state — "The X is node n_… its status reads Y"
   withheld	negative control	the same resolved state, plus an explicit hold: "DO NOT CALL update_node YET… wait for their reply"

   Equalisation, enforced as a hard gate before the model loads

   Both arms render from one fact table (id, title, node_type, status field, current value, target value) and pad from one filler pool to the same target. assert_arms_comparable() aborts the run unless, for every case:

   1. the arms are not byte-identical (the failure that produced three clean, meaningless runs of the relationship-context experiment);
   2. every payload is within 1500 ± 60 chars, and the two arms are within 60 chars of each other — otherwise this is a size test;
   3. both arms name all six facts — an arm that omits one is the less informative arm and any difference is confounded;
   4. the shape contrast is actually in the text (procedural contains steps; declarative contains none).

   Padding appends whole sentences and never truncates mid-word. This is deliberate: the original sweep's instruction_of_size repeated one filler paragraph and then truncated mid-sentence, so its 2400 and 3200 points differed in repetition count and cut position as well as length. That is a candidate explanation for the non-monotonicity independent of any model behaviour.

   Six unit tests cover the gate. NODESPACE_ARMS_ONLY=1 prints the built payloads; they were read before running, not just asserted on.

   Scoring

   Scored on the narrowest observable step the variable affects: did the generation emit a tool call that advances the turn (update_node, or a search_nodes that legitimately precedes it)? Not a multi-step chain that can fail upstream. p counts generations emitting no tool call at all — production's failure shape. Degenerate empty generations are excluded from the denominator and reported separately as e; E4B emits them intermittently and production documents the same behaviour.

   Raw model output is retained for every scored generation via NODESPACE_BENCH_TRACE.

   Decision rule — fixed before any arm ran

   • n ≥ 30 per arm. 6 cases × 6 trials = 36 per arm. Arms run in separate processes with disjoint seed offsets, because the reference harness's Metal allocation growth capped a single process at n=3 per cell; distinct --seed-base values keep separate processes from re-running the same trajectories.
   • Control gate first. A controls-only run must land compact ≥ 90% and withheld ≤ 25%. If either misses, the run is discarded and the harness fixed before any arm runs — a probe whose positive control fails cannot detect action, and one whose negative control acts anyway cannot detect suppression.
   • The effect is real only if procedural and declarative separate by ≥ 25 points, the magnitude the original 100%→22% observation implies.
   • A separation below 25 points is reported as "no effect at measurable resolution," not as a small effect.
   • Direction matters. ADR-064 rule 4 predicts procedural < declarative. A ≥25-point separation in the opposite direction falsifies rule 4 as the mechanism here and is reported as such, not as a partial confirmation.

   Gate on Stage 2

   Per this issue: if Stage 1 reproduces the effect and accounts for the sweep's pattern, the mechanism is identified and the issue closes. Size, position, and imperative-directive arms are then not run.
3. malibio commented on Jul 30, 2026

   

   malibio

   on Jul 30, 2026

   CollaboratorAuthor

   More actions

   Stage 1 results

   Harness: nodespace-nlp-experiment, new instruction-shape experiment (src/instruction_shape.rs, uncommitted per this issue's instructions — procedure and numbers recorded here instead). Model: Gemma 4 E4B Q4_K_M, temp 0.1/top_p 0.9/min_p 0.05, fresh LlamaContext per generation. Scenario, conditions, equalisation gate, and decision rule as pre-registered in the prior comment.

   Harness validation (required before any arm)

   A harness bug surfaced immediately: three of the six original request phrasings were bare statements of fact ("The Kessler order shipped", "The Voss renewal came back signed"). These produced an empty generation on 100% of trials, deterministically per case — not intermittency, a dead prompt shape. E4B ends the turn immediately on input it reads as not requiring a response. That would have silently halved the sample in a case-correlated way. Fixed by rephrasing all six as imperatives ("Mark the Kessler order as shipped"), same resolved nodes and facts, nothing about the variable under test changed. Empty rate went 46% → 3%.

   Control gate, n=36 each, separate process:

   control	result	threshold	verdict
   positive (compact match, no payload)	100% (35/35, 1 empty)	≥90%	✓
   negative (resolved state + explicit hold)	0% (0/36)	≤25%	✓

   Both landed exactly where predicted. The probe can detect action and can detect suppression.

   Registered arms, n=36 each, separate processes

   condition	acted
   procedural ("STEP 1 — search for X... STEP 3 — call update_node...")	100% (36/36)
   declarative ("X is node abc-123, status pending, use update_node")	100% (36/36)

   Separation: 0 points against a pre-registered ≥25 threshold. Null result — content shape does not suppress action in this design.

   Not a harness artifact: the two arms produced materially different behaviour despite identical action rates. Procedural induced a redundant re-resolution step (search_nodes before update_node, re-deriving an id the payload already stated) in 36% of trials (13/36), deterministically per case (6/6 on two cases, 0/6 on three, 1/6 on one). Declarative went straight to update_node in 100% of trials. Every update_node call across both arms used the correct id (59/59). So shape didn't move whether the model acted, only whether it took an unnecessary extra step first.

   Accounting for the 2400-fail/3200-pass non-monotonicity (required by AC)

   Mechanically: instruction_of_size (reference harness) builds every point by repeating one filler paragraph and then String::truncate-ing mid-sentence. Reproducing it, 700/1500/2400/3200/4500 chars correspond to 1.26/3.13/5.23/7.10/10.14 repetitions, each ending at a different word. The sweep was never a one-variable size test — length, repetition count, and cut position all varied together.

   Isolating each factor at n=36:

   condition	acted
   declarative payload, cut mid-word instead of on a sentence	100% (36/36)
   declarative payload, id/facts stripped out (same length/shape)	100% (36/36)
   declarative payload, but for the wrong operation (create-instance guidance on an update request)	81% (25/31, 5 empty — one case empty 5/6, flagged by the harness's own empty-rate gate, so treat this cell as indicative, not clean)
   original sweep payload, verbatim, at 2400 chars	3% (1/36)
   original sweep payload, verbatim, at 3200 chars	17% (6/36)

   Two things resolve here:

   1. The inversion was sampling noise. At n=36 both sizes collapse, and 2400 is (slightly) worse than 3200 — monotonic, not inverted. The original observation was 0/3 vs 1/3 at n=3, which is exactly the kind of separation sampling noise produces at that n.
   2. Neither shape nor truncation nor missing facts caused the collapse — topical mismatch did. The sweep's payload is create_node guidance for a new instance, returned to a request to update an existing record. Raw output from the collapsing cells: "I need more information to mark the Kessler order as shipped. Could you please provide the order ID?" — the model is reading guidance for the wrong operation and asking for information it already has in the conversation, because the payload told it a different (wrong) procedure for obtaining it. Mismatched-topic guidance alone gets to 81%; the full replica (mismatch + repetition + mid-word cut) gets to 3%, so the effects compound rather than any single one dominating.

   Verdict

   • Stage 1 is a null on the registered variable. Procedural vs. declarative framing, at fixed length, with identical facts, does not move action rate. ADR-064 rule 4 ("tool results own facts, not procedures") is not supported as the mechanism for this collapse — the procedural arm acted at 100%.
   • The original collapse is real, reproduced at n=36, and has an identified mechanism: retrieval mismatch, not delivery channel. search_skills returning a skill whose instructions describe the wrong operation drives the model into asking for information it already has, or into following the wrong procedure. This is a retrieval-precision problem (ADR-064 rule 3 territory — skill descriptions as the retrieval index), not a channel problem (rule 4).
   • ADR-038 line 81's Provisional marker: retracted, but not for the reason it anticipated. The evidence does not show that returning skill instructions as a tool result per se suppresses action — compact, procedural, declarative, no-facts, and mid-word-truncated payloads all acted at or near 100% through that exact channel. What suppresses action is returning instructions for the wrong skill relative to the request. The fix implied is tightening retrieval precision (or gating on confidence) ahead of the search_skills result, not moving instructions to a different channel.
   • Per the pre-registered gate, this closes Stage 1 without triggering the standard Stage 2 arms (size/position/imperative-directive) — the mechanism is identified and none of those three remaining variables plausibly reopens it. One follow-up is warranted, not the general Stage 2 sweep: confirm the retrieval-mismatch finding against the real LocalAgentLoop → search_skills path in nodespace-core, since a positive result there is what actually gates any change to skill_ops.rs or the retrieval/confidence logic per this issue's "Where to run it" section. That confirmation has not been run yet.

   Raw model output retained for every scored generation across all 9 conditions (controls, both arms, and 6 diagnostics) via NODESPACE_BENCH_TRACE.
4. malibio commented on Jul 30, 2026

   

   malibio

   on Jul 30, 2026

   CollaboratorAuthor

   More actions

   Core-side confirmation — pre-registered before any run

   Per this issue's "Where to run it" section: Stage 1 identified a mechanism (topical mismatch between a returned skill and the request) rather than confirming the original hypothesis (payload shape). This run confirms that mechanism against the real LocalAgentLoop → search_skills path — the thing that actually gates a change to skill_ops.rs. It is not a fresh exploration of the resident prompt, context block, or retrieval rate; those are #1813/#1817. Scope is bounded to one claim: does a genuinely wrong skill match degrade action in the real loop the way it did in the Stage 1 harness.

   What's real, what's substituted

   Harness pattern follows the reference e2e_prompting_stack.rs design (uncommitted per this issue's instructions, same as before): the actual nodespace_agent::local_agent::agent_loop::LocalAgentLoop::run_turn drives the turn — duplicate-write guard, contains_action_claim, iteration cap, all real. Two substitutions only:

   • AgentToolExecutor → an in-memory stub graph, so the mutation is observable without a daemon or DB.
   • Inference → LlamaChatInferenceEngine loading Gemma 4 E4B in-process (production's real inference path, no server).

   search_skills's result is not fabricated text — it returns the actual seeded skill content read from packages/agent/src/skill_pipeline.rs:

   • Correct-match control: the real "Graph Editing" skill — description and markdown_content exactly as graph_editing_guidance() produces them (skill_pipeline.rs:119-142), tool_whitelist ["update_node", "update_task_status", "get_node", "search_nodes", "search_semantic"].
   • Mismatch arm: the real "Bulk Import" skill — bulk_import_guidance() (skill_pipeline.rs:185-201), description "Import documents and create node hierarchies from markdown…", tool_whitelist ["create_nodes_from_markdown"] only. This is the specific skill named in the earlier retrieval-wording finding (35%→75% rank-1) as a semantic attractor that captured unrelated intents — so the mismatch under test is a real, previously-observed failure mode, not an invented one.

   Scenario

   Same request class as Stage 1: ask the agent to change an existing node's status ("Mark the Kessler order as shipped"). search_skills has already been called and returned one of the two skills above; its result is the input under test. The registered update_node tool schema (real, from the agent's tool registry) is offered alongside whatever the matched skill's tool_whitelist exposes — for the Bulk Import arm this means update_node is not in the whitelist the skill claims, mirroring what a real wrong-match would hand the model.

   Mismatch is seeded synthetically — search_skills's result is forced rather than waiting for real retrieval to misfire — per this issue's guidance: the mechanism is already established from Stage 1, so the open question is only "does it hold in the real loop," not "how often does it occur." Occurrence rate is explicitly out of scope here (it's what the retrieval-wording work already measures).

   Scoring

   Outcome-scored on the stub graph, same principle as Stage 1: did update_node land on the correct id with status → shipped? Not tool-name matching, not turn-by-turn parsing — the actual mutation. Raw transcripts retained for every trial.

   Decision rule — fixed before any trial runs

   • n ≥ 30 per condition, run across separate processes (same Metal-leak mitigation as Stage 1).
   • Control gate first: correct-match (Graph Editing) must land ≥ 90% successful mutation. If it doesn't, the harness — not production — is suspect, and the run is discarded.
   • Confirmed if the mismatch arm (Bulk Import) separates from the control by ≥ 25 points in the failing direction, and the failure mode matches Stage 1's signature — the model either follows the wrong skill's procedure or asks for information it already has, rather than a generic empty/refusal.
   • Not confirmed if separation is < 25 points, or the mismatch arm fails for a different reason than Stage 1 identified (e.g., tool_whitelist scoping alone blocks it structurally, which would be a scoping artifact, not evidence of the same suppression mechanism).
   • Six phrasings minimum (reuse Stage 1's six cases where the status-update framing applies), same seed-disjointness-across-processes discipline as Stage 1.

   Explicitly out of scope for this run

   No resident-prompt ablation, no context-block variation, no retrieval-rate measurement, no exploration of other skills beyond the one confirmed pairing (Graph Editing correct / Bulk Import mismatch). A result that surfaces something outside this claim becomes a new issue, not additional scope here.
5. malibio commented on Jul 30, 2026

   

   malibio

   on Jul 30, 2026

   CollaboratorAuthor

   More actions

   Core-side confirmation results

   Harness follows the pre-registration above: real LocalAgentLoop::run_turn (duplicate-write guard, iteration cap, all real), real LlamaChatInferenceEngine loading Gemma 4 E4B in-process, real seeded skill content (graph_editing_guidance() / bulk_import_guidance() transcribed from skill_pipeline.rs), real update_node tool schema (tools.rs::def_update_node), real search_skills wire shape ({"query", "matches"}, matching exec_search_skills's rename of find_skills's skills key). Uncommitted per this issue's instructions.

   Two harness bugs surfaced building this, both before any scored run — consistent with the "six harness bugs, all clean-looking" pattern from the original spike:

   1. AgentSession::dynamic_context is silently ignored whenever system_prompt_override is set (agent_loop.rs:437-444 — override bypasses the PromptAssembler entirely, and only the assembler interpolates dynamic_context). A first control run scored 0/6 for a reason unrelated to the variable under test: the model had no workspace context and correctly asked for a node id.
   2. Fixing (1) still scored 0/6. The actual cause: system_prompt_override also bypasses agent_guidance::TOOL_STRATEGY_RULES — the real resident rule set that includes "ALWAYS search_nodes first before update_node" and the read-then-write completion rule. Without it (bare identity string only), the model — correctly, given what it was shown — had no instruction telling it a search tool existed for this purpose. Importing TOOL_STRATEGY_RULES directly (it's pub) fixed this.

   Control gate (n=36 total, two batches)

   batch	correct	errors
   n=6	6/6	0
   n=30	29/29 scored	1 (GPU allocation failure under concurrent load on this machine, auto-retried once, still failed — excluded from denominator, not scored as suppression)
   total	35/35 scored (100%)	1

   ≥90% threshold: met.

   Mismatch arm (n=36 total, two batches)

   batch	correct	errors
   n=6	6/6	0
   n=30	30/30	0
   total	36/36 (100%)	0

   What actually happened

   In every trial, both arms, the model called search_nodes → update_node directly and never called search_skills at all. Confirmed via full tool-call/args/result dump on sampled trials — not an artifact of my scoring, the turn genuinely never touched skill discovery.

   Root cause: agent_guidance.rs:48, TOOL_STRATEGY_RULES — "ALWAYS search_nodes first before update_node or update_task_status — even if a node ID appeared earlier in the conversation. Never skip the search step." — is resident prose that hardcodes the routing for this operation class directly, independent of and bypassing search_skills. Compare SCHEMA_CREATION_RULES/TOOL_STRATEGY_RULES's create_node line: "call search_skills(query) THEN immediately call create_node in the SAME turn" — create_node's routing genuinely depends on search_skills; update_node's does not.

   This means: the mismatched skill payload (Bulk Import's instructions, its create_nodes_from_markdown-only whitelist) never reached the model's context in any trial of either arm. The 100%/100% result is not evidence that mismatch is harmless — it's evidence that update_node doesn't route through search_skills at all, so this scenario cannot exercise the claim under test ("does a wrong skill match degrade action in the real loop").

   One more thing worth naming: Graph Editing's own skill instructions already say "FIND THEN UPDATE: ... call search_semantic or search_nodes first" — the same content as TOOL_STRATEGY_RULES:48, in a second channel. For update-type operations, the same routing rule currently lives in both resident prose (line 48) and skill instructions, and the resident copy is the one the model actually acts on here. That's a rule-4/rule-5 boundary question independent of #1814's core claim — resident prose is supposed to own identity/policy only, not operation-routing detail that's also expressed in a skill.

   Where this leaves the pre-registered decision rule

   The rule as written doesn't have a clean branch for "the harness never exercised the payload." Neither "confirmed" (≥25pt separation in the failing direction) nor the not-confirmed conditions I listed anticipated this — I didn't pre-register what to do if the variable under test never reaches the model. Flagging that gap rather than picking a reading unilaterally.

   Question for you

   Two ways I see to close this, and I don't want to choose without checking:

   1. Close as-is, report the boundary. Stage 1's finding (topical mismatch → suppression, mechanism identified) stands on its own as harness-level evidence. This confirmation is inconclusive for update_node specifically — not a refutation, not a confirmation, a scope miss caused by that operation's routing being hardcoded outside the skill system. ADR-038 line 81 retraction rests on Stage 1 alone; core-side confirmation neither supports nor contradicts it for update-type operations.
   2. One more pre-registered run, create_node scenario. create_node's routing genuinely depends on search_skills per TOOL_STRATEGY_RULES ("call search_skills THEN immediately call create_node") — same operation class Stage 1's original 100%→22% measurement used (schema/instance creation). Re-running the confirmation there guarantees the mismatched payload actually reaches the model, which this run did not.

   I'd lean toward (2) since it's what the pre-registration was trying to test and (1) leaves the core-side confirmation AC technically unmet rather than answered — but this is your call, not mine to make unilaterally given the scope discipline set for this issue.
6. malibio commented on Jul 30, 2026

   

   malibio

   on Jul 30, 2026

   Last edited by malibio

   CollaboratorAuthor

   More actions

   Guidance on closing out the confirmation run

   Read the full comment above, not just the summary. Two things changed since the fork was framed, and they affect the recommendation.

   The finding here is not "inconclusive" — it's a second, independent discovery

   This run surfaced the same class of problem Stage 1 found, one layer up. Stage 1 found that a mismatched skill's instructions degrade action once they reach the model. This run found that update_node never reaches the skill system at all — TOOL_STRATEGY_RULES:48 hardcodes its routing in resident prose, bypassing search_skills unconditionally. And you found the same routing rule duplicated in Graph Editing's own skill instructions, with the resident copy winning — the exact coreValues/core_values pattern from #1809, in a different tool.

   So don't write this up as "the confirmation was inconclusive." Write it up as two things: (a) the core-side confirmation for update_node specifically is inconclusive, for the reason given, and (b) a new, independently valid finding — resident prose hardcodes routing for at least one operation class, duplicating content that also exists as a skill, and the duplicate is dead code the model never reaches. That second part stands on its own regardless of what a create_node run shows. Don't let it get compressed into a footnote.

   Proceed with (2) — the create_node run — but scope it tighter than last time

   Agreed with your lean, for the reason you gave: the pre-registration was trying to test whether a mismatched skill suppresses action once reached, and update_node structurally couldn't test that. create_node can, and it's the same operation class Stage 1's original 100%→22% measurement used, so a positive or negative result there closes the loop back to where this issue started.

   Before running it:

   1. Pre-register the same way, in a comment, before any arm runs. Same control gate (≥90% positive, ≤25% negative), same n≥30, same ≥25pt threshold. Don't reuse this run's numbers as a baseline — different operation class, different routing path, needs its own control.
   2. Verify the routing precondition before scoring anything. You were burned twice building this harness by an untested assumption (dynamic_context silently dropped, then TOOL_STRATEGY_RULES silently dropped). Before running the mismatch arm, confirm on a handful of trials that search_skills is actually being called and the mismatched payload actually lands in context — the same class of check that caught both bugs this time. Don't wait for a suspicious 100%/100% to go looking.
   3. Amend the decision rule to cover the gap you found. You're right that "the harness never exercised the payload" has no branch in the current rule. Add one now, before the create_node run, not after: if the mismatched skill is not observably present in the model's context on inspected trials, the run is invalid and must be discarded before scoring — the same standing as a failed control gate. This closes the gap for any future confirmation run too, not just this one.
   4. If create_node also fails to exercise the payload (unlikely given the routing rule you quoted, but you've been surprised twice already) — stop there. Two operation classes both failing to reach the claim under test would itself be the finding: the real loop's routing may be too hardcoded in prose for search_skills mismatch to be reachable in practice, which would matter a great deal to Implement ADR-038 Stage-1 and Stage-2 gates; shipped loop is the rejected single-turn pull #1815's design.

   On closing #1814 itself — updated

   The update_node routing bypass has its own issue now: #1820, "TOOL_STRATEGY_RULES conflates safety policy with tool routing." Filed with the evidence from this run already in it (the search_nodes-before-update_node hardcode, and the Graph Editing skill duplication it shadows). Cross-reference #1820 rather than writing that finding up again here.

   #1820 also changes the precondition for the create_node run below. TOOL_STRATEGY_RULES lines 41/55 ("call search_skills THEN immediately call create_node") are the same category of prose-routing that turned out to bypass search_skills for update_node — a doc claim, not yet independently verified at the trial level. Given this run already found two silent-drop bugs (dynamic_context, then TOOL_STRATEGY_RULES itself) before scoring anything real, don't take "create_node's routing depends on search_skills" on faith either.

   Before scoring the create_node arm: verify on a handful of sampled trials that search_skills is actually being called and the mismatched payload actually lands in context — the same check that caught both bugs this time, done proactively instead of after a suspicious result.

   • If it fires as expected: proceed with the pre-registered run as planned.
   • If it does NOT fire: that is not a second inconclusive result to shrug at. Two operation classes both failing to reach the claim under test is itself a finding — it means TOOL_STRATEGY_RULES conflates safety policy with tool routing; split it and delete routing in favor of skill/schema-carried routing #1820's problem is broader than the two lines currently identified, and worth a comment on TOOL_STRATEGY_RULES conflates safety policy with tool routing; split it and delete routing in favor of skill/schema-carried routing #1820 rather than a third confirmation attempt here. In that case, close Resolve whether skill instructions delivered as a tool result suppress action #1814 citing TOOL_STRATEGY_RULES conflates safety policy with tool routing; split it and delete routing in favor of skill/schema-carried routing #1820 rather than attempting another operation class.

   Once the create_node run lands (confirm, refute, or "didn't reach the model" per above), close with these findings, not one bundled conclusion:

   • Stage 1's mechanism (topical mismatch suppresses action) — stands, harness-level
   • Core-side confirmation for create_node — whatever it shows, including a null-because-never-reached result
   • The routing bypass — cross-referenced to TOOL_STRATEGY_RULES conflates safety policy with tool routing; split it and delete routing in favor of skill/schema-carried routing #1820, not restated
7. malibio commented on Jul 30, 2026

   

   malibio

   on Jul 30, 2026

   CollaboratorAuthor

   More actions

   create_node confirmation — pre-registered before any run

   Per your last comment: update_node's routing is hardcoded in TOOL_STRATEGY_RULES:48 (now #1820), so that confirmation could not exercise the claim under test. create_node's routing is different per agent_guidance.rs:41 — "For create_node (adding an instance): call search_skills(query) THEN immediately call create_node in the SAME turn" — genuinely gated on search_skills, and it's the same operation class Stage 1's original 100%→22% measurement used (schema/instance creation). This re-runs the confirmation there.

   New precondition, added per your instruction

   Before scoring any batch: verify on sampled trials that search_skills is actually called and the arm's payload actually lands in the model's context (visible in tool_calls_made's search_skills entry and in what the next tool call's reasoning reflects). This is the same class of check that caught both silent-drop bugs in the update_node run, done proactively now rather than after a suspicious result.

   New decision-rule branch, added now per your instruction, applies to this and any future confirmation run under this issue: if the arm's payload is not observably present in the model's context on inspected trials, the run is invalid and discarded before scoring — same standing as a failed control gate. If this happens for create_node too, that is not a third inconclusive result to shrug at — it is itself a finding (routing may be too hardcoded in prose for search_skills mismatch to be reachable in practice at all), gets a comment on #1820, and #1814 closes citing #1820 rather than attempting a fourth operation class.

   Scenario

   Real seeded "Node Creation" skill (skill_pipeline.rs:283-320) as the correct-match control — description: "Create new nodes, records, entries, or instances of any type... Use when user wants to add, create, or insert a new item, record, entry, or example of an existing type," tool_whitelist: ["create_node", "search_semantic", "search_nodes", "get_node"], guidance opens with "⚡ IMMEDIATE ACTION REQUIRED: Call create_node NOW..." — an unusually forceful positive control, good for isolating whether mismatch specifically (not weak guidance generally) is the variable moving the number.

   Mismatch arm: real seeded "Bulk Import" skill again (tool_whitelist: ["create_nodes_from_markdown"]) — same skill as the update_node run, for consistency across this issue's confirmations, and because it's a real, narrowly-scoped, genuinely-wrong skill for an instance-creation request (a single freeform document-import tool vs. a typed single-record creation).

   Request: "Add a freelance gig with client Hollis, rate 900, status booked" against a seeded freelance_gig schema (client, rate, status fields) — mirrors Stage 1's harness-validated request shape (packages/agent's reference harness used the identical phrasing for its create_schema control), reused here for a create_node instance rather than a new schema.

   WORKSPACE_CONTEXT extended to list freelance_gig in RELEVANT ENTITY TYPES so the type is resolvable, matching how the update_node run's purchase_order entry was required.

   Scoring

   Outcome-scored on the stub graph: did create_node land with node_type="freelance_gig" and the three properties correctly set? Not tool-name matching — the actual mutation, same principle as both prior confirmations in this issue.

   Decision rule

   • Precondition check first (see above) — on a minimum of 3 sampled trials per arm before any scored batch, confirm search_skills fires and the arm's real payload is visible in context.
   • n ≥ 30 per arm, separate processes, disjoint seeds, same as before.
   • Control gate: ≥90% correct creation. Below that, harness is suspect, run discarded.
   • Confirmed if mismatch arm separates from control by ≥25 points in the failing direction, and the failure mode matches Stage 1's signature (follows the wrong procedure, or asks for information already available — not a generic empty/refusal).
   • Not confirmed if separation <25 points.
   • Invalid (new branch) if the precondition check fails — see above.

   Out of scope, same as before: no resident-prompt ablation beyond what's needed to reach the model's actual behavior, no retrieval-rate measurement, no exploration of other skills.
8. malibio commented on Jul 30, 2026

   

   malibio

   on Jul 30, 2026

   CollaboratorAuthor

   More actions

   create_node confirmation results

   Precondition check (run before any scored batch, per the pre-registration): search_skills reached the model 5/5 trials in both arms. Proceeded to scored batches.

   Control — real "Node Creation" skill (n=38, two batches)

   batch	correct	search_skills called	errors
   n=8	8/8	8/8	0
   n=30	30/30	30/30	0
   total	38/38 (100%)	38/38	0

   Mismatch — real "Bulk Import" skill (n=30)

   correct	search_skills called	create_node called with right node_type	errors
   0/30 (0%)	30/30	30/30	0

   100-point separation, well past the pre-registered ≥25pt threshold, in the failing direction predicted.

   The failure mode

   Not empty output, not refusal, not a wrong tool. All 30 mismatch trials are identical: the model called search_skills (received Bulk Import's payload), then called create_node with the correct node_type: "freelance_gig", but content: "Hollis freelance gig booking" and properties: {} — every field the user supplied (client, rate, status) silently dropped. The turn then reports success: "Done — node creation completed successfully."

   This tracks Bulk Import's guidance shape exactly, not Node Creation's: Bulk Import says "pass the markdown content directly... default node_type to text," nothing about a properties map; Node Creation says "REQUIRED FIELDS: ... MUST be included in the properties map." The model followed the wrong skill's structure (content-only, no structured fields) applied to the right tool.

   This is a different failure signature than Stage 1's ("I need more information, could you provide the ID?") and different again from the update_node run's non-finding. It's silent data loss reported as success — arguably a worse failure mode than either, since nothing in the transcript signals anything went wrong.

   Some caveats to weigh

   • One thing this doesn't rule out: whether the missing-field failure follows specifically from this mismatch (Bulk Import) or would occur with any payload that omits field-shape guidance — I didn't run a third arm isolating that. Flagging rather than overclaiming: this confirms mismatch-degrades-action, but doesn't yet isolate mismatch-specifically vs missing-structure-generally as the operative variable within the payload.
   • Machine was under heavy contention through parts of this run (documented in-thread); scored batches were only taken from processes that returned clean (no errors), consistent with the pre-registered exclusion rule.

   Closing verdict for #1814

   Three separate findings, not one bundled conclusion, per your instruction:

   1. Stage 1's mechanism — stands, harness-level. Topical mismatch between a returned skill and the request suppresses/corrupts action. Isolated harness, n=36 per arm, controls validated, 2400/3200 non-monotonicity resolved as sampling noise at the original n=3.

   2. Core-side confirmation for create_node — CONFIRMED. Real LocalAgentLoop, real seeded skill content, real tool schemas, real wire shape. Control 38/38 (100%), mismatch 0/30 (0%), 100-point separation. Precondition verified before scoring (search_skills fires reliably for this operation, unlike update_node). Failure mode: silent field-data loss reported as success, not the narration/refusal Stage 1 measured — the mechanism generalizes across the delivery channel but the symptom is operation-dependent.

   3. update_node confirmation — inconclusive, cross-referenced to #1820, not restated here. That operation's routing is hardcoded in resident prose (TOOL_STRATEGY_RULES:48), bypassing search_skills entirely — a separate, now-filed problem.

   ADR-038 line 81 Provisional marker

   Retracted, and now confirmed at both the harness and production-loop level for at least one real operation class (create_node). The evidence does not show that returning skill instructions as a tool result per se suppresses action — the channel itself is not the problem (confirmed twice: Stage 1's shape arms, and this run's clean 100% control through the same channel). What's confirmed harmful is returning a mismatched skill's instructions — verified now through production's actual search_skills → LocalAgentLoop path, not just an isolated harness. The fix implied is retrieval precision (ADR-064 rule 3), not delivery channel (rule 4) — consistent with #1820's independent finding that resident prose currently duplicates and sometimes preempts routing that should live in skill instructions.

   Recorded per this issue's requirement that the procedure and numbers live in this thread since neither harness is committed. Both confirmation_1814_skill_mismatch.rs and confirmation_1814_create_node_mismatch.rs remain uncommitted scratch in the worktree unless you want them kept.
9. malibio

   closed this as completedon Jul 30, 2026