A request for a new kind of record leads with another skill when its nouns are that skill's subject #3632

Overview

A request to set up a new kind of record is retrieved as a request about records when its nouns belong to another skill's subject. Graph Editing or Play Authoring then leads the turn in place of Schema Creation. Two eval:decisions scenarios fail 0 of 3 on it and nothing else tracks it. Measured on gemma-4-e4b-q4km and the locked embedding model.

Problem Statement

What Stage 1 makes of the request, and how retrieval ranks it:

The shape leads where the nouns are neutral. An incident and its severity read as Graph Editing's completion states, and a runbook as an automation.

What the miss costs: create_schema is still offered, and the model called it with both fields in every rep of both scenarios. But the fields declared on the write tools come only from the candidates at the turn's top score (routing::declare_write_tool_fields), so update_node and create_node get the leading record skill's declared fields on a turn that should create a type, and the turn is recorded as routed to that skill.

Wording does not fix it. About twenty variants were measured:

• Naming the details a new kind carries in Schema Creation's description ("…such as a priority, an owner, or a date") won the first colliding request by 0.004 and lifted the skill by 0.01 to 0.02 on requests of every kind. It then led "add Lantern Autosave to specs we keep" and "move the offline sync spec's review to next week", and passed Research & Search on a lookup.
• Narrowing Graph Editing's description to one named item moved it by 0.005 at most.
• An exclusion on Graph Editing naming the setting-up verbs did not fire: these requests sit closer to its description than to any exclusion.
• Having Stage 1 write "record type" or "a new kind of record" lifted both skills together.

Proposed Solution

The same defect was closed for adds by reading the verb off the query Stage 1 wrote (routing::is_add_shaped, routing::retrieve_candidates): an embedding weighs a request's nouns far above its verb. Measure whether the same approach holds for a new kind of record, and decide on the measurement:

• A shape read off Stage 1's query for a request to set up or start keeping track of a kind of thing, used the way the add shape is: it decides no route, it makes sure the skill that holds create_schema is among the candidates Stage 2 judges and, if measured to be needed, that it leads.
• The hard part is the cost side. "keep track of the Q4 cycle's status" and "start tracking the offline sync spec's sign-off" open with the same verbs and are about one existing record; the workspace's own type names (stage1_type_names) are one signal for telling a named kind that does not exist yet from a record of a kind that does.
• If no shape separates the two without a cost on the single-record controls, record that and change what the two scenarios assert to the outcome that matters (a type exists after the turn, with the named fields), which is what the turn already does.

Also in Scope: the Confusion Matrix Ranks Retrieval Alone

The skill confusion matrix (packages/agent/tests/it/skill_confusion_matrix.rs) ranks each request with find_skills and nothing else. Routing does more: it reads the add shape off the query and changes which candidates Stage 2 judges. So the matrix's NOT_YET_WON list mixes two things:

• Requests that routing already serves. Five are adds that Graph Editing leads on retrieval ("make a task to renew the domain": Graph Editing 0.933, Node Creation 0.888; "new note: call the plumber"; "log today's gig at the Blue Room, paid 300"; "log a new invoice for Acme, $2400, due next month"; "add another album: Blue Train by Coltrane"). Three Node Creation rewordings were measured against them and each broke more than it fixed, which is the same finding as above: the verb has to be read, not embedded.
• Requests this issue is about. "I want to keep a record of the books I lend out" leads with Research & Search (0.872; Schema Creation third at 0.848), and "add a severity field to tickets" and "rename the owner field on projects to lead" lead with Graph Editing by 0.003 and 0.024.

Rank the matrix through the candidates routing judges (routing::retrieve_candidates, as routed_candidates does in the live suite), so the list holds only requests the product does not serve, and record it again. Whatever shape this issue adds then shows up in the matrix as rows coming off the list.

Two wording leftovers from the audit ride along, each only if the measurement supports it once the shape decides the lead: Schema Creation's use_for still ends in a trigger-phrase list, and Relationship Management's last sentence ("Show which records depend on or are linked to one.") was not measured for whether it changes any rank.

One comment in the matrix is wrong and is corrected with this work: the NOT_YET_WON note on dependency questions says Research & Search "does not hold the traversal tool, so either way the question is answered without it". It does not hold get_related_nodes, but it holds search_nodes, whose relationship filter selects the nodes whose path reaches a given node, and get_node_context.

Documentation

• ADR-038 - retrieval ranks skills, the leading candidate is the one a turn is routed to, and what is offered and declared follows from the ranking
• ADR-064 rule 3 - skill descriptions own retrieval matching and hold only words for what the skill does, which is why another description rewording is not the fix
• Prompt & Skill System - how a lookup and an add are retrieved by capability, the pattern this would extend
• Updates: architecture/prompt-skill-system.md for any new retrieval shape; development/agent-eval.md if a scenario's assertion changes

Architecture Impact

Extends ADR-038, reverses nothing. A shape read off Stage 1's query that changes which candidates Stage 2 judges is the mechanism already used for lookups and adds; a third instance needs its own measured cost side and a paragraph in architecture/prompt-skill-system.md.

Acceptance Criteria

• The three colliding requests above, and the two scenario prompts as Stage 1 routes them, are measured with the chosen mechanism, with scores before and after
• Either skill-type-request-direct and skill-type-request-indirect pass 3 of 3, or their assertion is changed to the outcome with the reason recorded on the scenario
• tracking_one_existing_record_still_reaches_a_record_skill, updates_to_one_record_do_not_lead_with_schema_creation and adds_to_an_existing_list_keep_a_skill_that_can_create still pass
• a_new_kind_of_record_with_named_details_leads_or_places_schema_creation asserts the lead for every request the mechanism fixes
• No eval:decisions scenario that passes 3 of 3 beforehand regresses
• The confusion matrix ranks through the candidates routing judges; current.tsv and its digest are recorded again, and every request that is then won is off NOT_YET_WON
• Schema Creation's trigger-phrase list and Relationship Management's last sentence are each measured for removal, and kept or removed on the result
• Code passes bun run quality:fix and bun run test:changed

Technical Specifications

Reference Files

• Retrieval shapes: packages/agent/src/local_agent/routing.rs - is_add_shaped, retrieve_candidates, create_retrieval_query, lookup_retrieval_query, declare_write_tool_fields
• Skill descriptions: packages/agent/src/skill_pipeline.rs - the Schema Creation and Graph Editing seeds, with what was measured in their comments
• Retrieval guards: packages/agent/tests/it/live_skill_retrieval_stability.rs - the new-kind test records the scores above and asserts only top 3 for the colliding requests
• Eval fixture: scripts/eval/fixtures/decisions.ts - skill-type-request-direct, skill-type-request-indirect
• Eval method: ../nodespace-docs/development/agent-eval.md

Implementation Details

• The retrieval tier needs only the embedding model and runs in seconds: cargo test -p nodespace-agent --test it live_skill_retrieval_stability:: -- --ignored --nocapture
• Measure on the shape over several nouns, not on the two scenario sentences

Related Issues

• Fix the skill routing and follow-up misses the re-themed decision eval surfaced #3560 (where this was measured and recorded)
• Two plain requests end with nothing created: an add whose noun sounds like another skill, and a bare 'start tracking'; the routing eval passes the second #3533 (the add-shaped mechanism this follows)