Repository navigation
Description
Overview
Asked to add a record of a type the user created a turn earlier, the local agent calls create_node with the type's display name as node_type ("feature spec"), where the type's id is feature_spec. The call is refused as an unknown type and no record is created. The routing eval's instance-not-schema-invoice scenario fails on it.
Measured on gemma-4-e4b-q4km, isolated daemon.
Problem Statement
instance-not-schema-invoice runs "Create a database for tracking our feature specs" as a prior turn, then scores "Add a spec for offline sync that's due for sign-off next Friday".
The scored turn's call, the same in each failing rep:
create_node {"content":"Offline Sync Spec","field_values":{"due_date":"2026-10-13"},"node_type":"feature spec"}
→ Unknown node_type 'feature spec': no such core type or schema. Create the schema first with create_schema, or use an existing type id.
In one rep the turn first called create_schema for feature spec again, was told the type exists as feature_spec, and still sent "feature spec" to create_node.
Results:
- Baseline build (
mainbefore Read a new kind of record off the routing query, and create the task a refused field key used to lose (closes #3603, closes #3632) #3655): 0 of 3. Two reps ended on the refusedcreate_node; one calledcreate_schemaand failed the scenario's adversarial check. - The branch of Read a new kind of record off the routing query, and create the task a refused field key used to lose (closes #3603, closes #3632) #3655 before its review fixes: 0 of 3, each on the refused
create_node. - The final build of Read a new kind of record off the routing query, and create the task a refused field key used to lose (closes #3603, closes #3632) #3655: passed in the one pass that was run. One pass does not show it fixed.
The scenario only became visible as a record that was never created once the routing eval scored record creation on an accepted create_node call (createsRecord, added in #3655). Before that it passed or failed on which tools were called.
create_node's node_type description already tells the model to copy the id exactly from the EXISTING SCHEMAS block, "character for character, including underscores". The block renders the type as - feature_spec "feature spec" -> status: text, with the id first and the display name quoted after it.
Proposed Solution
None is known. Candidates, each to be measured on the scenario at three reps before it is chosen:
- Resolve a
node_typethat names no type but is the display name of exactly one, at the local agent's tool boundary, and say so on the result. ADR-064's amendments record this shape working twice where a more informative refusal did not: the tool does what its description asks. - Change how the EXISTING SCHEMAS line renders the id and the display name, so the name is not the more copyable of the two.
- Derive schema ids in kebab-case so every type id follows the naming convention #3646 changes how type ids are derived (kebab-case). Whether a display name is then closer to or further from its id is worth measuring with it, not separately.
Documentation
- ADR-064 rule 1 and rule 4, and the 2026-10-05 amendments - tool schemas own argument shape; a refusal is a result the model has to turn into a second call, and what was done in two cases where it could not
- ADR-038 - the offered set of types a turn is held to, which a resolved name must still respect
- Agent evals - the routing layer,
createsRecord, and how to run part of a fixture - Updates: ADR-064, if the tool boundary resolves a display name
Architecture Impact
No ADR is reversed. A tool-boundary resolution would be a third instance of the pattern ADR-064's 2026-10-05 amendments record, and belongs there with its measurement.
Acceptance Criteria
-
instance-not-schema-invoicecreates the record in 3 of 3 reps - The call that creates it names the type the prior turn created, and no second type is created
- A
node_typethat matches no type and no single display name is still refused - A turn held to an offered set of types cannot reach a type outside it through a display name
- Code passes
bun run quality:fixandbun run test:changed
Technical Specifications
Reference Files
create_nodedefinition and execution:packages/agent/src/local_agent/tools.rs-def_create_node,exec_create_node,off_menu_type- The refusal:
packages/core/src- search for "Unknown node_type" - How the schemas block renders a type:
packages/core/src/ops/entity_types_block.rs-EntityTypeDescriptor::render_line - Id from a name:
packages/core/src/services/node_service/mod.rs-normalize_schema_id - The scenario:
scripts/eval/fixtures/routing.ts-instance-not-schema-invoice
Reproducing
Isolate the daemon with NODESPACE_HOME and NODESPACED_MODEL_PATH (see "Setup" in agent-eval.md), load gemma-4-e4b-q4km, then:
bun run eval:routing e4b out.json --runs 3 --between-runs <reset> --only instance-not-schema-invoiceTool arguments and error text are in the daemon log's Tool executed lines at RUST_LOG=debug.
Related Issues
- Creating a task is refused for undeclared field keys, sometimes on every call, and the routing eval passes it #3603 (where the record scoring that shows this was added)
- Derive schema ids in kebab-case so every type id follows the naming convention #3646 (changes how a type id is derived from its name)
Activity
malibio commentedon Oct 7, 2026
Frontend: 5863 passed (323 files)
Resolve a display name sent as node_type to the one offered type it n…
malibio commentedon Oct 7, 2026
The scenario id
instance-not-schema-invoiceis renamed toinstance-not-schema-spec(PR #3688); the scenario is about feature specs. Use the new id in--onlywhen reproducing. PR #3685 (display-name resolution) is merged; the 3-rep run and the ADR-064 amendment are still open.Rename the routing scenario id that still said invoice for a feature-…
Rename the routing scenario id that still said invoice for a feature-…