Remove create_schema's "ADR" worked example; record the tool-surface fidelity gap and two retractions

[malibio]

What this changes in production

Two description strings in create_schema (packages/agent/src/local_agent/tools.rs). That is the entire production change. No wire format, no param struct, no handler, no skill_rules.rs.

-"description": "Display name for the entity type, singular (e.g. \"Ticket\", \"ADR\")."
+"description": "Display name for the entity type, in the user's own words, singular."

…and the same example removed from the tool's own top-level description.

Why: on a request reading "We need to start tracking architecture decision records" — which never abbreviates — the model named the created type "ADR" 3/3. The abbreviation appears nowhere else in the prompt.

Located by prompt dump, not inference. Genericising the guidance's worked example changed nothing; deleting it changed nothing; a NODESPACE_PROMPT_DUMP capture found the only occurrence in the schema.

This is the mirror of what #1932 guards. That guard keeps eval prompts from sharing text with guidance so a pass cannot prove memorisation. Here a schema supplied an incidental detail the model adopted as the answer to a question the example was not addressing.

Verified: "Architecture Decision Record" 3/3 after the fix, where trimmed arms previously produced "ADR" 3/3.

The larger part: measurement evidence, including two retractions

The fidelity gap that started this

The golden corpus hand-authors 1–2 tools per turn. Production Stage 2 sends the union of the retrieved candidates' whitelists — 9 tools with full parameter schemas — because the read tools appear in 5–6 of the 8 seeded skills.

	prompt	tool declarations
dev-status-change-enum turn 1	2,679 chars	1
real production Stage 2	24,636 chars	9

So the corpus measured 11% of what production sends, and the missing 15,373 chars were entirely tool declarations. Handing the model one tool and asking it to call that tool does not test tool selection — no wrong answer is available.

Rebuilt at fidelity (production-baseline.toml, 9 tools / 3 candidates / 22,064 chars) the case still passes 5/5, and its filter is better than the 1-tool arm's: the real schema teaches the structured form where the hand-authored one produced a loose {"status":"in_dev"}.

What the arms established

• Dropping all instruction subtrees changed nothing at fidelity — but keeping only rank-1's failed 5/5. Partial injection is worse than none.
• The repair is one sentence, not a mechanism. A load_procedure tool was offered in three arms and called zero times. Lazy loading was not needed.
• K=1 breaks multi-turn: scoping to one skill's whitelist removes tools later turns need.
• Neither shape nor semantics explains which rule works. Three unrelated rules each independently restore a missing coreValues.value; the one that fails is the longest of the four.
• An irrelevant rule in an emptied guidance slot produced malformed output, 3/3 — the model echoed create_schema's own description back as an argument value. Off-topic prose is worse than an empty slot.

Two retractions, recorded rather than deleted

No trim was applied to skill_rules.rs. The KEEP/DROP rule I derived held on the corpus's own guidance but does not survive proper pairing against production's, so it does not license editing source.

core#2182 was wrong and is closed by #2183. Its evidence came from a fixture declaring value as a bare string, so the model was complying with the schema in front of it.

Both errors are the same one: measuring a hand-authored fixture and attributing the result to production. FINDINGS.toml sections 5A/6A/7A record it as a method note, the six superseded arms are marked in place, and what still holds is listed separately from what does not.

build_prod_paired.py aborts if a DROP claim cannot be verified against the real schema, so the class of error cannot recur silently.

Verification

check	result
test:all	exit 0 — 4,794 passed, 227 files
cargo test -p nodespace-agent	522 passed, 0 failed
Snapshot gate (#2119)	5 passed; golden diff is exactly the two intended lines
All 8 dev-domain golden cases, 3 reps	every one matches its recorded baseline
Rebased onto 458d5de2	#2183/#2184/#2185/#2186; gate re-run, still passes

Agent matrix: not used as a gate here, and deliberately. Its scenarios are equipment/albums/venues — the domain #1977 exists to retarget — so it cannot speak to a dev-domain schema-description change. A partial control run on plain main reproduced scenario 4's failure with the identical message, and given this eval's own noise band (identical code has scored 6/7/7) no regression claim is made in either direction.

For the reviewer

A real call I'd rather surface than bury: this adds ~45 ablation arms and 13 generator scripts, and two of my conclusions were retracted mid-spike. Whether that volume of negative-result material belongs in the repo — or would be better as an issue with the arms dropped — is a judgement worth making explicitly.

The dump_tool_defs bin is small and load-bearing for building future arms from source rather than transcription.

Reproduce any arm, no daemon or DB:

cargo build --release -p nodespace-agent --bin golden_runner
./target/release/golden_runner --reps 5 packages/agent/goldens/ablation/<arm>.toml

Related: #2122 (corpus), #2119 (snapshot gate), #1977 (eval retarget, still open), #1932 (contamination guard).

[malibio]

Pragmatic Code Review — PR #2188

Verdict: APPROVE (with one follow-up worth filing, detailed below)

The production change is 17 lines, correct in direction, and well-evidenced. The bulk of the diff is measurement provenance and is defensible in this repo given established precedent. The one substantive finding below is a completeness gap in the fix, not a defect in the change — it does not block merge, but it should be tracked before this is treated as closed.

---

Summary

Net positive, clearly. This PR does three things well that are rare in prompt-engineering work:

1. It locates a defect by prompt dump rather than inferring it. The NODESPACE_PROMPT_DUMP capture pinning the occurrence of "ADR" to create_schema's own schema is the difference between a fix and a guess.
2. It retracts two of its own conclusions in-branch, in writing, with the mechanism of the error named (FINDINGS.toml §5A/6A/7A). Retracting a recommendation you have already made is the expensive kind of honesty.
3. It changes the minimum surface consistent with the evidence. Everything else stayed measurement.

The git log shape reinforces this: e01bd43f records evidence, af9916fb makes the change and retracts the claim. Those are two different kinds of commit and they are correctly separated.

---

Findings

Improvements

1. packages/agent/src/local_agent/tools.rs — the contaminating token is still in the prompt, in the same tool.

This is the one finding worth acting on. The PR removes (e.g. "Ticket", "ADR") from two places in create_schema. But adr survives in three other model-facing descriptions that reach the same prompt, one of which is inside create_schema itself:

• tools.rs:1194 — create_schema.relationships.items.targetType: "...(e.g., 'task', 'ticket', 'adr'). Do NOT invent types that don't exist yet."
• tools.rs:811 — create_node.node_type: "...(e.g. 'ticket', 'adr')..."
• tools.rs:1217 — update_schema.schema_id: "ID of the schema to update (e.g. 'ticket', 'adr')"

And in guidance (skill_rules.rs:124-125), RELATIONSHIP_VS_FIELD's worked example reads ADR supersedes adr — the abbreviation in both display and id form.

Why this matters for the claim as stated: FINDINGS.toml §6A says "Two occurrences in create_schema, both removed" and "the abbreviation appearing nowhere else in the prompt", and the comment at tools.rs:1136-1144 repeats "a prompt dump located the only occurrence here." That is not what the shipped prompt contains. The prod-paired-current.toml arm's own tool surface is create_schema, update_schema, get_node, create_node — all three surviving occurrences are inside that surface. The arm that measured "Architecture Decision Record" 3/3 did so with 'adr' present at least three times.

Two readings, and they have different consequences:

• The mechanism is narrower than stated — the name parameter's own example is what the model copies into name, because it is the description of the field being filled. An example in a sibling field's description is not adjacent to the question. This is the more likely reading and is supported by the 3/3 result, which is a genuine after-measurement.
• The result is fragile — the fix removed enough contamination to flip this case, but the residue is still available on an adjacent axis (e.g. targetType: "adr" when the type does not exist, which prod-guidance-trimmed-v3 already observed).

Either way, the code comment and FINDINGS overstate the search. The comment is load-bearing documentation — it exists to stop a future contributor re-adding the example — so its factual claim should match the file it sits in.

Concretely, I'd suggest:

• Soften the comment at tools.rs:1136-1144 from "the only occurrence" to something like "the only occurrence in this parameter's own description, which is the one the model was filling" — that is the defensible claim and it is a stronger mechanism statement, not a weaker one.
• Same correction to FINDINGS.toml §6A's "nowhere else in the prompt".
• File a follow-up to decide the remaining three. Note these are not obviously the same defect: create_node.node_type and update_schema.schema_id describe ids that must already exist, and their description text explicitly says "copy the id exactly from the EXISTING SCHEMAS block" / "Do NOT invent types". An example there illustrates format, not a candidate answer. That distinction is exactly the doctrine this PR is establishing, and it deserves to be written down rather than left implicit.

2. FINDINGS.toml §5 is retained but its retraction is 200 lines away.

§5A/6A/7A are inserted before §5/6/7 in file order, with a "CORRECTION (appended after sections 5-7 were written)" banner. That works when read top to bottom, but §5's own text ends with "Treat the Schema Creation trim as ready to apply and measure" — a live recommendation that §5A revokes. A reader who greps for "trim" or lands mid-file can act on the revoked line.

Nit-adjacent but cheap to fix: prefix the §5, §6, §7 headers themselves with SUPERSEDED — see §5A inline, the way the six arm files are marked. The arms got this treatment; the sections didn't.

---

On the four questions raised in review

Repo hygiene / scope — acceptable, and not a CLAUDE.md violation. packages/agent/goldens/ablation/ is established on main (9 files, via #2124 / #2159 / #2183), with a README.toml establishing the same "not a case file, a .toml so it sits beside the arms" convention FINDINGS.toml follows. This PR extends a precedent rather than setting one.

On the specific rules: the docs/ prohibition targets specs, design, architecture, ADRs — durable documents that belong in nodespace-docs. FINDINGS.toml is none of those; it is the results record for a set of fixtures, colocated with them, and it is explicitly not an ADR (the durable conclusion it feeds is ADR-064, which already exists). The scripts/ TypeScript rule scopes to scripts/ tooling; these are throwaway generators under a fixtures tree, and their README says so in the first sentence. I would not have written "do not try to generalise these into a framework" any differently.

The one real cost is that ~58 files land as inert data — nothing enumerates goldens/ablation/, so none of it is exercised by test:all or the pre-push gate. That is fine (these are run on demand by golden_runner) but it means rot is silent: if a tool schema changes, every committed arm's embedded copy goes stale with no signal. The prod-paired-* arms embed a full snapshot of model_facing_tool_definitions() output — this PR itself just changed that surface, which is exactly the drift class. dump_tool_defs + the generator abort-on-missing-key check mitigate it at regeneration time, but nothing detects staleness at rest. Worth knowing; not worth blocking on, since arms are dated evidence rather than a live contract.

Retracted material — keeping it is the right call. "A retracted measurement is still evidence about method" is correct, and the METHOD NOTE ("measuring a hand-authored fixture and attributing the result to production is the recurring error in this spike") is the single most valuable paragraph in the file precisely because it generalises from two failures. Deleting the refuted arms would delete the evidence that the method note is earned. The only change needed is §5/§6 header marking, above.

The production change itself — safe, and consistent with ADR-064. Three reasons it clears the ADR-038 Finding 2 concern:

• The removed example was in the name parameter description, not in the tool's Example call: block. That worked example survives intact ({"name": "Ticket", "fields": [...]}), so the structural pattern the model reproduces is untouched. What was removed was a value suggestion, not a shape demonstration.
• name is type: "string", required, with a snake_case derivation rule stated immediately after in the tool description. The shape is fully determined by the schema; the example was carrying no shape information. Removing it is exactly ADR-064 rule 1 ("tool schemas own argument shape") applied within a schema.
• "in the user's own words" is a good replacement because it states the provenance of the value rather than illustrating it — which is the only instruction that can distinguish "Architecture Decision Record" from "ADR". No example could have.

This is also deletion, not substitution in ADR-064's sense, and the schema-creation-irrelevant-long arm (off-topic prose → malformed output 3/3) is real support for that rule from a direction the ADR did not have.

dump_tool_defs.rs — justified as a bin. A test asserts; this dumps. Making it a test to get it compiled would be an abuse of the test harness, and the alternative — hand-transcribing 9 tool schemas into fixtures — is precisely the error class §7A's METHOD NOTE identifies. It routes through model_facing_tool_definitions(), the same accessor production and the #2119 gate use, so it cannot drift into a parallel list. 36 lines, no dependencies, documents itself. Fine.

One caveat: the doc comment says "TEMPORARY investigation aid". Per CLAUDE.md's stance on scripts/ ("prefer deleting a one-off script once its purpose is served"), either it is temporary and gets a deletion trigger, or it is the supported way to build fidelity arms and the word "TEMPORARY" should go. Given the generators README lists it as a required input for regeneration, I'd argue it is now infrastructure, not temporary. Nit: drop the word.

Python generators reading uncommitted /tmp/*.json — acceptable, with the caveat already documented. The README is candid that the captures are machine- and run-specific and gives the regeneration procedure. Two of the thirteen (build_prod_trim.py, build_adr_probe.py) read skill_rules.rs directly and cannot drift at all, which is the right property for the arms that speak to production. The parameters_schema-is-a-toml::Value-not-a-string trap being written down is genuinely valuable — that is a bug that "parsed and ran cleanly while measuring the wrong thing", found only by diffing rendered sizes, and it would certainly have been re-hit.

---

Verification spot-checked

• cargo test -p nodespace-agent --test prompt_assembly_snapshot → 5 passed. The golden diff matches the source change exactly on both lines.
• Golden update is a faithful reflection of the two description edits and nothing else — no incidental churn.
• Confirmed goldens/ablation/ is pre-existing on main and that no test enumerates it (so the 58 new files add zero test-surface risk).

I did not re-run the model arms; taking the recorded 3-5 rep results and the main agent's test:all / cargo test -p nodespace-agent runs as reported.

---

Nitpicks

• Nit: tools.rs:1136-1144 — a 9-line comment on a one-line description is unusual, but justified here: it is a do-not-re-add marker with the measurement attached, and without it the next contributor "helpfully" restores the example. Keep it; just correct the "only occurrence" claim per finding 1.
• Nit: dump_tool_defs.rs:4 — "TEMPORARY investigation aid" now understates its role (see above).
• Nit: prod-paired-trimmed.toml:44-45 — the note that the case's tool_results/expect still say allowed_values while production uses coreValues, with "read the enum assertion by intent, not by key name", is honest but is a live footgun for whoever next runs these arms. Worth an issue to re-author the corpus against production schemas rather than a standing instruction to read fixtures charitably.

[malibio]

Automated pragmatic-code-review: APPROVE (posted as a comment — GitHub blocks self-approval). Full findings in the review comment above.

[malibio]

Review addressed — 49b62959

Thank you for checking the claim rather than taking it. The central finding is correct and I had asserted it without verifying.

✅ Addressed

'adr' is still in the prompt — claim corrected. Verified all four occurrences independently:

location	text
tools.rs:811	create_node.node_type — (e.g. 'ticket', 'adr')
tools.rs:1194	create_schema.relationships.targetType — (e.g., 'task', 'ticket', 'adr')
tools.rs:1217	update_schema.schema_id — (e.g. 'ticket', 'adr')
skill_rules.rs:124	RELATIONSHIP_VS_FIELD — ADR supersedes adr

All three tools.rs occurrences sit inside prod-paired-current.toml's tool surface, so the 3/3 measurement was taken with the token present at least three times. The code comment and FINDINGS §6A both overstated this, and the comment is load-bearing — it exists to stop a contributor re-adding the example.

Both now state the narrower thing actually demonstrated: removing the token from THIS parameter fixes the name. Not that the token is absent.

Your read on why the survivors differ is recorded, and marked untested: each names an id that must ALREADY exist and says so ("do NOT invent types that don't exist yet"), so the example illustrates a format. This parameter names the thing being CREATED, which is what made its example answerable. Written into the comment so it survives without the measurement — and flagged as a prediction that could be refuted.

FINDINGS §5 revocation. You were right that marking the six arm files but not the sections left §5 ending on "ready to apply and measure". It now carries the revocation inline pointing at §5A.

Dropped "TEMPORARY" from dump_tool_defs.rs.

⏭️ Not addressed, deliberately

Removing the three surviving 'adr' examples. They are a different shape of description, the distinction is untested, and changing model-facing text on an untested theory is precisely what this branch spent its length arguing against. Worth a follow-up that measures the survivors rather than assuming in either direction.

Verification

• Comment-only in production code — confirmed no non-comment line changed in tools.rs
• Snapshot gate (Prompt-assembly snapshot gate: assert production reproduces the tuned golden prompt (zero model calls) #2119) passes unchanged — the check that would catch any model-facing text moving
• quality:fix clean, no unrelated reformatting
• Pre-push gate passed in full

Re-review

NO RE-REVIEW NEEDED. All changes are comments and documentation; the snapshot gate confirms production output is byte-identical. No logic, no model-facing text, no new functionality.