Golden prompt corpus: dev-workflow cases + context ablation (closes #2122)

[malibio]

Closes #2122.

Authors the golden prompt corpus: production-shaped, dev-workflow prompt strings tuned against the locked native model until each reliably produces the right tool call. No production source modified — 12 new .toml files, nothing else.

What's here

8 dev-workflow cases, every scenario the issue lists, all measured 3/3 byte-identical:

case	result
dev-status-change-enum	3/3 — real uuid + exact enum member
dev-schema-creation	3/3 — one schema, enum with allowed_values, no contamination
dev-ambiguous-clarify	3/3 first try — route_clarify with both real ids
dev-relationship-creation	3/3 first try — correct tool and direction
dev-indirect-reference	3/3 all 3 turns — description beat recency
dev-empty-result-query	3/3 — correct outcome is prose, not a tool call
dev-unseen-schema	3/3 — the generalisation test
dev-instance-creation	listed fields 3/3; unlisted value dropped 3/3 — left failing on purpose

Plus the two inherited minimal controls, kept distinguishable as controls, and 3 ablation arms.

The ablation is the headline

The corpus was built as arms rather than one tuned shape, because production's shape was itself untested. Three arms subtracted from one case so exactly one block moves — tool surfaces verified hash-identical across all four before spending model time.

arm	turn-1 prompt	result
full	1507	3/3 correct
no-candidate-block	575 (−62%)	3/3 identical — inert
no-instruction-subtree	1204	3/3 identical — inert
no-entity-types	826	3/3 wrong — enums collapse

Removing the entire Stage-2 candidate block — REFERENCE header, candidate framing, Purpose line, instruction subtree — produced byte-identical output at 62% less prompt. The finer arm resolves which half mattered: neither. Tool descriptions alone selected the tool, which is ADR-064 rule 2 working as specified.

RELEVANT ENTITY TYPES is the only block that paid. Without it both enum values collapse to prose ("dev" not in_dev, "ready for review" not ready_for_review); both writes would be rejected and the filter would silently match nothing. What earns its context is the {a,b,c} allowed-values rendering — not the prose framing around it.

A tie doesn't license deleting a block; one case is not the workload. ablation/README.toml carries that caveat, and the issue records which axes were not run rather than sampling silently.

Two production defects found

Both are invisible-input classes where prose tuning provably cannot help:

1. Tool parameter named properties is silently deleted from the rendered tool schema — create_node/update_node guidance never reaches the model #2123 (filed) — a tool parameter named properties is silently deleted by the Gemma-4 template (key collides with JSON Schema's own properties). Production create_node/update_node both use that name, so their tuned descriptions — the allowed_values rule, ADR-063's custom: prefix rule — never reach the model. Control: one word changed, 420 chars rendered vs 694.

2. A role="tool" history message is dropped entirely. Probed with five markers; all four other roles survive. This made the model pass a ticket's title as id, 3/3. Adding "never the node's title" to the schema changed nothing — correctly, since the id was never in the prompt. Fixed by expressing prior calls as chained turns, which emit the real tool_call/tool_response pair.

Methodological upshot, relevant to #2119: dump the prompt and confirm a fact is present before tuning words that reference it. Two separate 3/3 "failures" here were both missing inputs, and only the post-template string reveals them.

Other findings worth keeping

The worked example is a lever in one case and inert in another. In dev-schema-creation a dropped detail resisted prose in two channels, then was fixed 3/3 by adding a third field to the worked example — ADR-038 Finding 2 exactly. In dev-instance-creation the identical edit did nothing. So "put it in the example" isn't general: it works when the example models a list the model is already building, and fails when the value must first be recognised as unlisted. That case is left failing rather than tuned around — a case whose recorded result flattered production would be worse than no case.

The continuation gap did not reproduce. dev-indirect-reference turn 3 went straight to the write, 3/3, where scenario 6 stops at resolve_query 0/3. Difference: the record was already in the conversation, so no resolve step stood between reference and write. Points at the resolve→write handoff specifically.

dev-unseen-schema is the sharpest case. A Release type with {cut,soak,shipped,rolled_back} — vocabulary in no example or instruction, while the worked example still says ticket/status/in_dev. The model produced soak from "baking on staging" (a word the user never says, not guessable from general knowledge) 3/3, and resisted shipped, the plausible wrong answer its own sentence invites. Inference over supplied context, resting on the block the ablation proved load-bearing — which is the evidence that users' own invented schemas work.

Provisional on #2123

Every write case uses field_values, not properties. Production says properties, so these cases do not reproduce production's tool surface today and must not be read as validated production shape. Each states the conditional explicitly: fixed by renaming, the case becomes correct as written; fixed any other way, it needs rewriting and re-measuring rather than re-annotating.

Testing

• Full-set re-run: all 10 cases + 3 arms byte-identical across 3 reps, no case broke another
• test:all exit 0 — 4703/4704 frontend, the 1 failure being the known core-plugins viewer timeout flake (passes in 2.2s isolated against a 5s limit; this PR adds no code)
• 21 golden unit tests pass; quality:fix clean; pre-push gate passed
• Baseline was 216 files / 4704 tests

Caveats

All measurements are one session. #2118 measured tool names stable while arguments varied across sessions (equipment vs Product, each 3/3 internally) — reproduced here firsthand. A second session is worth more than more reps in the first, so cross-session stability remains unverified.

The capture recipe for NODESPACE_PROMPT_DUMP is recorded on #2122 for #2119's use — notably it needs no daemon and no DB, so the isolation and stale-binary traps don't apply.

🤖 Generated with Claude Code

[malibio]

Code Review Summary

Verdict: REQUEST CHANGES — one blocking finding, in the PR's headline result.

This is unusually disciplined empirical work. The corpus is production-shaped against sources I verified rather than assumed, the standing failure is documented honestly rather than tuned around, the field_values/properties divergence is disclosed as a conditional rather than a footnote, and the prior measurements the notes lean on are real — I checked +types 25%→8%, 10,493 chars at 50% vs 445 at 73%, 72/72 inert, and 35%→75% against ADR-064's Evidence section and ADR-038 Finding 2, and every one of them is quoted accurately. The notes fields citing context_ops.rs, routing.rs, and entity_types_block.rs describe those files correctly, including the subtle claim that RELEVANT_ENTITY_TYPES_HEADER is rendered at two independent sites.

The blocker is narrow and specific: I verified the ablation arm isolation myself with --check, as asked, and one of the three arms is not the subtraction it claims to be. Since the arms are the PR's headline result, that one defect undercuts the strongest conclusion being drawn.

I made no modifications; the working tree is clean.

---

Findings

Critical Issues

1. no-instruction-subtree turn 2: the Purpose sentence is truncated mid-clause — the arm is not isolated

packages/agent/goldens/ablation/no-instruction-subtree.toml:167-172

The arm's own notes state its design contract at line 7-8: "The candidate framing kept — --- Candidate N: and its Purpose line — but the rendered instruction subtree removed." Turn 1 honors that. Turn 2 does not. The deletion took the tail of the Purpose sentence with it, and the surviving \ line-continuation splices the Purpose line directly into the following header.

What the model actually receives on turn 2 (from --check, decoded):

--- Candidate 1: Node Update
Purpose: Change a field on an existing record — status, assignee, dates, or any stored value — when the user describes a new state for something that RELEVANT ENTITY TYPES (existing types — do not recreate; do not copy their fields onto a new type):
- ticket "Ticket" -> title: text, required; ...

The words already exists. were deleted along with the blank line, so the sentence terminates mid-clause and the RELEVANT ENTITY TYPES heading is swallowed into the middle of a prose sentence. Compare turn 1 of the same arm, which is correct.

Why this is blocking rather than cosmetic:

• It is a second moved variable. The arm claims exactly one block moved. In fact two things changed: the instruction subtree was removed and the Purpose line was mutilated and the entity-types header lost its own line. Per the README's own standard (line: "two arms differing in two ways measure nothing, and a difference between arms is only attributable if exactly one thing moved"), this arm does not meet the bar it sets for itself.
• It damages the block the sibling arm proved load-bearing. no-entity-types establishes that the {a,b,c} rendering is the one mechanism that pays. This arm partially degrades that same block's presentation on turn 2 — the header no longer starts a line — while claiming to hold it constant.
• It weakens the specific conclusion drawn. The arm reports 3/3, IDENTICAL TO THE FULL ARM and the notes at lines 51-54 conclude "neither half is doing work here." A tie measured on a prompt carrying a spliced, truncated sentence is a weaker result than a tie on a clean subtraction — arguably it is the stronger finding (the model tolerated corruption too), but it is not the finding recorded, and nobody reading the file would know the difference.
• It does not reproduce what production would emit. I checked routing.rs:render_candidates_for_prompt: it writes Purpose: {}\n and then \n{instructions}\n as separate pushes, so a real absent instruction subtree leaves the Purpose sentence whole and the following header on its own line. The arm's shape is not reachable from production code.

Fix: restore already exists. and the blank line before RELEVANT ENTITY TYPES on turn 2 so it matches turn 1's shape, then re-run the arm and re-record the result. The re-run is the point — re-annotating without re-measuring is exactly what commit b2644d3d argues against for the #2123 dependency, and the same standard applies here. Verify with --check before spending model time, as ablation/README.toml instructs.

For reference, my verification of all three arms (tool surfaces hashed over name + description + serialized schema):

arm	turn	tool surface	user	tool_results	system delta
no-candidate-block	1	identical	same	same	−916, clean
no-candidate-block	2	identical	same	same	−1272, clean
no-entity-types	1	identical	same	same	−668, clean
no-entity-types	2	identical	same	same	−668, clean
no-instruction-subtree	1	identical	same	same	−296, clean
no-instruction-subtree	2	identical	same	same	−589, SPLICED

The good news: tool surfaces are byte-identical across all four cases on both turns, exactly as claimed, and no-candidate-block and no-entity-types are clean single-block subtractions. The isolation claim holds for 5 of 6 arm-turns.

---

Suggested Improvements

2. The recorded character counts do not reproduce

packages/agent/goldens/ablation/no-candidate-block.toml:47 states "The system prompt fell from 1507 to 575 characters on turn 1, a 62% cut."

Measured from the committed files via --check: 1480 → 564. The derived percentage is right (61.9% ≈ 62%), and no-instruction-subtree.toml:53's "932 characters" measures 916 — so the conclusions are unaffected. But the absolute figures are off by 27 and 11 characters, which means they were taken against a case revision that is not the one committed.

This matters more than a typo would, because this corpus's entire value proposition is that recorded measurements match what the files actually contain — the notes elsewhere are rigorous about exactly this ("dump the prompt and confirm a fact is present before tuning the words that reference it"). A reader who re-derives these numbers and gets different ones has to work out whether the prompts drifted or the arithmetic did. Please re-measure against the committed files, or state which figure is being counted (I could not reproduce 1507 as system, system+user (1511), or system+1).

3. dev-schema-creation: the causal claim is not separable from the confound it describes

packages/agent/goldens/dev-schema-creation.toml:24-36

The notes describe two edits in order and attribute the fix to the second:

1. PROSE, in the fields description … NO EFFECT. Still 3/3 without supersedes.
2. THE WORKED EXAMPLE: added a third field, blocks … supersedes appeared immediately, 3/3.

The prose from edit 1 was not reverted — it is still in the committed file at line 149 ("Work through the user's message detail by detail…"). So the winning configuration is prose + example, not example-alone. The described procedure cannot distinguish "the example is the lever" from "the example works only once the prose has primed it." No reverted-prose control is described.

This is load-bearing because the same paragraph then generalizes:

"When a case here needs an argument SHAPE changed, change the example — the prose slot is measurably not the lever."

That is a corpus-wide rule ("a case here") from one field on one case at 3 reps in one session, resting on a two-arm sequence that never isolated the arm being credited. The PR is elsewhere admirably careful about exactly this failure mode — dev-instance-creation explicitly refuses to generalize the same lever, and ablation/README.toml warns that "one case is not the workload." Applying that same discipline here would mean either running the third arm (example alone, prose reverted) or softening the claim to what was measured.

Note this claim is also doing cross-case work: dev-instance-creation.toml:33-40 builds its "same edit, opposite outcome" finding on top of it. If the schema-creation arm is confounded, the contrast is between prose+example and prose+example, which is still a valid contrast — but it is not the "identical edit" the notes describe.

Nit-adjacent, same family: line 17's "IT IS FINDING 2 REPRODUCING EXACTLY" — "exactly" overstates what 3 reps on one field supports.

4. Two cases point the model at content that is not there

Both say "Copy the type id and the field name exactly as they appear below" with nothing below it:

• packages/agent/goldens/dev-ambiguous-clarify.toml:92 — the system string ends at that line; the entity-types block is ~7 lines above.
• packages/agent/goldens/ablation/no-entity-types.toml:95 — "below" now dangles because the arm deleted the block it referred to.

In dev-status-change-enum.toml:120 the same sentence is correct, because the candidate block's own second entity-types rendering does follow it — which is what makes the two above genuine divergences rather than a house style. The sibling cases that place it correctly say "above" (dev-empty-result-query.toml:62, dev-indirect-reference.toml:86).

Impact is low — dev-ambiguous-clarify's turn 1 is scaffolding, and both cases measured 3/3 — but it means the supposedly-identical Node Search candidate blocks are not identical across cases, which quietly undercuts cross-case comparability. For no-entity-types specifically it is worth a line in the notes: the arm leaves a dangling pointer, and a reader should know that was a consequence of the subtraction rather than a second deliberate edit.

5. Overclaiming: generally well-controlled, with the specifics worth keeping

Assessed as a first-class axis per the request. The corpus is, on the whole, honest about its limits — notably better than typical. It states repeatedly that all measurements are one session at 3 reps, that tool arguments are less stable than tool names across sessions (citing the equipment/Product divergence), that a tie does not license deletion, that single runs are not decision-grade, and it records which axes were not run rather than letting a partial sweep read as complete. dev-indirect-reference explicitly scopes its "no continuation gap" claim to this surface. That is the right posture and it should survive review intact.

The exceptions are #3 above, plus one framing point worth flagging:

The commit message and issue comment both lead with "2 of 3 context mechanisms are inert." That phrasing generalizes from dev-status-change-enum alone — the only case ablated — to "context mechanisms" as a class. The case files themselves are careful to scope it ("ON THIS CASE", capitalized, in both arms), so the corpus is fine; it is the headline that outruns it. Given these findings will be read by whoever picks up #2120, I would recommend the summary say "on one case" wherever it says "inert."

---

Requirements check against #2122 (as corrected by the issue comments)

Reviewed against the correction comment, which establishes that expect is free-text prose never evaluated, that there are no typed expectation kinds (clarify, min_properties_on), and that there is no chain = false. The ACs are read accordingly.

Requirement	Status
Production-shaped goldens (same blocks, order, tool surface)	Met, with the disclosed field_values/properties exception. Block order verified against context_ops.rs and routing.rs; the double entity-types rendering is reproduced correctly.
Dev-workflow domain, no eval-fixture contamination (#1932)	Met. All 12 files use ticket/adr/release vocabulary; no equipment/laser-cutter/invoice leakage.
N reps per case, per-case results recorded	Met. reps = 3 in every file; results recorded per case, with stable fields recorded and not just tallies.
At least one case whose correct outcome is asking rather than acting	Met — dev-ambiguous-clarify, route_clarify with both real ids, stated in prose in expect per the correction.
Control vs production-shaped distinguishable	Met. The two inherited minimal controls remain; dev-indirect-reference is the production-shaped counterpart to indirect-reference-resolves for the same behavior.
Intent recorded in expect	Met, and well — expect fields name the specific miss worth telling apart, not just the target.
No production source modified	Met. Diff is 12 new .toml files, +2540 lines, zero source changes.

Verified independently: every uuid referenced in prose resolves to that case's own tool_results; every enum value claimed in expect appears in that case's own entity-types block (including dev-unseen-schema's {cut,soak,shipped,rolled_back}, where shipped is present precisely so it functions as a real distractor); turn references match actual [[turn]] counts; no case uses the role = "tool" history pattern the corpus documents as broken. Whitespace hygiene is clean across all four ablation files — no leading newlines, doubled spaces, or TOML indent leaking into model-visible strings.

The standing failure is documented honestly. dev-instance-creation states the drop in notes ("STANDING FAILURE, TUNED AT AND NOT CLOSED"), repeats it in expect ("measured DROPPED 3/3 and is a standing failure, not a solved case"), and enumerates three distinct tuning channels — candidate-instruction prose, schema-description prose, and a worked example carrying reported_by. Those are three genuinely different channels, not three phrasings of one. The file also correctly separates this failure from the #2123 one it would otherwise be conflated with. This is the right call and the case should stay failing.

---

Standards questions raised in the request

ablation/README.toml does not violate the docs rule. I checked the standard directly: documentation.md scopes the prohibition to .md files — docs/ directories, nested README.md, "any other .md documentation." A .toml sitting beside the arms it describes is not what that rule governs, and the content is operational instructions for interpreting an experiment, not architecture or design documentation. The file even opens by declaring what it is ("This directory is not a case file"). Substantively it belongs there: it carries the "a tie does not license deleting the block" caveat, which is the single most important guard on this PR's headline result and needs to travel with the arms. No change requested.

The #2123 / #2118 / #1932 / #1846 references in notes are appropriate. The rule targets documentation prose in ../nodespace-docs, for a stated reason — standalone docs that survive a migration off GitHub. These are test-fixture annotations recording which open defect a fixture is provisional on, which is exactly the traceability that makes the field_values caveat actionable rather than mysterious. Decisions are already correctly cited by ADR number throughout (ADR-064 rule 1/2, ADR-038 Finding 2, ADR-056), which is what the rule actually asks for. Existing production code sets the same precedent — routing.rs cites (#1846) in a doc comment, and context_ops.rs, tools.rs, and agent_loop.rs do likewise. Flagging these would be mechanical; no change requested.

---

Nitpicks

• Nit: no-candidate-block collapses turn 1 and turn 2 to byte-identical system prompts. That is correct and unavoidable for this arm, but it is worth one line in the notes — a reader diffing the two turns will wonder whether turn 2 lost content by accident.
• Nit: ablation/README.toml documents how to build and read an arm but does not record the base case each arm derives from as a machine-checkable field. A derived_from = "dev-status-change-enum.toml" key in each arm would make finding Set up Tauri + Svelte project structure #1 catchable by a diff rather than by inspection — relevant to Prompt-assembly snapshot gate: assert production reproduces the tuned golden prompt (zero model calls) #2119, which will need exactly that.

---

What I did not do

I did not run the model. All findings above are from static analysis and --check (no GGUF load), so the recorded 3/3 results are taken on trust except where a file's own contents contradict them. Given finding #1, re-running no-instruction-subtree after the fix is the one measurement I would want repeated before this merges.

[malibio]

Automated pragmatic-code-review: REQUEST CHANGES (posted as a comment — GitHub blocks self-approval). Full findings in the review comment above.

One blocking finding, in the PR's headline result: the no-instruction-subtree ablation arm's turn 2 is not the single-block subtraction it claims — the Purpose sentence is truncated mid-clause (already exists. deleted) and the surviving line-continuation splices RELEVANT ENTITY TYPES into the middle of it. Verified with --check. Tool surfaces ARE byte-identical across all four cases as claimed, and the other two arms are clean, so isolation holds for 5 of 6 arm-turns.

Also: recorded character counts (1507/575) do not reproduce against the committed files (1480/564); the dev-schema-creation example-vs-prose causal claim is confounded (the prose edit was never reverted); two cases carry a dangling "as they appear below".

The corpus is otherwise strong work — source citations, ADR-064/ADR-038 priors, uuid/enum consistency, and the standing failure's documentation all verified accurate. ablation/README.toml and the issue-number references in notes are both fine; no change requested on either.

[malibio]

Review addressed — all four findings, by re-measuring rather than re-annotating

Thanks for verifying arm isolation independently instead of trusting the claim. That caught a real defect in the PR's headline result, and it was in the one place where a bad arm would have been most misleading.

🔴 Blocker: no-instruction-subtree was not isolated — fixed and re-run

Correct as reported. The generator's heuristic for finding the end of the Purpose sentence consumed already exists. along with the instruction body, and the surviving \ continuation spliced the RELEVANT ENTITY TYPES header into mid-sentence on turn 2.

Two variables moved, not one — and the damaged half was the block the sibling arm proves load-bearing. Your point that this shape is unreachable from render_candidates_for_prompt is the right test for whether an arm is measuring anything production can produce.

Fixed and re-run. Isolation is now verified line-by-line against the base rather than by marker presence:

turn 1: 1507 -> 1204   ADDED: none   removed: 3 instruction lines
turn 2: 1865 -> 1285   ADDED: none   removed: 4 instruction lines

My original check tested for marker presence, which by construction cannot catch a block that is present but mangled. That was the hole.

The finding survived the correction: still 3/3 identical to the full arm — now on an arm that earns the claim.

🟡 Confounded causal claim — ran the missing arm

Also correct. dev-schema-creation claimed "prose had no effect, the example fixed it" while the arm-1 prose was still at line 149, so the winning arm was prose+example and could not isolate the cause.

Rather than soften the wording, I ran the arm that was missing: reverted the prose, kept the example. Still 3/3 with supersedes. The example alone carries it — the original claim was right but unearned, and now it is measured.

The committed case is leaner as a result: it carries the example and not the prose. The notes record all three arms in order and scope the conclusion to this case, since dev-instance-creation records the identical edit moving nothing.

🟡 Character counts didn't reproduce — corrected, with an unambiguous referent

You measured 1480/564 against my 1507/575, and the discrepancy was real: "prompt size" had three different referents in this PR — the system field alone, the escaped --check rendering, and the full post-template string.

Now quoted from the post-template prompt captured with NODESPACE_PROMPT_DUMP — what the model actually receives — with a note in no-candidate-block.toml explaining why that referent and not the others:

arm	turn 1	turn 2
full	2708	4091
no-candidate-block	1788 (−34%)	2790 (−32%)
no-entity-types	2036	3416
no-instruction-subtree	2412	3515

The −62% headline was the system-prompt field alone. A third of the whole prompt is the honest number. Conclusions unchanged, but the figure now reproduces.

🟢 Dangling pointers

dev-ambiguous-clarify said "as they appear below" with nothing below it — now names RELEVANT ENTITY TYPES directly. Re-run, still 3/3.

The one in no-entity-types I've deliberately left dangling, now documented as such. It points at a block that arm deletes, which is exactly the state production reaches when guidance outlives the block it references — a plausible contributor to the failure rather than a flaw in the arm. Tidying it would make a two-variable arm answering a question production never asks.

On the two standards questions

Agreed on both, and thanks for reasoning them through rather than flagging mechanically — particularly the point that the "a tie does not license deletion" caveat is the most important guard on this result and should travel with the arms.

On the headline outrunning the case files

Fair, and worth stating plainly: only one case was ablated. The case files scope this correctly and the issue comment lists the axes not run; the summary framing was looser than the artifacts. The claim that holds is "on this case, these two mechanisms were inert," not a general statement about the mechanisms.

Full-set re-run in progress to confirm no case broke another; will report.

[malibio]

Review addressed — all 5 findings + both nits

Each finding was re-verified against current HEAD rather than assumed covered, since substantial work landed after the review was posted (declared-field pattern applied corpus-wide, EXISTING SCHEMAS heading change, per-candidate scoping fix, arm rename). Two findings needed further action; one needed none.

#	finding	status
🔴 1	no-instruction-subtree turn 2 spliced	Fixed + re-run. Isolation now verified line-by-line (nothing added, only instruction lines removed). Finding survived: still identical to the full arm.
🟡 2	char counts don't reproduce	Re-fixed. Had gone stale a second time — see below.
🟡 3	confounded causal claim	Fixed by running the missing arm, not by softening. Prose reverted, example alone still 3/3. Case is leaner: it now carries the example and not the prose.
🟢 4	dangling "below" pointers	Fixed and standardized.
🟢 5	headline overclaims the case files	No file change needed — see below.
nit	identical turns in no-candidate-block	Recorded in the file.
nit	machine-checkable derived_from	Added as a comment — see the adaptation note.

#2 went stale twice, which is the more useful finding

You caught figures that didn't reproduce. Those were corrected — and then the EXISTING SCHEMAS heading change made them stale again. Re-measured on the post-template prompt:

arm	turn 1	turn 2
full (base)	2681	4401
no-candidate-block	1765 (−34%)	3136 (−29%)
no-schemas-block	2031	3762
no-instruction-subtree	2371	3828

Twice in one PR is a pattern, not an accident: measurements transcribed into prose drift from the files they describe. The durable fix is deriving them rather than recording them, which is #2119's job — this corpus can't solve it for itself. Worth carrying into that issue.

#4 arrived from the opposite direction

The instruction had drifted into three phrasings across cases — "as they appear above", "below", and "in EXISTING SCHEMAS". Adding a per-candidate block to every turn made "above" stale in two cases while "below" became correct, so your cross-case comparability point held even as the specific errors inverted.

All four now name the block: "exactly as they appear in EXISTING SCHEMAS". A pointer that names its referent cannot dangle when a block moves or is ablated — worth having, given that three separate findings in this work involved guidance outliving what it pointed at.

The one in no-schemas-block stays dangling on purpose and is documented as such: it measures the state production reaches when guidance survives a block that didn't.

#5 — verified, and you were right that the files are clean

grep across every committed file: no occurrence of "2 of 3 mechanisms are inert" or equivalent. The case notes scope every claim with ON THIS CASE, exactly as you observed.

The overclaim was only in my commit message and issue comment. Correcting it here: only dev-status-change-enum was ablated. The claim that holds is "on this case, these two mechanisms were inert" — not a statement about the mechanisms as a class. The un-run axes (related-entities, COLLECTIONS, resident length, the other seven cases) are recorded on #2122 rather than left implicit.

Adaptation on the derived_from nit

Attempted as a real TOML key, as suggested. It does not parse — GoldenCase uses serde(deny_unknown_fields), so a real field means editing production Rust, which #2122 must not do.

Each arm now carries # derived-from: dev-status-change-enum.toml as a comment. That preserves the provenance today; the machine-checkable form belongs to #2119, which needs the parser change anyway and is the consumer you identified for it.

Verification

8/8 cases, both controls, all three arms, 5 reps, byte-identical, with argument values asserted rather than tool names — blocked_by, ready_for_dev, stage=soak, the real uuids, from_id/to_id ordering, allowed_values, supersedes, route_clarify's option ids, assignee=priya.

Rep count raised from 3 to 5 after a 3/3 run misled me during this work: it wrongly cleared a heading change of causing a regression, and only the 5-rep comparison settled it. Your closing note — that re-running no-instruction-subtree was the one measurement you'd want repeated — was right, and it's now been re-run several times over.

Not addressed

Nothing. All five findings and both nits are resolved or explicitly answered.

🤖 Generated with Claude Code

[malibio]

Code Review Summary (re-review, b2644d3d..eff53b22)

Verdict: APPROVE with minor changes — the blocker is genuinely fixed, and the eleven commits that landed after the last review are, on the whole, better work than what they replaced. Three small accuracy defects remain, all in notes prose rather than in what the model sees; none of them changes a measured conclusion.

I re-verified every prior finding against current HEAD rather than trusting the summary, as asked, and re-derived arm isolation myself with --check rather than reading the author's isolation note. The headline correction holds up.

The most valuable thing in this round is that the corrections were made by re-running, not re-annotating. The confounded dev-schema-creation claim was fixed by running the missing arm — which returned the same answer, meaning the claim was right but previously unearned, and the case is now leaner because the prose that turned out to be inert was deleted rather than kept "just in case". The no-schemas-block re-sync went the other way: re-measuring against the new base overturned the arm's own headline, and the file records the revision above the superseded original instead of quietly replacing it. Overturning your own strongest finding and leaving the audit trail is the behavior this corpus exists to produce.

I made no modifications; the working tree is clean.

---

Verification of the 5 prior findings

#	prior finding	status at HEAD	how I verified
🔴 1	no-instruction-subtree t2 spliced	Resolved	Re-derived unified diffs of all 6 arm-turns against the base from --check. Nothing added on any turn now; every removed line is an instruction-subtree line. already exists. is intact and the header starts its own line on t2.
🟡 2	char counts don't reproduce	Resolved in substance, see Improvement #2	Post-template figures are unverifiable without a model, but I cross-checked their deltas against the system-field deltas I can measure.
🟡 3	confounded causal claim	Resolved, and well	Arm 3 (example alone, prose reverted) was run; the prose is gone from the committed file — I confirmed line 149's old text is absent.
🟢 4	dangling "as they appear below"	Resolved	All 15 production-shaped turns now say in EXISTING SCHEMAS. The one remaining below is in no-schemas-block and is documented as deliberate.
🟢 5	headline overclaims	Resolved	grep confirms no case file makes the class-level claim; every conclusion is scoped ON THIS CASE.
nit	identical turns in no-candidate-block	Recorded in the file.
nit	derived_from machine-checkable	Adapted, and the adaptation's justification is true — see below.

The derived_from claim checks out empirically, not just by inspection. I copied an arm to scratch, added a real derived_from key, and ran it: unknown field 'derived_from', expected one of 'name', 'notes', 'reps', 'temperature', 'max_tokens', 'turn'. GoldenCase at packages/agent/src/golden/case.rs:43 carries #[serde(deny_unknown_fields)]. Adapting to a # derived-from: comment was correct, and all three arms carry it.

Arm isolation — re-derived independently

Tool surfaces hashed over name + description + serialized schema; system prompts diffed line-by-line after unescaping.

arm	turn	tool surface	user	tool_results	system delta	lines added	verdict
no-candidate-block	1	identical	same	same	−914	0	clean
no-candidate-block	2	identical	same	same	−1256	0	clean
no-instruction-subtree	1	identical	same	same	−310	0	see Improvement #1
no-instruction-subtree	2	identical	same	same	−572	0	clean
no-schemas-block	1	identical	same	same	−650	1 (documented)	deliberate
no-schemas-block	2	identical	same	same	−636	0	clean

Tool surfaces are byte-identical across all four cases on both turns, and the mangled subtraction that blocked the last review is gone. This is the claim the PR's headline rests on and it now holds.

---

Findings

Suggested Improvements

1. no-instruction-subtree turn 1 drops the blank line before EXISTING SCHEMAS, producing a shape production cannot emit

packages/agent/goldens/ablation/no-instruction-subtree.toml:110-112

The turn-1 subtraction removes the instruction lines and the blank line that separated the Purpose sentence from the following header. What the model receives:

--- Candidate 1: Node Search
Purpose: Find, list, or filter existing records by title, type, or stored field value.
EXISTING SCHEMAS (do not recreate these; do not copy their fields onto a new type):
- ticket "Ticket" -> title: text, ...

Turn 2 of the same arm is correct — the blank line is present there. So the two turns of this arm disagree with each other on the shape of the block they are supposed to be holding constant.

This is the same class as the blocker I raised last round, though far milder: last time the Purpose sentence was truncated mid-clause and the header was spliced into it; here the sentence is whole and the header starts its own line, it just isn't preceded by a blank one. Nothing is added, and I want to be clear the arm is now a genuine single-block subtraction in content — this is a whitespace fidelity issue, not a second moved variable.

Why it still merits a fix rather than a shrug — the same test I applied last round gives the same answer. routing.rs:340-378 builds the candidate block as:

out.push_str(&format!("Purpose: {}\n", c.description));
if !c.instructions.is_empty() {
    out.push_str(&format!("\n{}\n", c.instructions));
}
if let Some(meta) = render_schema_metadata(&c.schema_metadata) {
    out.push_str(&format!("\n{RELEVANT_ENTITY_TYPES_HEADER}\n{meta}\n"));
}

The metadata push starts with its own \n, unconditionally — it does not depend on the instructions branch having run. So a candidate with empty instructions and present schema_metadata — exactly what this arm models — emits Purpose: …\n + \n + header. Production always separates them. The arm's turn 1 does not, and its own turn 2 does.

Since PATTERN.toml:141-152 separately records that a dangling pointer changed the model's output (turn 1 emitting the literal characters """ as query), this corpus has already established that presentation-level details in this exact region are not always inert. That is the argument for making the arm match production rather than assuming the blank line cannot matter.

Fix: restore the blank line on turn 1 so it matches turn 2 and matches render_candidates_for_prompt. Re-run — the arm is 2 turns at 3 reps and the result will almost certainly be unchanged, but "almost certainly" is what the last correction also looked like before it was measured.

2. PATTERN.toml: "every case carrying a write tool" is contradicted by dev-schema-creation, and "all five" enumerates six

packages/agent/goldens/PATTERN.toml:65-81

Two defects in one claim:

The pattern is applied to every case carrying a write tool … All five hold at 3/3 after the change:

followed by a list of six cases. Count mismatch, minor.

The substantive one: I enumerated every field_values parameter in the corpus by parsing the --check JSON. Ten of eleven declare typed sub-properties. One does not:

dev-schema-creation.toml  create-adr-type  create_node  field_values: BARE OBJECT (no declared sub-properties)

packages/agent/goldens/dev-schema-creation.toml:210-212 is type = "object" with description = "Field values for this record." and nothing inside. So "every case carrying a write tool" is false as written — there are seven cases with a write tool, six follow the pattern, and the seventh is the one the sentence's own list silently omits.

I think the omission is probably defensible: that case's subject is create_schema, create_node is only on the surface as a distractor the model must not pick, and declaring the ADR's fields on it would make the wrong answer more attractive — which is arguably the point. But the file doesn't say that, and a reader auditing "is the pattern applied everywhere?" finds a counterexample with no explanation, which is precisely the drift this corpus is trying to prevent. Either declare it and re-measure, or state the exception and why.

Note this compounds slightly: PATTERN.toml frames itself as the corpus-wide contract for #2120 to implement. A contract with an undocumented exception is the thing most likely to be implemented wrongly downstream.

3. dev-instance-creation.toml:70 still names a heading that no longer exists — and names a losing arm

`blocked_by` is now declared in two places: on the type in RELEVANT NODE
TYPES, and as a typed sub-property of `field_values` in the tool schema.

The block is called EXISTING SCHEMAS everywhere else in this file (lines 93, 135, 151, 159, 202) and everywhere else in the corpus. RELEVANT NODE TYPES is not merely a stale name — per PATTERN.toml:173 it is one of the rejected candidate headings, measured 0/5. So the sentence points a reader at a heading that (a) is not in the file and (b) was specifically discarded for causing the regression that file measures.

Straight find-and-replace to EXISTING SCHEMAS. No re-run needed — it is a comment, not a model-visible string.

---

Assessment of the previously-unreviewed work

This was the larger surface and the request was right to prioritize it. I checked each substantive commit against the files and against production source.

db84aff4 — declared-field pattern (the biggest behavioural change). Verified applied to 10 of 11 write parameters (exception in Improvement #2). The reasoning chain behind it is the strongest empirical argument in the PR: eight wording arms are individually enumerated at dev-instance-creation.toml:30-42, spanning genuinely distinct channels rather than eight rephrasings — candidate prose, schema description, three example shapes, a content boundary rule, a procedure, and that procedure with output forced. Arm 8's printed enumeration is what makes the diagnosis land: the model lists five particulars and the dropped one is not among them, which locates the loss upstream of every instruction and explains why all eight were inert. The two bounding probes (asked directly, it knows; offered route_clarify, it writes silently anyway) correctly separate "selection failure" from "knowledge failure". The arm-6 detail — declaration plus the competing prose fails, declaration minus prose passes — is the kind of finding that only shows up if you actually ran it, and it is the practical instruction for #2120.

4c0e29bb — EXISTING SCHEMAS heading. The 5/5 vs 0/5 measurement is not reproducible without a model and I'm taking it on trust. Two observations. First, the conclusion drawn is sharper than the data strictly requires and the file says so itself — it rejects the obvious reading ("the noun matters") in favor of "RELEVANT X reads as a topic label; EXISTING X (do not recreate these) reads as a constraint", which is the better hypothesis and correctly flags that ENTITY was working accidentally. Second, and worth flagging for #2120: production's constant is already constraint-framed — RELEVANT ENTITY TYPES (existing types — do not recreate; do not copy their fields onto a new type): at context_ops.rs:118. So the measured contrast is between two constraint-framed headings that differ in the noun and the leading adjective, not between a bare label and a constraint. The file's own gloss slightly undersells this; the RELEVANT NODE TYPES 0/5 row is the interesting one precisely because it is the same constraint framing with a different noun. Not a defect — the conclusion still points the right way — but the rationale would be more robust stated against what production actually emits.

1d350957 + 8365ff03 — two-site uniformity, then per-candidate scoping. Both verified mechanically and both hold exactly:

• Every one of the 15 production-shaped turns renders the schemas block exactly twice (I distinguished true block renderings from mere name-references in tool descriptions; the raw string count of 3 on some turns is a description mentioning the block, not a third rendering). Matches routing.rs + context_ops.rs being two independent sites on one shared constant.
• The per-candidate copy is scoped to the candidate's own type on all 15 turns — dev-relationship-creation and dev-unseen-schema correctly show adr/release per-candidate against a two-type resident block. The 8365ff03 fix is real and complete; the "early pass put ticket fields on the ADR case" note describes a defect that is genuinely gone.

5dd47f9a — arms re-synced, one finding revised. This is the commit I scrutinized hardest and it is the best one. The revision is fully supported by the committed files: I parsed the schemas and confirmed update_node's field_values.status carries enum = [ready_for_dev, in_dev, ready_for_review, in_review, done] while search_nodes's filters declares no enum at all. That is exactly the asymmetry the note claims — the write recovers because a second channel now carries the values, the read does not because nothing does. The revised conclusion ("the block is not uniquely load-bearing; it was the only carrier present") is more accurate than the headline it replaces, and the original is retained beneath it rather than overwritten. The follow-up it names — declaring enums on a read tool's filter parameter — is correctly recorded as not run rather than assumed.

13cdb6a3 — two-site redundancy measurement. The finding that removing the duplicate is not free (turn 1 emitting literal """ as query) is well-observed, and connecting it to the dangling-pointer class is the right generalization: guidance outliving its referent yields a plausible wrong argument, not a visible error. Three separate findings in this PR are instances of that, which is a genuine pattern.

Claim discipline

Held up better than in the last round. Spot-checking the load-bearing assertions:

• "8 arms" — individually enumerated and channel-distinct. Supported.
• "5/5 vs 0/5" — not reproducible without a model; the file states its own method note (3 reps was insufficient, a 3/3 run wrongly cleared the rename, 5 reps settled it) which is the right disclosure. Taken on trust, appropriately caveated.
• "the model is not skipping the step" — this is the claim I most expected to outrun its evidence, and it doesn't. It rests on arm 8's printed enumeration where the dropped value never enters the list, which genuinely distinguishes "skipped" from "absent from input". Correctly stated.
• "44 uses in tools.rs" (PATTERN.toml:167) — reproduces exactly for the quoted-string form "node_type" (44); the bare-word count is 96. Fine as written since it is the JSON key being counted, though the disambiguation would cost four words.
• Scoping — grep confirms zero occurrences of the class-level "2 of 3 mechanisms are inert" phrasing in any committed file. Prior finding Basic storage integration #5 is genuinely closed in the artifacts.

One residual tension worth naming, not a defect: every case file still declares reps = 3, while the PR comment reports 5-rep verification and PATTERN.toml:184-188 says rep counts "were raised to 5 for heading-level changes". These are reconcilable — --reps 5 is a CLI flag that overrides the file — so the measurements are consistent. But the files are the deliverable, and a reader running a committed case gets 3 reps against notes describing 5, with nothing in the file saying which claims need the higher count. For #2119, which will re-run these, the rep count that a claim depends on is worth carrying in the file rather than in the invocation.

Requirements check against #2122 (as corrected)

Re-read the correction comment: expect is free-text prose never evaluated, no typed expectation kinds, no chain flag. Judged accordingly.

Requirement	Status
Production-shaped (blocks, order, tool surface)	Met, with the disclosed field_values/properties exception. Block order, the two-site rendering, and per-candidate scoping all verified against context_ops.rs and routing.rs.
Dev-workflow domain, no #1932 contamination	Met. ticket/adr/release throughout; no equipment/invoice leakage.
N reps, per-case results recorded	Met, with the file-vs-notes tension noted above. Stable fields recorded, not just tallies.
A case whose correct outcome is asking	Met — dev-ambiguous-clarify, and it survived the declared-field change still asking rather than writing, which was the real risk.
Control vs production-shaped distinguishable	Met. Both inherited minimal controls untouched.
Intent recorded in expect	Met. dev-instance-creation's rewritten expect now names blocked_by as the specific regression to catch.
No production source modified	Met. 13 files, all .toml under packages/agent/goldens/, +3132/−0.

All 13 case files parse under --check. dev-instance-creation's transition from documented standing failure to passing is clean — no stale "standing failure" language survives anywhere in the corpus (grepped), expect was rewritten rather than patched, and the #2123 section correctly re-frames the case as blocked on the rename rather than merely annotated by it, since declarations inside a deleted parameter reach the model no better than its description does. That is the right reading and it is a sharper statement of the dependency than the previous version.

PATTERN.toml is accurate about the codebase where I could check it (the two-site claim, the shared constant, node_type as the tool parameter, the #2123 blocker) apart from Improvement #2.

---

Nitpicks

• Nit: PATTERN.toml:65 — "All five hold" above a list of six.
• Nit: PATTERN.toml:167 — "44 uses in tools.rs" is the count of the quoted JSON key; the identifier appears 96 times. Worth saying "44 as a JSON key".
• Nit: The post-template char figures can't be re-derived without a model. I sanity-checked them indirectly — their deltas should track the system-field deltas I can measure, and they do to within 1–9 characters (e.g. no-candidate-block t1: claimed −916 vs measured −914). The residual is consistent with template/turn-boundary overhead rather than staleness, so I'm satisfied these are current. Given this number has now gone stale twice, no-candidate-block.toml's note explaining which referent is quoted is the right durable fix, and the observation that the real remedy is deriving rather than transcribing — Prompt-assembly snapshot gate: assert production reproduces the tuned golden prompt (zero model calls) #2119's job — is correct.

---

What I did not do

I did not run the model, so recorded N/N results are taken on trust except where file contents contradict them. All findings above are from static analysis, --check, and reading production source. Given the corrections here, none of the three findings requires a re-measurement to resolve except Improvement #1, whose re-run is cheap (2 turns × 3 reps) and which I'd want done rather than reasoned about — that is the same call I made last round, and last round the measurement is what turned an assumption into a result.

[malibio]

Automated pragmatic-code-review (re-review): APPROVE with minor changes (posted as a comment — GitHub blocks self-approval). Full findings in the review comment above.

The blocker is genuinely resolved: I re-derived arm isolation independently with --check rather than trusting the isolation note, and all 6 arm-turns now add nothing to the base, with tool surfaces byte-identical across all four cases. The 5 prior findings are all closed, and the derived_from adaptation claim I verified empirically (a real key is rejected by deny_unknown_fields).

Three minor accuracy defects remain, all in notes prose, none changing a measured conclusion: (1) no-instruction-subtree turn 1 drops the blank line before EXISTING SCHEMAS, a shape render_candidates_for_prompt cannot emit and which its own turn 2 gets right; (2) PATTERN.toml's "every case carrying a write tool" is contradicted by dev-schema-creation's bare field_values, and "all five" lists six; (3) dev-instance-creation.toml:70 names RELEVANT NODE TYPES, a heading that no longer exists and that PATTERN.toml records as a losing arm at 0/5.

The unreviewed work is stronger than what it replaced. Both corrections were made by re-running rather than re-annotating — and no-schemas-block re-measuring overturned its own headline finding, with the superseded original kept beneath the revision. The two-site uniformity and per-candidate scoping claims I verified mechanically across all 15 turns and both hold exactly.

[malibio]

Re-review addressed — 3 findings + 2 secondary observations

Each verified against the source before fixing, not taken on the review's word.

#	finding	status
1	missing blank line, no-instruction-subtree turn 1	Fixed. Confirmed against routing.rs:375 — the metadata is pushed as format!("\n{HEADER}\n{meta}\n") unconditionally, independent of the instructions branch, so production always separates them and turn 1's shape was unreachable.
2	PATTERN.toml overclaims its own rule	Fixed, with the exception documented.
3	stale RELEVANT NODE TYPES	Fixed. Worse than stale — it named an arm PATTERN.toml's own table records as measured 0/5 and rejected.

#2 — the exception is now stated rather than silently true

PATTERN.toml said "every case carrying a write tool" (false) and "all five" over a list of six. Both correct as reported.

Now reads "every case whose write tool is actually exercised", and names the exception: dev-schema-creation's create_node keeps a bare field_values object because it is a distractor — measured never called, present so the model can choose wrongly between defining a type and creating an instance. Declaring its fields would tune a path the case does not measure, and would mean declaring fields for the very type (adr) the case asks the model to create and therefore deliberately withholds from EXISTING SCHEMAS.

The rule for #2120 is now exact: declare the fields of any write tool the turn can legitimately call. A tool present only to be rejected does not need them.

reps = 3 → 5 in the files

You flagged the tension between declared reps and 5-rep claims in the notes. Reconcilable via --reps, but the files are the deliverable and shouldn't need a flag to mean what they say. All 11 now declare 5 — the count adopted after a 3/3 run misled me mid-PR.

The observation that mattered most — a claim I had overstated

Verified at context_ops.rs:118: production's constant already carries the parenthetical —

RELEVANT ENTITY TYPES (existing types — do not recreate; do not copy their fields onto a new type):

So the heading arms were not label-vs-constraint, as PATTERN.toml claimed. Both 5/5 arms are constraint-framed; the two 0/5 arms dropped the parenthetical and changed the noun together.

The measurement supports "keep the parenthetical" and is consistent with the noun being irrelevant, but does not isolate them. The isolating arm — bare EXISTING SCHEMAS: with no parenthetical — is now recorded as NOT RUN rather than left implying more than was measured.

What is safe to carry into #2120: keep the parenthetical. The noun change is a consistency fix with no measured cost, not a measured improvement. That is a weaker and more accurate claim than the one I committed, and it is the second time in this PR that re-measurement narrowed a finding I had stated too strongly.

Verification

8/8 cases, both controls, all three arms — 25 turn-entries byte-identical at the files' own declared 5 reps, argument values asserted rather than tool names.

test:all exit 0; the one pre-push gate failure was the known core-plugins viewer timeout flake (passes standalone, and the retry pushed clean).

Not addressed

Nothing. All three findings and both secondary observations are resolved.

🤖 Generated with Claude Code