Repository navigation
Measure whether the three surviving 'adr' schema examples contaminate output, or only illustrate format - #2260
Merged
Conversation
… output (#2190) core#2188 removed create_schema's name-parameter "ADR" example after it caused the model to name a created type "ADR" verbatim. Three other model-facing descriptions still carry the token (create_node.node_type, create_schema.relationships[].targetType, update_schema.schema_id), each naming an id that must already exist rather than one being created — an untested prediction that they are a different, harmless case. Adds a fresh golden case (must-exist-id-{baseline,armA,armB}.toml) that exercises all three parameters against a type deliberately close to the "ADR" concept (arch_decision: proposed/accepted/superseded, a supersedes field) without ever spelling "ADR" or "adr" in the prompt. Compared at 3 reps across three arms — current text, 'adr' replaced with a neutral 'my_type', and the example deleted outright — the model produced the correct id in all 27 tool calls with zero variation between arms. The survivors neither contaminate output nor does removing them degrade id-copying: inert in both directions, not merely non-contaminating. Records the measurement in FINDINGS.toml section 6B and proposes a choosing-vs-copying addendum to ADR-064 (an example for a value being created supplies a candidate answer; an example for a value that must already exist and is already shown in context does neither, so it isn't licensed to invent one) for review rather than amending the ADR directly. Updates the in-code comment on create_schema.name to record the measured (not just predicted) status. No functional code change — the three descriptions are kept as-is; the negative result is the outcome. Also fixes an unrelated pre-existing clippy failure (collapsible match in live_resolve_query_decomposition_shapes.rs) that blocks `cargo clippy --all-targets` on main regardless of this change, discovered while running the mandatory quality gate. Co-Authored-By: Claude <noreply@anthropic.com>
…this branch's own pre-push gate)
Contributor
Author
Independent verification (mstomar125)Checked the only non-fixture file touched ( APPROVE. (posted as a comment — same account authored the PR) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PR #2188 removed the "ADR" worked example from
create_schema'snameparameter after measuring it caused the model to name a created type "ADR" verbatim. Three other model-facing descriptions still carry the'adr'token (create_node.node_type,create_schema.relationships.targetType,update_schema.schema_id), on the theory (explicitly flagged as a prediction, not a finding, intools.rs's own comment) that these are a different case: they name a value that must already exist, so the example illustrates a format, not a candidate answer to adopt — unlikename, which names the thing being created.This PR measures that prediction directly, in both directions:
Result — negative both directions, 27/27 real-inference tool calls
"adr"never appeared in any of the 27 tool calls across all three arms. Removing the examples doesn't help (nothing to fix) and doesn't hurt (id-copying ties baseline exactly) — the prediction holds.Outcome
No code removed — there's nothing to fix.
FINDINGS.toml§6B records the measurement, and a choosing-vs-copying addendum to ADR-064 is proposed there (flagged for review, not merged into the ADR doc directly): an example in a description for a value the model is choosing supplies a candidate answer and can contaminate output; an example for a value that must already exist illustrates a format and is safe to leave.Closes #2190.