Repository navigation
Remove held-off-menu-type, whose premise is false (closes #3693) - #3695
Conversation
Code Review SummaryVerdict: APPROVE (comment-only, self-authored PR). Net positive: removes a scenario whose premise was measured false, along with the scoring only it used. The diff is a clean deletion (21 additions, 215 deletions across Checked:
FindingsCritical IssuesNone. Suggested Improvements
Nitpicks
|
malibio
left a comment
There was a problem hiding this comment.
Code Review Summary
Verdict: APPROVE (comment-only, self-authored PR). Net positive: removes a scenario whose premise was measured false, along with the scoring only it used. The diff is a clean deletion (21 additions, 215 deletions across decisions.ts and decisions.test.ts).
Checked:
- No dead code left. Grep for
held-off-menu-type,heldToOfferedTypes,afterRefusal,createdRecordfinds nothing inscripts/,packages/or the docs checkout.persisted,turnHeld,offMenuCallRan,notHeldandCLARIFICATION_OPENERall still have live callers (held-off-menu-node, the control,extra). - Comment accuracy. The replacement comment in the fixture list matches the measurements in the PR and issue (menu
bug-report, checkbox, task, text; thepersonrewording was never named). It records why the scenario is gone, which is what the issue asked for. - Tests. Control tests (unfenced -> "nothing here was measured", held -> pass) are kept and simplified; the refusal-scoring tests went with the code they covered.
- Acceptance criteria.
- Criterion 1 (request names a type absent in 3/3) is superseded by the issue's own fallback ("if no such request keeps the turn held and the type off the menu, delete"). Fair deviation:
personwas off-menu but never named, so it can't assert a refusal. - Criterion 2 is met via "removed with the reason recorded".
- Criterion 3 is only half met (see below).
- Criterion 4: PR states
test:changedpasses.
- Criterion 1 (request names a type absent in 3/3) is superseded by the issue's own fallback ("if no such request keeps the turn held and the type off the menu, delete"). Fair deviation:
Findings
Critical Issues
None.
Suggested Improvements
- Acceptance criterion 3 (
held-off-menu-nodepasses 3 of 3) is not met, and the PR is honest that it is 0/3 on unchanged main too. That is the right call for scope (not caused by this change), but this PR now removes the other scenario that exercised a refused off-menu call end to end, and the comment indecisions.tssays refusal is "covered end to end byheld-off-menu-node". While that scenario fails 0/3, the claim is true only in the sense that it exists. Please make sure the pre-existing failure has its own tracked issue (the PR body says "needs its own look" but does not link one) so that the end-to-end refusal coverage this PR leans on is not silently red. Per the team's fold-in preference, it is also acceptable to say in the PR why it is not folded here (a different scenario, a different cause:update_noderan withtype_refused=false off_menu_ran=false, which may be a seeding or menu issue rather than scoring). - Do not close #3693 as fully satisfied without noting the unmet criterion 3 on the issue; closing with a link to the follow-up is enough.
Nitpicks
- Nit: the tombstone comment in
decisions.tsis five-plus lines of history in a list of live fixtures. It is useful as a "don't re-add this" guard, but consider trimming to the conclusion (a create outside the linked set can't be tested with the current skill set) since the measurements live in the PR/issue.
|
Addressed: shortened the tombstone comment (nit). Skipped: filing a follow-up for |
…native is never attempted (closes #3693) The held menu on the turn is bug-report, checkbox, task, text: the built-in Implementing a Task skill links task. Asking for a person keeps person off the menu, but the model never names it (it searches and updates the bug report), so nothing is refused and the pass says nothing about the hold. Refusal of an off-menu call stays covered by held-off-menu-node and the off_menu_type unit tests. Drops the scoring only this scenario used. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
a7635e4 to
02daf61
Compare
Summary
held-off-menu-typedecisions scenario and the scoring only it used (heldToOfferedTypes,afterRefusal,createdRecord, two summary lines, their tests). Keeps the control's unheld/held tests.bug-report, checkbox, task, text(the built-in Implementing a Task skill linkstask), so the request never tested the hold. Rewording toperson(absent from the menu in 3/3 reps) fails the other way: the model never names it, searches and updates the bug report instead, 0 calls refused, so the pass asserts nothing. Not reworded further to win a rep.held-off-menu-nodeand theoff_menu_typeunit tests.Measured (gemma-4-e4b-q4km, isolated daemon, 3 reps, DB reset between)
held-on-menu-type: 3/3 on this branch and on main.held-off-menu-node: 0/3 on this branch and 0/3 on unchanged main (anupdate_nodewriting the seeded spec's signed-off date runs withtype_refused=false off_menu_ran=false). Not caused by this change, but the issue's criterion that it "still passes 3 of 3" does not hold; it needs its own look.test:changedpasses.🤖 Generated with Claude Code