Skip to content

Remove held-off-menu-type, whose premise is false (closes #3693) - #3695

Merged
malibio merged 2 commits into
mainfrom
issue-3693-held-off-menu-type
Oct 7, 2026
Merged

malibio merged 2 commits into
mainfrom
issue-3693-held-off-menu-type

Conversation

@malibio

@malibio malibio commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Deletes the held-off-menu-type decisions scenario and the scoring only it used (heldToOfferedTypes, afterRefusal, createdRecord, two summary lines, their tests). Keeps the control's unheld/held tests.
  • Why: the held menu is bug-report, checkbox, task, text (the built-in Implementing a Task skill links task), so the request never tested the hold. Rewording to person (absent from the menu in 3/3 reps) fails the other way: the model never names it, searches and updates the bug report instead, 0 calls refused, so the pass asserts nothing. Not reworded further to win a rep.
  • Refusal of an off-menu call stays covered by held-off-menu-node and the off_menu_type unit tests.

Measured (gemma-4-e4b-q4km, isolated daemon, 3 reps, DB reset between)

  • held-on-menu-type: 3/3 on this branch and on main.
  • held-off-menu-node: 0/3 on this branch and 0/3 on unchanged main (an update_node writing the seeded spec's signed-off date runs with type_refused=false off_menu_ran=false). Not caused by this change, but the issue's criterion that it "still passes 3 of 3" does not hold; it needs its own look.

test:changed passes.

🤖 Generated with Claude Code

@malibio

malibio commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator Author

Code Review Summary

Verdict: APPROVE (comment-only, self-authored PR). Net positive: removes a scenario whose premise was measured false, along with the scoring only it used. The diff is a clean deletion (21 additions, 215 deletions across decisions.ts and decisions.test.ts).

Checked:

  • No dead code left. Grep for held-off-menu-type, heldToOfferedTypes, afterRefusal, createdRecord finds nothing in scripts/, packages/ or the docs checkout. persisted, turnHeld, offMenuCallRan, notHeld and CLARIFICATION_OPENER all still have live callers (held-off-menu-node, the control, extra).
  • Comment accuracy. The replacement comment in the fixture list matches the measurements in the PR and issue (menu bug-report, checkbox, task, text; the person rewording was never named). It records why the scenario is gone, which is what the issue asked for.
  • Tests. Control tests (unfenced -> "nothing here was measured", held -> pass) are kept and simplified; the refusal-scoring tests went with the code they covered.
  • Acceptance criteria.
    • Criterion 1 (request names a type absent in 3/3) is superseded by the issue's own fallback ("if no such request keeps the turn held and the type off the menu, delete"). Fair deviation: person was off-menu but never named, so it can't assert a refusal.
    • Criterion 2 is met via "removed with the reason recorded".
    • Criterion 3 is only half met (see below).
    • Criterion 4: PR states test:changed passes.

Findings

Critical Issues

None.

Suggested Improvements

  • Acceptance criterion 3 (held-off-menu-node passes 3 of 3) is not met, and the PR is honest that it is 0/3 on unchanged main too. That is the right call for scope (not caused by this change), but this PR now removes the other scenario that exercised a refused off-menu call end to end, and the comment in decisions.ts says refusal is "covered end to end by held-off-menu-node". While that scenario fails 0/3, the claim is true only in the sense that it exists. Please make sure the pre-existing failure has its own tracked issue (the PR body says "needs its own look" but does not link one) so that the end-to-end refusal coverage this PR leans on is not silently red. Per the team's fold-in preference, it is also acceptable to say in the PR why it is not folded here (a different scenario, a different cause: update_node ran with type_refused=false off_menu_ran=false, which may be a seeding or menu issue rather than scoring).
  • Do not close held-off-menu-type's type is on the held menu, so the scenario does not test the hold #3693 as fully satisfied without noting the unmet criterion 3 on the issue; closing with a link to the follow-up is enough.

Nitpicks

  • Nit: the tombstone comment in decisions.ts is five-plus lines of history in a list of live fixtures. It is useful as a "don't re-add this" guard, but consider trimming to the conclusion (a create outside the linked set can't be tested with the current skill set) since the measurements live in the PR/issue.

@malibio malibio left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Verdict: APPROVE (comment-only, self-authored PR). Net positive: removes a scenario whose premise was measured false, along with the scoring only it used. The diff is a clean deletion (21 additions, 215 deletions across decisions.ts and decisions.test.ts).

Checked:

  • No dead code left. Grep for held-off-menu-type, heldToOfferedTypes, afterRefusal, createdRecord finds nothing in scripts/, packages/ or the docs checkout. persisted, turnHeld, offMenuCallRan, notHeld and CLARIFICATION_OPENER all still have live callers (held-off-menu-node, the control, extra).
  • Comment accuracy. The replacement comment in the fixture list matches the measurements in the PR and issue (menu bug-report, checkbox, task, text; the person rewording was never named). It records why the scenario is gone, which is what the issue asked for.
  • Tests. Control tests (unfenced -> "nothing here was measured", held -> pass) are kept and simplified; the refusal-scoring tests went with the code they covered.
  • Acceptance criteria.
    • Criterion 1 (request names a type absent in 3/3) is superseded by the issue's own fallback ("if no such request keeps the turn held and the type off the menu, delete"). Fair deviation: person was off-menu but never named, so it can't assert a refusal.
    • Criterion 2 is met via "removed with the reason recorded".
    • Criterion 3 is only half met (see below).
    • Criterion 4: PR states test:changed passes.

Findings

Critical Issues

None.

Suggested Improvements

  • Acceptance criterion 3 (held-off-menu-node passes 3 of 3) is not met, and the PR is honest that it is 0/3 on unchanged main too. That is the right call for scope (not caused by this change), but this PR now removes the other scenario that exercised a refused off-menu call end to end, and the comment in decisions.ts says refusal is "covered end to end by held-off-menu-node". While that scenario fails 0/3, the claim is true only in the sense that it exists. Please make sure the pre-existing failure has its own tracked issue (the PR body says "needs its own look" but does not link one) so that the end-to-end refusal coverage this PR leans on is not silently red. Per the team's fold-in preference, it is also acceptable to say in the PR why it is not folded here (a different scenario, a different cause: update_node ran with type_refused=false off_menu_ran=false, which may be a seeding or menu issue rather than scoring).
  • Do not close #3693 as fully satisfied without noting the unmet criterion 3 on the issue; closing with a link to the follow-up is enough.

Nitpicks

  • Nit: the tombstone comment in decisions.ts is five-plus lines of history in a list of live fixtures. It is useful as a "don't re-add this" guard, but consider trimming to the conclusion (a create outside the linked set can't be tested with the current skill set) since the measurements live in the PR/issue.

@malibio

malibio commented Oct 7, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed: shortened the tombstone comment (nit). Skipped: filing a follow-up for held-off-menu-node 0/3. It fails the same on unchanged main and its cause is not yet known (an update_node on the seeded spec ran with type_refused=false off_menu_ran=false), so it is recorded on #3693 for the owner to triage rather than as a guessed issue. No re-review needed: comment-only change.

malibio and others added 2 commits October 8, 2026 00:22
…native is never attempted (closes #3693)

The held menu on the turn is bug-report, checkbox, task, text: the built-in Implementing a Task skill links task. Asking for a person keeps person off the menu, but the model never names it (it searches and updates the bug report), so nothing is refused and the pass says nothing about the hold. Refusal of an off-menu call stays covered by held-off-menu-node and the off_menu_type unit tests. Drops the scoring only this scenario used.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@malibio
malibio force-pushed the issue-3693-held-off-menu-type branch from a7635e4 to 02daf61 Compare October 7, 2026 22:23
@malibio
malibio merged commit b742945 into main Oct 7, 2026
@malibio
malibio deleted the issue-3693-held-off-menu-type branch October 7, 2026 22:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant