Skip to content

System-1: a gate nothing waits for goes back to the structure pass - #239

Merged
trevhud merged 2 commits into
mainfrom
gate-blocks-something
Oct 23, 2026
Merged

trevhud merged 2 commits into
mainfrom
gate-blocks-something

Conversation

@trevhud

@trevhud trevhud commented Oct 22, 2026

Copy link
Copy Markdown
Owner

An overnight audit of 15 default compiles found hitl_gates that block nothing: no later node reads the gate or lists it in after, so the step the approval guards can run before the person answers. The reference run waits. 3 of the 16 gates in the audit's default compiles had this shape.

What changed

  • structure._check_gate_waiters, run inside structure.check: a top-level gate that no node reaches by reads or after counts as a contradiction. It goes back to the pass like the other evidence checks, names the gate, asks which step in the skill the approval guards, and says how to add the dependency (reads if the step uses the answer, after with after_quote if it doesn't).
  • Terminal gates: a gate that is the run's only exit passes as the final sign-off. Every BDR pipeline under examples/ ends this way (manual_enrollment_handoff, gold included). A gate with another exit running beside it does not pass.
  • If the gate is still unguarded after the two repairs, resolve lists it as an Open question. It adds no edge, because which step a gate guards is a reading of the skill.
  • One sentence added to the structure prompt's gate bullet, so the model usually gets it right on the first round.

Why not IR validation: every committed BDR pipeline ends in a gate nothing waits for, and that is valid. The rule depends on what the skill means, not on what the IR can run. I left the legacy (--no-system1) path alone. It could get the same check as a verification finding; that's a follow-up if wanted.

Tests (tests/test_system1_structure.py) cover the finding text, clearing it through reads and through after, a transitive dependent, the terminal sign-off and its negative half, two terminal gates, repair end to end, and the Open question after repairs run out. Four mutations were checked against them (check removed, terminal exemption removed, after ignored, resolve note removed), and each one fails at least one test.

Measurement: see the comment below.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RPP1Mjr1neTbmgg1eUaQZg

An overnight audit of 15 default compiles found gates that block nothing:
no later node read the gate or listed it in `after`, so the step the
approval guards could run before the person answered, or without the
answer. The skill's reference run waits.

The structure pass now checks it from the data flow alone. A top-level
hitl_gate with no reader through reads/after is a contradiction, sent back
like the pass's other evidence checks, naming the gate and asking which
step the approval guards. A gate that is the run's only exit passes: that
is a final sign-off, as the BDR gold's manual_enrollment_handoff is. What
survives the repairs becomes an Open question; resolve adds no ordering,
since which step a gate guards is a reading of the skill.

Not IR validation: every committed BDR pipeline ends in a terminal gate,
and the rule is about the skill's meaning, not about what the IR can run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RPP1Mjr1neTbmgg1eUaQZg
@trevhud

trevhud commented Oct 22, 2026

Copy link
Copy Markdown
Owner Author

Measured with the default compile (rote bench --fidelity through scripts/isolated-rote, repeat 1, dev skills only):

skill run gate nodes waiting on it fidelity cost
expense-audit main 2f6f891 controller_review_gate build_audit_report 0.94 $0.43
expense-audit this branch controller_review build_audit_report 0.98 $0.61
ops-report main 2f6f891 request_manager_inputs 3 check nodes 0.95 $0.47
ops-report this branch manual_data_gate 3 check nodes 0.96 $0.51

The check didn't fire on either dev skill. Their gates already have a dependent, both here and in 24 earlier default compiles. So these runs show fidelity holding, not the fix. The failing shape comes from the audit: 3 of 16 gates in its default compiles (ooo-coverage-handoff connector_gate, sdr-sales-meeting-notes-reminder calendar_event_gate, workday-adp-reconciliation request_missing_inputs) block nothing, and this check flags them. Those skills weren't recompiled here.

— Claude

review6's mutation (count in_loop nodes' reads) survived every test.
Also say plainly that one direct dependent is enough: the old docstring
described a transitive search the code does not do.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RPP1Mjr1neTbmgg1eUaQZg
@trevhud
trevhud marked this pull request as ready for review October 23, 2026 01:19
@trevhud
trevhud merged commit 2afd3cf into main Oct 23, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant