Skip to content

Fail closed when harness pass@end is zero - #18

Merged
jmjava merged 1 commit into
mainfrom
cursor/harness-fail-closed-pass-end
Sep 11, 2026
Merged

Fail closed when harness pass@end is zero#18
jmjava merged 1 commit into
mainfrom
cursor/harness-fail-closed-pass-end

Conversation

@jmjava

@jmjava jmjava commented Sep 10, 2026

Copy link
Copy Markdown
Owner

Summary

  • scripts/run_harness.py now exits non-zero when pass@end is 0 (or below --min-pass-end). Printed rows are not a pass; the table still prints.
  • The three CI stub MCP steps (golden / apply / failover) keep invoking that same script. A stub that refuses every call can redden those steps without rewriting the workflow.
  • Added tests/test_harness_exit.py: refuse-every-call → harness exit != 0, plus a lock that those CI steps call run_harness.py without continue-on-error.

Q-SLM-20 leftovers 1 and 2 only. No live pass rate is claimed.

Test plan

  • PYTHONPATH=src .venv/bin/python -m unittest discover -s tests -v (115 tests)
  • Stub refuse (LOCAL_CODING_SLM_STUB_REFUSE=1) on golden --case whitespace_extract exits non-zero and prints pass@end=0.00
  • Same sabotage on --orchestrate --job mcp_extract_accept exits 1
  • Unsabotaged golden one-case still exits 0
  • Confirm GitHub Actions stub MCP golden / apply / failover stay green on this branch

Made with Cursor

Printed rows are not a pass; the same stub sabotage can now redden the CI stub MCP steps.

Co-authored-by: Cursor <cursoragent@cursor.com>
@jmjava
jmjava merged commit 4db47ea into main Sep 11, 2026
3 checks passed
@jmjava
jmjava deleted the cursor/harness-fail-closed-pass-end branch September 11, 2026 01:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant