Repository navigation
PR stats parity tests can publish or overwrite real gists #6
Description
What's broken
The PR stats JSON parity tests call dashboard.main() with gist publishing enabled. When gh is authenticated with the gist scope, running these unit tests can overwrite a real dashboard gist or create a new secret gist in the developer's account, even though both tests pass.
Which layer
tools/pr-management/tests/stats/test_json_parity.py: _run() does not pass --no-publish, and _install_stubs() replaces only the fetch functions.
The publishing branch is in tools/pr-management/src/pr_management/stats/dashboard.py.
The parity test helper describes the calls as stubbed, but the publishing branch remains reachable.
How to reproduce
Use a disposable Linux checkout of baseline 4556f693455ef2f250d9e07b224bd59a4c04387c, with the workspace development dependencies installed.
The recording executable below performs no network requests and never invokes the real gh.
set -euo pipefail
stub_dir="$(mktemp -d)"
export H09_GH_LOG="$stub_dir/calls.log"
cat > "$stub_dir/gh" <<'SH'
#!/bin/sh
printf 'gh %s\n' "$*" >> "$H09_GH_LOG"
case "${1:-} ${2:-}" in
'auth status') printf "Token scopes: 'gist', 'read:org', 'repo'\n" ;;
'api -X') cat > /dev/null ;;
'gist create') printf '%s\n' 'https://gist.github.com/tester/0123456789abcdef0123456789abcdef' ;;
*) exit 98 ;;
esac
SH
chmod +x "$stub_dir/gh"
PATH="$stub_dir:$PATH" uv run --directory tools/pr-management --project . \
python -m pytest -q tests/stats/test_json_parity.py
cat "$H09_GH_LOG"Observed on the unmodified baseline:
| Session state | Test result | Recorded side effects |
|---|---|---|
| Tracked legacy gist ID, no personal session state | Both tests pass | Two gh auth status calls and two gh api -X PATCH gists/<existing-id> calls |
| Neither legacy nor personal session state, tested in a separate disposable run | Both tests pass | gh gist create, then PATCH of the new gist; its ID is written to .apache-magpie-local/session-state.json |
The second scenario was verified by temporarily removing the legacy file only inside an isolated validation copy and restoring all state afterwards.
Expected vs actual
Expected: fixture-based unit tests validate the JSON output without reaching gist authentication, publishing, or session-state writes.
Actual: the tests also execute the publishing branch outside their assertions.
The proposed fix passes --no-publish to the dashboard invocation and makes both publishing entry points fail immediately if reached.
The flag must be dashboard-only: reference.main() does not accept it.
Surface area
One test file, ten added lines.
The existing JSON superset, shared-value parity, and fixture-count assertions remain in place.
Production publishing defaults and CLI interfaces are unchanged.
The tracked legacy session-state file, missing-gh handling, and a suite-wide subprocess guard are independent follow-up work.
Environment
Reproduced from baseline 4556f693455ef2f250d9e07b224bd59a4c04387c in an isolated Linux Docker container with CPython 3.12.12 and a recording gh replacement.
After the fix, both state scenarios pass with zero gh calls and no session-state writes.
Removing only the new --no-publish argument makes both tests fail with AssertionError: unit test reached the dashboard gist publish step, without invoking gh.
AI-generated with Codex (GPT-6), submitted to this fork for human audit.
Human review is pending.
Activity