Repository navigation
feat(ci): nightly live-e2e canary with a four-state verdict (#502) - #560
Merged
Merged
Conversation
The live tiers are invisible between releases, so Flow-side drift surfaces ad-hoc during feature work. Hosted CI cannot cover them: they need a real authenticated Chrome profile. This adds the council-adjudicated shape from #502 — a local scheduled run publishing a sanitized verdict to a rolling issue, gating nothing. Scope is the $0 tiers only (`-m e2e_auth`); credit tiers stay manual. Publishing uses the maintainer's already-authenticated `gh`, so no new secrets, and the canary only ever EDITS its rolling issue — it never opens one, because an issue per night trains red-blindness. Rot vs. drift is decided by re-probing auth after a failure rather than by matching error names. The first live run showed why: an AuthExpiredError from the aisandbox upload path looked like session rot, but the session verified clean seconds later — a real divergence between two auth surfaces that a name-matching classifier would have buried as AUTH-EXPIRED. DEFERRED covers profile-state preconditions (held lease, or a profile/engine Chromium mismatch). Both fail closed before any browser starts, so the run never reached Flow and cannot evidence drift; they share ConfigurationError's exit 11, so only the class name discriminates. `--pull` refuses a dirty checkout instead of resetting over it: a nightly task that can destroy uncommitted work is worse than a stale canary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…gnostic Investigating the canary's first RED (#561) was only possible because tmp/canary-junit.xml happened to still be on disk — the next run overwrites it, so a Monday red would be untriageable by Wednesday. On RED the canary now copies the JUnit to a stamped filename. It stays LOCAL and out of the issue: raw tracebacks are exactly what the sanitization contract keeps off a public repo. tmp/ is gitignored. Also adds scripts/diag/diag_aisandbox_bearer.py, which issues a read-only aisandbox call and the uploadImage POST with the same token in one session so "is uploadImage broken" and "is the whole Bearer path broken" can be told apart. It takes --transport so the two transports can be A/B'd against one profile. #561 did not reproduce: uploadImage returns 200 on both the default and evaluate_fetch transports, and a full e2e_auth re-run is green. No fix is proposed for a cause that has not been identified — the retry that already exists did not help, and adding another would paper over it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI's generate_website_docs.py --check gate caught the drift: docs/E2E_TESTING.md gained the nightly-canary section without its website/docs mirror being regenerated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… leak Council review of #560 (8 dimensions) found defects that unit tests, CI, SonarCloud and a live run had all missed. **A real regression could publish as DEFERRED.** `_PRECONDITION_MARKERS` substring-matched the whole pytest transcript, and two e2e_auth tests mention `ProfileLockedError` in ordinary source — one in an assertion message, one in a COMMENT. pytest echoes the failing function's source into its traceback, so a genuine failure in `test_incident_quality_e2e` or `test_cancellation_e2e` was labelled "nothing ran". Drift published under the one word that means ignore me. Now anchored on the raised-exception line. **An all-skipped run published GREEN.** pytest exits 0 when every test skips, and every e2e test skips on a missing profile dir — so a moved profile would have pinned the dashboard green forever. GREEN now requires passed > 0. **Parametrize ids reached the public issue verbatim.** Reproduced: a param id carrying a filesystem path flowed into the rendered body. Tracebacks never leaked (the hard boundary held), but test names were an unguarded channel. The `[...]` suffix is dropped. Also: credit-spending markers are refused outright rather than merely defaulted away from; a refused `--pull` and a hung tier now publish DEFERRED/RED instead of exiting silently (silence is indistinguishable from a machine that was off); `--pull` re-syncs deps so a lockfile bump cannot masquerade as a RED import error; the PowerShell task quotes its argv (the documented `-Markers 'e2e_auth or e2e_scene'` example would have split into three tokens); and `--simulate` is gone — publishing a fake verdict to a trust signal is an anti-feature, and the render tests already cover all four states. 25 classifier tests, each new one pinning a specific finding. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CLAUDE.md makes docs/INDEX.md the routing layer for every doc, and the canary section had no entry — so "has Flow drifted since the last release?" had no path to an answer. Council review, D9 nice-to-have. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ffroliva
marked this pull request as ready for review
August 21, 2026 06:14
ffroliva
added a commit
that referenced
this pull request
Sep 13, 2026
…#801) The canary had been RED with an identical failure set since 2026-09-11, and every run posted the same comment ending "Canary gates nothing. Triage at your convenience." A standing condition -- a capability Flow took away from this account, already understood -- looked exactly like fresh drift, which is the thing the canary exists to catch. The set being UNCHANGED was the informative part, and nothing reported it. RED is not demoted. The module docstring is emphatic that labelling a real regression "ignore me" is how the signal dies, and #560's council review already caught one such demotion. This only adds what a single run cannot say: continuity(previous, failing) -> (state, note) a pure function beside `classify`, fingerprinting the sorted failing set (order-insensitive -- pytest promises no stable order and a reshuffle is not new). Same set as last time -> "Unchanged for N consecutive runs", and say that a red nobody owns is how a canary trains red-blindness. Different set -> "The failing set changed since the last run", the arm worth looking at today. The note REPLACES the flat triage line rather than sitting beside it: a report saying both "at your convenience" and "broken for four nights" is arguing with itself, and the reader believes the softer half. Only RED and GREEN update the streak. AUTH-EXPIRED and DEFERRED learned nothing about Flow, so one night with Chrome open must not erase a week-old red. State lives in tmp/canary-state.json (gitignored, disposable). Every read and write degrades to "no continuity known" rather than taking the nightly down -- a test caught int() raising on a hand-edited value, which would have done exactly that. Verified by running the real script twice, dry-run, $0: run 1 -> RED, first-ever fingerprint, no streak claimed (correct) run 2 -> RED, {"fingerprint": "7ac1eca0bf29", "runs": 2}, "Unchanged for 2 consecutive runs." Those runs also produced the finding the feature exists to surface: today's set is NOT the nightly's. The four credits/aisandbox failures are identical -- the standing condition (#795, this account losing the ya29 Bearer around 2026-09-11) -- while the two agent-mode transport tests now pass and a different test has joined. A stable core with flapping edges, which is exactly what the report could not say. Gates: ruff, ruff format, pyright strict, 4 doc/hygiene checks, 4261 passed, 92%. Refs #559
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the council-adjudicated canary from #502: a local nightly run of the
$0 live tiers, publishing a sanitized verdict to a rolling GitHub issue (#559).
Hosted CI can't cover the live tiers — they need a real authenticated Chrome
profile, and Google bot-detection plus reCAPTCHA make hosted auth infeasible. So
the run happens where the warm profile already lives, and reports to GitHub.
Design decisions
Rot vs. drift is decided by a second probe, not by error names. When a tier
fails, the canary re-runs
gflow auth status. Session still valid ⇒ drift (RED);session now dead ⇒ genuine rot (
AUTH-EXPIRED). Costs ~45s, and only after a failure.DEFERREDcovers profile-state preconditions — a heldProfileLeaseor aProfileEngineDowngradeError. Both fail closed before any browser starts, so the runnever reached Flow and cannot evidence drift. They share
ConfigurationError's exit11, so the class name is the only discriminator.
Keeping
AUTH-EXPIREDandDEFERREDout ofREDis the whole point.REDmustalways mean "code or Flow drifted", never "please re-login" or "you had Chrome open".
Gates nothing, never opens issues. A gate on a machine that might be off is
self-DoS; an issue per night trains red-blindness. Publishing uses the maintainer's
already-authenticated
gh— no new secrets.--pullrefuses a dirty checkout rather than resetting over it. A nightly taskthat can destroy uncommitted work is worse than a stale canary, so the scheduled task
must point at a dedicated clone.
What the first live run found
Ran for real against
denon82before this was committed: 17 passed, 1 failed, 1skipped, 150s →
RED. It caught two classifier bugs in the canary itself:AuthExpiredError/AisandboxAuthErrorwere missing from the hardcoded authmarker list — the verdict was right only by accident.
verified clean immediately after the failure, so the aisandbox
uploadImage401is a divergence between two auth surfaces, not rot. That motivated replacing name
matching with the post-run probe.
A third gap surfaced running against the other profile:
ProfileEngineDowngradeErrorwould have reported
RED. Hence theDEFERREDwidening.The underlying
uploadImagefailure is not fixed here — it's on the experimentalevaluate_fetchtransport, and needs its own issue once scoped.Test plan
--simulate× 4)ghpublish path verified end to end against [GREEN] gflow nightly canary — 2026-11-14 02:00 UTC #559--pulldirty-tree refusal verifiedcheck_doc_links.py,check_repo_hygiene.pyRefs #502