Skip to content

Canary: 9 of the 13 standing failures are one /about precondition, not a defect #888

Closed

Description

@ffroliva

The nightly canary (#559) has been RED for three consecutive runs with an unchanging set of 13 failures. All 13 are now reproduced locally and accounted for. 9 of them are one missing precondition, not a defect.

Measurement (2026-09-20, local, $0)

The canary profile denon82 is in the /about state: Flow answers https://flow.google.com/about for every navigation, including the bare root.

project.labs_route_refused   status=404                       ← correct, labs is retired (#868)
ui_driver.known_landing      kind=public
                             url=https://flow.google.com/about
                             requested=https://flow.google.com/
FlowAppError                 exit 31

This is account-scoped, not a product failure. gflow project create on the same build, same minute:

profile result
ffroliva created 692c6190-3889-4066-9aef-5ecd8a60215a
ci-probe created 44ca4db5-079f-4b3c-b378-2c474d5b6591
denon82 exit 31, /about

Consistent with the 2026-09-11 spike, which measured the redirect stable per account (5/5 over ~3 minutes on a healthy session).

The 13, decomposed

9 — /about precondition on denon82 (reproduced; /about appears in every failure log)

  • test_project_create_e2e ×3 — file added by v0.78.0 and never green in the canary since
  • test_project_naming_e2e::test_project_creation_and_rename_live_browser_e2e
  • test_transports_e2e::test_e2e_unrecoverable_auth_expiry ×3 (parametrised)
  • test_transports_e2e::test_e2e_agent_mode_recovered_before_mode_switch
  • test_transports_e2e::test_e2e_agent_chat_panel_recovered_before_mode_switch

4 — already tracked by #795 (credits labs-gated on migrated accounts)

  • test_aisandbox_auth_live::test_rest_upload_image_authenticates_after_sapisidhash
  • test_credits_e2e::test_credits_http_fast_path_live
  • test_credits_mcp_e2e ×2

Proposed fix — a sibling of skip_on_migrated_host

tests/e2e/conftest.py::skip_on_migrated_host exists for exactly this shape, and its docstring names this very issue:

That is a missing precondition, not selector drift. Left unguarded they report RED nightly: in issue #559 (2026-09-06) three of five canary failures were exactly this, and a canary that cries wolf stops being read.

Same trap, one state over. Add skip_on_about_landing, catching FlowAppError where the landing kind is public, and apply it to the 9. The canary then reports what it is: a precondition this account no longer satisfies.

This is a skip, not a lowered bar. The tests still fail on any profile that can reach Flow — ffroliva and ci-probe both create projects fine, so the coverage is real wherever the precondition holds.

Related, not blocked on

Done when

  • skip_on_about_landing in tests/e2e/conftest.py, with the measurement above in its docstring
  • Applied to the 9 tests
  • A canary run that is either GREEN or red for a reason nobody has seen before

Activity

  1. ffroliva commented on Sep 20, 2026

    @ffroliva
    OwnerAuthor

    Correction: 7, not 9 — and the other 2 are a different condition

    I wrote this issue from the failure logs (all 13 mentioned /about). Implementing the guard and running it proved that wrong for two of them, so the split above needs amending.

    With the guard in place on denon82:

    tests/e2e/test_project_create_e2e.py     3 skipped   ← /about
    tests/e2e/test_project_naming_e2e.py     1 skipped   ← /about
    tests/e2e/test_transports_e2e.py         3 skipped   ← /about (unrecoverable_auth_expiry ×3)
    tests/e2e/test_transports_e2e.py         2 FAILED    ← NOT /about
                                             2 passed
    

    The two survivors are test_e2e_agent_mode_recovered_before_mode_switch and
    test_e2e_agent_chat_panel_recovered_before_mode_switch, and they fail on a different landing kind entirely:

    AuthExpiredError: Flow served one of its OAuth/sign-in routes
    (https://accounts.google.com/v3/signin/confirmidentifier) instead of the Flow
    gallery — this session is not signed in to Flow on that host.
    

    signin, not public. /about appeared in their logs only because an earlier step in the same test hit it.

    Revised split of the 13:

    count cause status
    7 /about landing — missing precondition guard (this issue)
    2 sign-in redirect on flow.google.com — needs its own determination see below
    4 credits labs-gated on migrated accounts #795

    I am deliberately not widening the guard to cover the 2. AuthExpiredError is exactly what an e2e should fail on; swallowing it would hide a genuine auth signal, which is the opposite of what this issue is for. Whether it is a real defect (a session degrading only on flow.google.com — possibly the #791 family) or a precondition (this profile needs a human re-login) is not yet measured, and gflow auth list reporting session: present for this profile is not evidence either way, since that check does not drive the gallery.

    So this issue now covers 7. The 2 stay red until someone measures them, which is the correct outcome — a canary that goes green by relabelling an unexplained failure is worth less than one that stays red for two.

  2. ffroliva commented on Sep 23, 2026

    @ffroliva
    OwnerAuthor

    The denon82 /about failures have a measured cause, cleared by hand on 2026-09-23

    The 9 canary failures that end at flow.google.com/about on denon82 were Google asking the account to re-verify its identity ("Confirm it's you — sign in again to continue"), while every cookie gflow checks was present and unexpired.

    • Before: /about on every visit. The page's "Create with Google Flow" CTA goes to accounts.google.com/v3/signin/confirmidentifier.
    • gflow auth login did not clear it. It reported verified in 0.4 s on the valid cookies and closed Chrome. Filed as auth login declares success on valid cookies and cannot clear Google's pending identity re-check (/about) #902.
    • Completing "Confirm it's you" by hand in real Chrome on the profile cleared it. Control visits now land on the app, and gflow project create --profile denon82 succeeds (it exited 31 on 2026-09-20).

    Tonight's canary is the check: the denon82 /about failures should be gone. If any come back, the account has been asked to re-verify again, which is worth knowing too.

    Evidence (PR #903): docs/superpowers/spikes/2026-09-23-about-cta-leads-to-google-reauth.md · scripts/dev/spike_about_cta_recovery.py

  3. added 2 commits that reference this issue on Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions