project.createProject returns 401 on the browser context while the on-disk Flow session is valid

Symptom

uploadImage via the experimental evaluate_fetch transport returns
Unauthorized while the Flow session is verifiably healthy.

tests/e2e/test_aisandbox_auth_live.py::test_rest_upload_image_authenticates_after_sapisidhash
gflow_cli.errors.AuthExpiredError: Authentication expired: Unauthorized

Found by the first live run of the nightly canary (#502 / #560) on 2026-08-20.

Why this is not session rot

gflow auth status was run immediately after the failure, on the same profile:

Flow session verified as denon82@gmail.com.     exit 0

So the labs.google NextAuth session is valid, but the aisandbox Bearer path
rejects the request. Two auth surfaces disagreeing — which is what makes this
worth a ticket rather than a re-login.

Repro

GFLOW_CLI_E2E_PROFILE=denon82 \
  python -m pytest tests/e2e/test_aisandbox_auth_live.py -m e2e_auth -q -p no:cov

$0 — uploading an image asset spends no generation credit.

Scope / what is NOT yet established

Deliberately stated as unknowns rather than guesses:

One observation only. The second profile (default) could not be tested:
it fails earlier with ProfileEngineDowngradeError (profile written by
Chromium 151.0.7922.137, bundled engine 149.0.7827.55).
Possibly long-standing, not fresh drift. The test last changed in e37cd83
(2026-05-31, "L0 Bearer pivot live-verified"), and it is e2e_auth, so it has
been runnable — and unrun — for ~3 months.
Not confirmed transport-specific. The test pins transport="evaluate_fetch"
(experimental). Whether the default transport uploads fine is untested.
Test naming is stale. ..._after_sapisidhash predates the Bearer pivot;
memory records SAPISIDHASH as disproven for aisandbox. The name may be
misleading about what is actually being asserted.

Next steps

Determine whether uploadImage succeeds on the default transport — that
isolates experimental-transport bug vs. shared Bearer-path bug.
Capture the actual request headers (GFLOW_CLI_LOG_REQUEST_HEADERS=1) and diff
against a working browser uploadImage, per the test's own docstring.
Decide whether the test's SAPISIDHASH-era naming and assertion message should be
corrected regardless of the outcome.

Impact

Low-to-unknown. evaluate_fetch is experimental and not the default path. Raising
it because it is a genuine auth-surface divergence with no open ticket, surfaced by
the canary on its first night — exactly the class of finding the canary exists for.

Refs #502, #560

Investigation update — does not reproduce on the default transport

Ran the same upload against the same profile (denon82) via scripts/diag/diag_aisandbox_bearer.py:

transport: (default)
[OK] create_project (labs.google tRPC, cookie auth): 873dda07-…
[OK] access_token from /fx/api/auth/session: len=429 prefix=ya29.… ya29=True
[OK] aisandbox POST flow/uploadImage: HTTP 200  → media 5706cbff-…

HTTP 200. Bearer auth, token fetch, and uploadImage all work.

Correcting this issue's "Scope" section

The original text said transport-specificity was unconfirmed, and I then argued it was not transport-specific because upload_image routes through the shared _post_json. That reasoning was wrong.

The upload code path is shared, but the session setup is not. The default transport logs ui_automation.setup_shared_page — it navigates the Flow app before any REST call. evaluate_fetch sets up its context differently. Since the failing test pins transport="evaluate_fetch" and the passing diagnostic used the default, transport is the one variable that actually differed.

Failure boundary (confirmed)

Raise site is api/client.py:2883 — a generic AuthExpiredError on HTTP 401 from aisandbox, with Unauthorized as the provider message. That rules out the token fetch: a failure there raises AisandboxAuthError ("aisandbox-pa authentication failed"), a different title. So the token was obtained, the Bearer header was attached, _run_with_aisandbox_retry re-fetched and retried once, and the server still refused.

Revised hypothesis

evaluate_fetch's context setup leaves the session in a state aisandbox rejects for Bearer POSTs — most likely because it never navigates to the app origin the way ui_automation.setup_shared_page does.

An A/B on --transport evaluate_fetch is running now to confirm. Not proposing a fix until it lands.

Also worth noting

No test in e2e_auth makes a successful aisandbox Bearer call — test_e2e_unrecoverable_auth_expiry is a negative test. So a total Bearer outage would currently pass the tier unnoticed. e2e_scene would cover it, but it skips silently without GFLOW_CLI_E2E_SCENE_WORKFLOW_ID, so adding it to the canary (#502 fast-follow) needs that env var wired or it is a false green.

feat(canary): preserve failing JUnit on RED; add aisandbox Bearer dia…

Does not reproduce — four hypotheses falsified

Hypothesis	Test	Verdict
`uploadImage` endpoint broken	diagnostic, default transport	dead — HTTP 200, media created
Whole aisandbox Bearer path down	same run	dead — `ya29.` token fetched, upload accepted
`evaluate_fetch` session setup	diagnostic, `--transport evaluate_fetch`	dead — HTTP 200
Intra-tier state pollution	full `-m e2e_auth` re-run	dead — 18 passed, 1 skipped

Same profile (denon82), ~90 minutes after the canary's RED. Classifying as a transient.

No fix proposed, deliberately

_run_with_aisandbox_retry already re-fetches the token and retries once on 401, and it did not help. Adding more retry would paper over a cause that has not been identified. Editing the auth headers against a failure I cannot reproduce would be guessing.

What did change

The canary now preserves the failing JUnit on RED (bbdc46c). This investigation was only possible because tmp/canary-junit.xml happened to survive — the next run overwrites it, so a Monday red would have been untriageable by Wednesday. The copy stays local; raw tracebacks are what the sanitization contract keeps off a public repo.

Added scripts/diag/diag_aisandbox_bearer.py for the next occurrence: read-only aisandbox call + uploadImage POST with the same token in one session, --transport to A/B transports.

Correction

My earlier comment blamed evaluate_fetch's session setup. That is now falsified. In fairness the reasoning was also methodologically wrong: I compared a single isolated call against a 19-test, 150-second tier run and attributed the difference to transport, when the environment differed too.

Recommendation

Keep open but do not investigate further without a recurrence. The canary is the watcher, and it will now carry evidence. Reopen-trigger: a second RED naming this test. If it has not recurred in a few weeks of nightlies, close as transient.

One coverage gap stands regardless (see previous comment): no test in e2e_auth makes a successful aisandbox Bearer call, so a real Bearer outage would pass the tier silently.

Correction — my earlier coverage claim was wrong

Two comments above I wrote that e2e_auth contains no successful aisandbox Bearer call, and that a total Bearer outage would therefore pass the tier silently. That is false. Caught by a council reviewer on PR #560 and verified:

$ git show 7c8c07a:tests/e2e/test_aisandbox_auth_live.py | grep -n 'pytestmark|assert asset'
30:pytestmark = [pytest.mark.e2e, pytest.mark.e2e_auth]
72:        assert asset.name, "uploadImage returned an asset id => SAPISIDHASH accepted"

This very test is in the e2e_auth tier and asserts a successful upload through the Bearer path. A total outage turns the tier RED — which is exactly what happened on the canary's first run.

My mistake: I read the list of passing tests, saw no Bearer call among them, and concluded the tier had none — forgetting that the one test that failed was the Bearer call.

The accurate version

Coverage is single-point, not absent: e2e_auth makes exactly one aisandbox Bearer call. That is enough to detect an outage, but not enough to localise it — one failing call cannot separate "uploadImage rejected" from "whole Bearer path down". That limitation is real and is what motivated scripts/diag/diag_aisandbox_bearer.py.

Consequence for the #502 fast-follow

My earlier recommendation was over-stated. Adding e2e_scene is still worthwhile for a second, independent Bearer datapoint, but it is not closing a hole that leaves outages undetected. And the GFLOW_CLI_E2E_SCENE_WORKFLOW_ID caveat stands on its own: without that env var the tier silently SKIPs, and an all-skipped pytest run exits 0 — which the canary would report GREEN with passed=0. That false-green is a real bug and is being addressed on #560.

feat(ci): nightly live-e2e canary with a four-state verdict (#502) (#560

Correction — this is not an aisandbox/Bearer failure

The reopen-trigger this issue set ("a second RED naming this test") has fired twice: 2026-08-23 and 2026-08-25, on top of the original 08-20. Three of the last four canary runs.

Triaging those with the preserved JUnits shows this issue has been pointed at the wrong host from the start.

The evidence

All three REDs carry a byte-identical traceback:

tests/e2e/test_aisandbox_auth_live.py:69
    project = await client.create_project(title="L0 Bearer auth smoke")
src/gflow_cli/api/client.py:1395: in create_project
resp = <APIResponse url='https://labs.google/fx/api/trpc/project.createProject'
        status=401 status_text='Unauthorized'>
body = {"error":{"json":{"message":"Unauthorized","code":-32001,
        "data":{"code":"UNAUTHORIZED","httpStatus":401,
        "path":"project.createProject"},"status":401}}}

The failing line is :69 — create_project — not :70, upload_image.
The route is labs.google/fx/api/trpc: the NextAuth cookie surface.
No Bearer token is involved. _aisandbox_auth_headers() is gated on is_aisandbox (client.py:1256) and is never called for this route.

The test is named for an upload, which is where the misreading came from. It never reaches the upload.

What that invalidates

The evaluate_fetch vs default-transport A/B falsified a hypothesis about a code path the failure never reaches. create_project runs before transport-specific setup matters.
scripts/diag/diag_aisandbox_bearer.py exercises the wrong host for this failure.
This comment above is also wrong: "_run_with_aisandbox_retry already re-fetches the token and retries once on 401, and it did not help." On this failure it never ran — it is gated on is_aisandbox (client.py:1211). A labs.google tRPC 401 gets no refresh and no retry: _run_with_retry retries 429 and 5xx only (client.py:1357-1382), so 401 falls straight through to _raise_for_non_retryable. That asymmetry is real, but see below on why it is not the thing to fix yet.

Hypothesis falsified: probe-induced cookie rotation

Worth recording so nobody re-runs it. GET /fx/api/auth/session returns a rotated __Secure-next-auth.session-token on every call, and verify_flow_profile discards it (throwaway httpx.AsyncClient, never written back — on-disk fingerprint unchanged). Since the canary runs probe_auth immediately before pytest, it looked like the canary might be invalidating the very session the tests were about to use.

Measured directly against the live profile:

arm	result
control: `createProject` with on-disk token, ×2	200, 200
after probe 1, same old token	200
the rotated token	200
after probe 2, old token again	200

Dead. Rotation does not invalidate the superseded token. The canary is not causing its own red.

Where this actually points

That experiment made createProject calls with the cookie read straight off disk — all 200. And in the 02:00 run, test_e2e_verify_flow_profile_authenticated passed, which reads and decrypts the same on-disk cookie and confirms the session is live.

So at the moment of failure the on-disk cookie was good, and a disk-cookie HTTP call to the same route works. The failing call goes through the Playwright browser context instead. Two carriers for one session, disagreeing.

That is the signature of #222 (macOS): the headed context loaded zero cookies because it could not decrypt the profile store, and project.createProject returned 401 while login and verification both passed — because those read cookies a different way.

Not reproducible on demand

Verified today against denon82, $0:

check	result
`gflow auth status --profile denon82`	exit 0
the failing test, standalone	1 passed, 13.97s
full `-m e2e_auth` tier	18 passed, 1 skipped

Warm, the context loads 59 cookies with the session present. It has only ever failed unattended.

Next step, and no fix proposed

The deciding line is already logged on every run — client.context_cookie_state (the #222 diagnostic), reporting whether the launched context actually loaded the session cookie. pytest captures it and the canary discards it, keeping only the traceback.

#572 adds -o junit_logging=all so the preserved JUnit keeps it. present=False ⇒ the browser never got the cookie, and #222 shows the shape of the fix; present=True ⇒ genuinely refused, and the count/SAPISID/expiry come with it.

Still no fix, deliberately — the standing rule on this issue holds: not reproducible, so not fixable without guessing. The change only makes the next occurrence readable.

Suggested housekeeping

Retitle this issue to name the real surface, e.g. "project.createProject returns 401 on the browser context while the on-disk session is valid". The current title routes every investigator to the wrong host — it routed me there for the first hour.

Refs #559, #572

chore(canary): keep the captured logs on a RED, and correct the 401's…

Live evidence: this exact call returns 200 today — and the origin/referer hypothesis is falsified

First, a correction to the record

The title says project.createProject and the body describes uploadImage. The title is right. The preserved canary JUnit shows the test fails in its setup step, not its subject:

project = await client.create_project(title="L0 Bearer auth smoke")
  → client.py:1395 create_project
  → _post_json(routes.CREATE_PROJECT, ...)
resp  = url='https://labs.google/fx/api/trpc/project.createProject' status=401
body  = {"error":{"json":{"message":"Unauthorized","code":-32001,
          "data":{"code":"UNAUTHORIZED","httpStatus":401,"path":"project.createProject"}}}}

So this is the labs.google tRPC lane, not aisandbox. The test name misleads — I inferred "aisandbox" from the filename earlier today and repeated it in PR #578's commit message ("it fails on the aisandbox lane, not this one"). That statement is wrong and I could not rewrite it once merged; correcting it here instead.

The hypothesis that is now dead

03a1e52 (parked on bugfix/agentic-error-contract-401) claimed this 401 is caused by the tRPC lane sending no origin/referer, which Google rejects as CSRF-shaped. That was the leading theory and PR #578 was cherry-picked to fix it.

Tested live, 2026-08-25 ~21:00 local, profile denon82 — interleaved A/B against this exact route, one session, alternating so ordering/rate/session-decay cannot masquerade as the effect:

session probe            -> HTTP 200 alive=True (flow_session_cookie_present=True)
A_control_no_origin      -> HTTP 200, 200      (content-type only — what develop sent)
B_with_origin_referer    -> HTTP 200, 200

Two controls on the experiment:

Session liveness probed first — a dead session 401s both arms and reads as a false null.
Instrumentation verified — my first wire-capture reported origin=None for both arms, including the one that explicitly passed one. That was broken instrumentation, not evidence: page.request is an APIRequestContext and emits no page-level request events. Re-checked against a header-echo endpoint: control → origin present? False, treatment → True. Playwright adds no Origin of its own, so the control arm was genuinely bare.

Harness: scripts/dev/spike_trpc_origin_referer_401.py (shipped in v0.60.0).

What this sharpens

The same account, the same route, hours apart:

	when	result
canary	02:00 UTC, automated	401
manual spike	~21:00 local, interactive	200 ×4

So it is not a missing header, and not a permanently rotted session — denon82 creates projects fine. Something differs between the 02:00 automated context and an interactive one.

The discriminator arrives tomorrow

#572 added flow_session_cookie_present to the canary's captured logs and merged after the 2026-08-25 02:00 run, so tonight's is the first to carry it. My interactive run logged:

client.context_cookie_state context_cookie_count=61 flow_session_cookie_expired=False
                            flow_session_cookie_present=True google_sapisid_present=True

If tomorrow's RED shows present=False, the cookie never reaches the browser context at 02:00 (profile-lock, unreadable cookie DB, or a cold-start read race) — a completely different bug from "Google rejected it". If it shows present=True, the cookie is there and Google is refusing it, and the next question is what else differs in the automated context.

Either way that single line splits the search space in half, at no cost.

fix(canary): run the version of run_canary.py that the pull just inst…

docs(spec): record the maintainer decision — reject the tRPC pre-flight

fix(transport): stop Flow's changelog modal from wedging runs with an…

docs: live-verification evidence for #626 (v0.64.0)

feat(video): omni-flash i2v now accepts an end frame (#626) (#627)

docs: live-verify #639 against the REAL migrated frontend (zero credits)

fix(transport): name Flow's flow.google.com migration instead of blam…

Reproduced on a fully migrated account (see #639), which may be the variable that makes this come and go.

Account: Indonesia, en UI, Google AI Pro. v0.66.0, Linux, real Chrome profile via Playwright cookie fallback. 2026-09-04, zero credits.

Sequence, all within ~2 minutes on the same profile:

call	route	result
`gflow auth status`	`labs.google/fx/api/auth/session`	"Flow session verified as …"
`gflow project list --json`	list projects	200, 5 projects returned
`gflow project create --json`	`labs.google/fx/api/trpc/project.createProject`	401 Unauthorized
`gflow image t2i "…"` (no `--project`)	same, via "Creating project…"	401 Unauthorized
`gflow auth status` (after)		still verified

So the pattern in the 08-25 correction holds here: the NextAuth session is valid, reads succeed, and only the createProject write is rejected. Your 08-25 note says the call returned 200 on your accounts that day; both those accounts were pre-migration at the time per #639. On this account the 401 showed up the day after the account stopped landing on labs.google at all (3/3 loads migrated).

Hypothesis, not established: the old-host tRPC mutation rejects sessions whose account has completed the move to flow.google.com, while the read routes still serve. That would make this issue a downstream symptom of #639 rather than a separate auth bug, and it means t2i without --project is dead on migrated accounts even before the frontend problem. Happy to run any read-only probe against this account if it helps confirm or falsify.

Here are the fields you asked for, plus a re-measurement on 0.67.0 that I think changes the picture.

Cookie state on the 0.66.0 run that returned 401 (project create --json, Linux, Playwright cookie fallback):

playwright_cookie_read      all_cookies=43  flow_path_root=2  flow_domain=2
client.context_cookie_state context_cookie_count=43  flow_session_cookie_present=true
                            flow_session_cookie_expired=false  google_sapisid_present=true

So the Google session cookie was present and the whole jar reached the context. The #644 harvest path doesn't explain this one.

Same command, same profile, 0.67.0, about a day later:

version	`project create`	cookies
0.66.0 (2026-09-04)	401 on `project.createProject`, twice	43, SAPISID present
0.67.0 (2026-09-05)	200, project id returned, exit 0	47, SAPISID present

Nothing changed on my side between the two beyond uv tool upgrade gflow-cli. No re-login. So on this account the 401 was real on 0.66.0 and gone on 0.67.0. Whether that's a change in 0.67.0's request path or Google flipping something server-side in the meantime, I can't separate from here. If you can point at what 0.67.0 changed in how createProject is issued from the browser context, that would settle it.

Noted on project list reading the local catalog; the live read in my sequence was auth status only.

Bundled into #864 with #863, which hit the same createProject 401 from the MCP path. The 401-then-200 history in this thread is exactly why #864 starts with a repeat measurement before changing behaviour. This issue will close with that fix.

fix(project): create projects on flow.google.com when labs' route ref…

Fixed and released in v0.78.0 (uv tool upgrade gflow-cli, pipx upgrade gflow-cli, or gflow update).

Your reading was right: this was never an auth failure. labs.google/fx/api/trpc/project.createProject is retired — it answers 404 "Flow RPCs have been deprecated and disabled. Flow has migrated to https://flow.google.com." for a session that holds a labs token, and 401 for one that doesn't. Measured 12/12 across four profiles, so the 401→200 flip you saw between 0.66.0 and 0.67.0 sits inside a route that no longer serves anyone here.

gflow now creates the project on flow.google.com when that route refuses, and reads the new id and title back from Flow's own reply. gflow project create, gflow project rename, and generating with no --project (images and video, CLI and MCP) all work again, and the misleading "run gflow auth login" is gone from this path.

Thanks for the careful repro and for re-measuring on 0.67.0 — that contradiction is what stopped us keying the fix on the host instead of on the observed refusal.

project.createProject returns 401 on the browser context while the on-disk Flow session is valid #561

Description