Skip to content

fix(image): re-introduce i2i ref-attach (#50) with locale-agnostic media-dialog selectors (#56) - #60

Merged
ffroliva merged 2 commits into
ffroliva:developfrom
svasakorn:feat/image-i2i-ref-attach-v2
May 24, 2026
Merged

ffroliva merged 2 commits into
ffroliva:developfrom
svasakorn:feat/image-i2i-ref-attach-v2

Conversation

@svasakorn

Copy link
Copy Markdown
Contributor

Summary

Re-introduces the i2i local-file ref-attach from #50 (reverted in #57) and fixes the #56 root cause by making the media-dialog upload selectors locale-agnostic. Branched off develop @ 5345c1b as you suggested.

Root cause (#56)

It turned out to be a locale leak, not an intermediate-menu / nesting difference. Your ffroliva dump (#issuecomment-4528075496) shows the same popover with the same "Upload media" button at the same level as mine — just rendered in pt-BR (Enviar mídia). The old UPLOAD_MEDIA_BUTTON = button:has-text('Upload media') didn't match the pt-BR label, so the click missed → expect_file_chooser never fired → the ~34s silent timeout.

Flow renders the dialog in the Chrome profile language, which --lang=en-US cannot override on an already-configured profile (matches your "server-side locale overrides the launch arg" observation; same class as #51 / #24).

Changes

src/gflow_cli/api/transports/ui_automation_video.py:

  • UPLOAD_MEDIA_BUTTON → the locale-free upload icon ligature, :text-is('upload') (exact — so it doesn't also grab the Uploads tab whose icon is drive_folder_upload), scoped to [role='dialog'][data-state='open']. This is the selector you recommended, with the exact-match refinement.
  • Fallback tier — the original English-text selector (has-text('Upload media')), tried only if the icon misses. Covers the narrow case where Google changes the icon ligature but keeps the English label. Short per-tier timeouts so two misses can't recreate a long silent hang.
  • 'Add to Prompt' has no icon (confirmed via DOM), so it's selected structurally as the lone iconless button in the open dialog (.filter(has_not=<i.google-symbols>)) instead of by localized text.
  • Fail-loud guard — if neither tier opens a chooser, screenshot + a clear RuntimeError. The message explicitly tells the operator to set the Chrome profile language to English (chrome://settings/languages) and that the Google account language is not enough — that distinction is genuinely easy to get wrong.

tests/e2e/test_transports_e2e.py:

  • Adds the C2-style i2i e2e variant (test_e2e_i2i_local_ref_attach), GFLOW_CLI_E2E_PROFILE-gated, ui-automation transport only (the REST transports can't drive the add-media dialog). Generates a tiny PNG inline — no asset dependency.

The shared _upload_via_open_dialog is also used by I2V/R2V, so this hardens those upload paths too.

Verification

Real CLI, English-rendering profile, this branch:

11:22:55.614  image_mode_entered
11:23:00.372  count_setter_completed
11:23:13.986  image_uploaded  status=200   <- upload fired (~13.6s, not the 34s hang)
11:23:15.670  reference_attached
11:23:16.318  prompt_submitted
11:23:50.935  batch_response_captured status=200  -> image returned

Honest caveat: I can only verify on an English-rendering profile, where both the icon and text selectors would match — so this proves the end-to-end path works, but not the pt-BR path specifically. The icon selector is locale-independent by construction and your ffroliva dump confirms the same upload ligature there. Could you run the gated e2e (or your capture_image_add_media_dom.py from #59) on ffroliva to confirm before closing #56? I deliberately wrote "Refs", not "Closes".

Not fixed here (focused PR — flagging per your preference)

FRAME_SLOT_BY_LABEL (I2V Start/End slots, div[aria-haspopup='dialog']:has-text('{label}')) is still text-based and will hit the same locale issue on a non-English profile. It's the I2V frame-slot path, separate from this upload fix, and belongs with #24 — happy to follow up separately rather than widen this PR.

Test plan

  • check_repo_hygiene.py — clean (255 files)
  • ruff check src tests — clean
  • ruff format --check src tests — clean (103 files)
  • pyright src — 0 errors
  • pytest -q -m "not e2e and not live" — 806 passed, 6 skipped (single local interpreter)
  • Real CLI gflow image i2i --ref <png> on an EN profile — image returned (timeline above)
  • CI matrix 3.11 / 3.12 / 3.13 (fork PR → action_required; needs your approve-and-run)
  • e2e i2i variant on a logged-in profile (GFLOW_CLI_E2E_PROFILE=...)

DCO-signed (both commits). Refs #56, #24.

svasakorn added 2 commits May 24, 2026 17:45
Local --ref paths now ride GenerateImageRequest.ref_paths and attach through the editor add-media dialog (reusing the R2V _attach_references path) instead of the REST uploadImage endpoint, which 401s (closes ffroliva#39). Bare-UUID refs still flow through refs unchanged. Adds CLI-layer and transport-layer test coverage. Live-verified end-to-end via the real CLI: image i2i --ref <png> -> uploadImage 200, imageInputs[] populated, image returned.

Signed-off-by: svasakorn <s.vasakorn@gmail.com>
The i2i ref-attach (ffroliva#50) hung on non-English Chrome profiles (ffroliva#56): the
'Upload media' button was matched by localized text, so the file-chooser click
missed and silently timed out (~34s). Flow renders the media dialog in the
Chrome PROFILE language (NOT the Google account language; the --lang=en-US arg
cannot override an already-configured profile).

- UPLOAD_MEDIA_BUTTON: anchor on the locale-free `upload` icon ligature
  (:text-is, exact, so it doesn't grab the 'Uploads' tab), scoped to the open
  dialog. Keep the original English-text selector as a graceful fallback tier
  (matches if Google changes the icon but keeps the English label).
- 'Add to Prompt' (no icon): select structurally as the only iconless button in
  the open dialog via .filter(has_not=<icon>).
- _upload_via_open_dialog: if neither tier opens a chooser within short per-tier
  timeouts, raise a clear error + screenshot (no silent hang) telling the
  operator to set the CHROME PROFILE language to English -- explicitly NOT the
  Google account language. Thread out_dir from both callers.
- Add an opt-in i2i e2e variant (GFLOW_CLI_E2E_PROFILE-gated).

Verified end-to-end via the real CLI on an English-rendering profile: upload
200 -> reference_attached -> image returned, no 34s gap. The icon/structural
selectors are locale-independent by construction; the ffroliva DOM in ffroliva#56
confirms the same `upload` icon under pt-BR.

Refs ffroliva#56, ffroliva#24.

Signed-off-by: svasakorn <s.vasakorn@gmail.com>
@svasakorn
svasakorn requested a review from ffroliva as a code owner May 24, 2026 11:29
@github-actions github-actions Bot added external-contribution Pull request from a non-maintainer or fork needs-copilot-review Ask GitHub Copilot for advisory code review needs-maintainer-review Requires maintainer review before merge labels May 24, 2026
@github-actions

Copy link
Copy Markdown

Thanks for the contribution. This PR has been marked as an external contribution and routed for maintainer review.

Before merge, please make sure:

  • The PR targets develop.
  • Commits use a real Git identity or GitHub noreply email.
  • External commits include a DCO sign-off: git commit -s.
  • The PR does not include secrets, cookies, account tokens, signed URLs, or private captured data.
  • Focused tests/docs are included for behavior changes.

GitHub Copilot code review may be requested as an advisory first pass, but maintainer approval is still required.

@ffroliva

Copy link
Copy Markdown
Owner

Verification on ffroliva (pt-BR) ✅ — production code passes. But the test has a bug that needs a 1-line fix before merge.

Verification ledger (5 layers)

Layer Result Evidence
1. Exit code 0 pytest PASSED after 58s
2. File count N/A Test asserts on the API contract (media_name, fife_url), doesn't download — fine, the structlog covers the upload step
3. Pillow dims N/A (no file downloaded)
4. structlog invariants ✅ all green image_mode_entered → count_setter_completed → image_uploaded status=200 target=ref0 (upload fired — NOT the 34s silent hang from before) → reference_attached index=0 → prompt_submitted (via locale-stable arrow_forward icon) → batch_response_seen/captured status=200
5. Gallery confirm pending repo owner Flow project becce409-a99b-4006-80bb-5ac516a562e1

The locale-agnostic upload icon ligature selector works on a pt-BR account. Mission accomplished on the production code — your fix direction was correct, and the exact-match :text-is('upload') refinement to avoid the Uploads tab confusion was a nice touch.

Test transport bug — please amend before merge

# tests/e2e/test_transports_e2e.py:169
async with _make_client("evaluate_fetch", profile) as client:  # ← REST
    image = await client.generate_image(req=req)

…but the docstring three lines up explicitly says:

UI-automation transport ONLY — the REST transports (bearer/sapisidhash) cannot drive the add-media dialog, so they never invoke _attach_references.

So the test as committed routes through evaluate_fetch (REST), which silently drops ref_paths (REST has no media-dialog code path). I had to manually patch this to "ui_automation" on a local checkout to actually exercise your fix — that's the run whose evidence is above.

This means your own EN-profile verification was a false positive (sorry to be the bearer): your test sent a REST request with the prompt, Flow generated a text-only image, the media_name + https:// assertion passed → green. The ref was never attached, the locale-agnostic selectors were never tested. On ffroliva we got a different false signal — AuthExpiredError: evaluate_fetch HTTP 401 because the REST session token is stale — but again, never reached the actual fix code.

Only after patching to "ui_automation" did the test do what its docstring promises.

One-line amend

- async with _make_client("evaluate_fetch", profile) as client:
+ async with _make_client("ui_automation", profile) as client:

(Could optionally inline a _make_client call with transport=None to use the registered default — same effect since _DEFAULT_TRANSPORT = "ui_automation" — but the explicit name is clearer.)

Optional follow-up (separate PR / your call)

The assertion image.media_name and image.fife_url.startswith("https://") would pass even if ref_paths were silently ignored (i.e. it doesn't detect the very class of bug we just lived through). A stronger assertion would be useful — e.g. inspect the captured wire request payload for an imageInputs[] entry, or assert the generated image dimensions match Aspect.PORTRAIT (text-only Narwhal vs ref-conditioned Narwhal usually differ in framing). Not blocking — just flagging.

State of PRs from this session

Thanks again for the speed on this — the focused scope, the :text-is('upload') refinement, the iconless structural selector for "Add to Prompt", the fail-loud RuntimeError with the Chrome profile vs Google account language hint, and the honest scoping of FRAME_SLOT_BY_LABEL to #24 are all great. Ready to merge as soon as the test transport is fixed.

@ffroliva
ffroliva merged commit 822866d into ffroliva:develop May 24, 2026
6 checks passed
@svasakorn

Copy link
Copy Markdown
Contributor Author

Thanks for merging #60! Heads-up: my test-transport fix (evaluate_fetch → ui_automation) landed a moment after the squash, so it missed the merge — the e2e on develop still routes through REST (the false positive you flagged). Small follow-up in #61. It also adds the reference_attached assertion so the test can't silently pass on a text-only image again.

ffroliva pushed a commit that referenced this pull request May 24, 2026
The i2i e2e (test_e2e_i2i_local_ref_attach, added in #60) routed through
`_make_client("evaluate_fetch", ...)` — the REST transport, which silently
drops `ref_paths` and cannot drive the add-media dialog. It returned a
text-only image and the `media_name`/`https://` asserts passed, so it was a
false positive: the locale-agnostic media-dialog selectors from #60 were
never exercised. (Caught in #60 review; the transport fix landed just after
the squash-merge, so it missed the merge.)

- transport: evaluate_fetch -> ui_automation (so the test drives the media
  dialog the fix actually changed).
- assert a `ui_automation_video.reference_attached` event fires (via the
  existing `install_log_capture` fixture) so a silently-dropped ref fails the
  test loudly instead of passing on a text-only image.

Test-only; no production change (the selector fix is already on develop).
Verified on an EN profile: gated e2e `test_e2e_i2i_local_ref_attach` 1 passed
(72s) + real CLI `gflow image i2i --ref <png>` -> image_uploaded 200 ->
reference_attached -> image returned (768x1376).

Refs #60, #56.

Signed-off-by: svasakorn <s.vasakorn@gmail.com>
15990166686ll-crypto pushed a commit to 15990166686ll-crypto/gflow-cli that referenced this pull request Aug 24, 2026
…ostic media-dialog selectors (ffroliva#56) (ffroliva#60)

* feat(image): bind i2i local refs via editor media dialog

Local --ref paths now ride GenerateImageRequest.ref_paths and attach through the editor add-media dialog (reusing the R2V _attach_references path) instead of the REST uploadImage endpoint, which 401s (closes ffroliva#39). Bare-UUID refs still flow through refs unchanged. Adds CLI-layer and transport-layer test coverage. Live-verified end-to-end via the real CLI: image i2i --ref <png> -> uploadImage 200, imageInputs[] populated, image returned.

Signed-off-by: svasakorn <s.vasakorn@gmail.com>

* fix(image): locale-agnostic i2i media-dialog selectors + upload fallback

The i2i ref-attach (ffroliva#50) hung on non-English Chrome profiles (ffroliva#56): the
'Upload media' button was matched by localized text, so the file-chooser click
missed and silently timed out (~34s). Flow renders the media dialog in the
Chrome PROFILE language (NOT the Google account language; the --lang=en-US arg
cannot override an already-configured profile).

- UPLOAD_MEDIA_BUTTON: anchor on the locale-free `upload` icon ligature
  (:text-is, exact, so it doesn't grab the 'Uploads' tab), scoped to the open
  dialog. Keep the original English-text selector as a graceful fallback tier
  (matches if Google changes the icon but keeps the English label).
- 'Add to Prompt' (no icon): select structurally as the only iconless button in
  the open dialog via .filter(has_not=<icon>).
- _upload_via_open_dialog: if neither tier opens a chooser within short per-tier
  timeouts, raise a clear error + screenshot (no silent hang) telling the
  operator to set the CHROME PROFILE language to English -- explicitly NOT the
  Google account language. Thread out_dir from both callers.
- Add an opt-in i2i e2e variant (GFLOW_CLI_E2E_PROFILE-gated).

Verified end-to-end via the real CLI on an English-rendering profile: upload
200 -> reference_attached -> image returned, no 34s gap. The icon/structural
selectors are locale-independent by construction; the ffroliva DOM in ffroliva#56
confirms the same `upload` icon under pt-BR.

Refs ffroliva#56, ffroliva#24.

Signed-off-by: svasakorn <s.vasakorn@gmail.com>

---------

Signed-off-by: svasakorn <s.vasakorn@gmail.com>
15990166686ll-crypto pushed a commit to 15990166686ll-crypto/gflow-cli that referenced this pull request Aug 24, 2026
…liva#61)

The i2i e2e (test_e2e_i2i_local_ref_attach, added in ffroliva#60) routed through
`_make_client("evaluate_fetch", ...)` — the REST transport, which silently
drops `ref_paths` and cannot drive the add-media dialog. It returned a
text-only image and the `media_name`/`https://` asserts passed, so it was a
false positive: the locale-agnostic media-dialog selectors from ffroliva#60 were
never exercised. (Caught in ffroliva#60 review; the transport fix landed just after
the squash-merge, so it missed the merge.)

- transport: evaluate_fetch -> ui_automation (so the test drives the media
  dialog the fix actually changed).
- assert a `ui_automation_video.reference_attached` event fires (via the
  existing `install_log_capture` fixture) so a silently-dropped ref fails the
  test loudly instead of passing on a text-only image.

Test-only; no production change (the selector fix is already on develop).
Verified on an EN profile: gated e2e `test_e2e_i2i_local_ref_attach` 1 passed
(72s) + real CLI `gflow image i2i --ref <png>` -> image_uploaded 200 ->
reference_attached -> image returned (768x1376).

Refs ffroliva#60, ffroliva#56.

Signed-off-by: svasakorn <s.vasakorn@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

external-contribution Pull request from a non-maintainer or fork needs-copilot-review Ask GitHub Copilot for advisory code review needs-maintainer-review Requires maintainer review before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants