Skip to content

Make UiAutomationTransport selectors language-agnostic (structural-first) #24

New issue
New issue
Closed
Closed
Make UiAutomationTransport selectors language-agnostic (structural-first)#24

Description

@ffroliva
ffroliva
opened on May 19, 2026

Problem

Google Flow renders its UI in the account's language, not the requested
locale. The video-generation design spike confirmed Flow ignores
locale="en-US" + ?hl=en — the test account's Flow renders in Portuguese
(Vídeo, Imagem, Criar, ...). See
docs/superpowers/specs/2026-05-18-ui-automation-video-generation-design.md
§10.4.

Several UiAutomationTransport selectors (the working t2i /
generate_images path) match on localized button text, so they only
work for accounts whose Flow language happens to be in our hard-coded list.

Current state — src/gflow_cli/api/transports/ui_automation.py

Language-agnostic already (good — keep):

  • PROMPT_INPUT_SELECTORS — leads with the data-slate-editor structural selector.
  • SUBMIT_BUTTON_SELECTORS — leads with the arrow_forward icon ligature.
  • GEN_SETTINGS_BUTTON_SELECTORS — all crop_* icon ligatures.
  • Aspect / count tabs — _ASPECT_TAB (numeric ratios 16:9, 9:16, ...) and
    _COUNT_TAB (1x, x2, ...). Numbers/symbols are not translated.

Fragile:

  • ONBOARDING_SELECTORS (~line 158) — purely localized button text
    (Agree / Aceitar / I agree / Concordo / Accept /
    Create with Flow / Criar com o Flow / Get Started / Começar), with
    no structural anchor at all. An account whose Flow renders in an unlisted
    language (German, Japanese, ...) cannot pass the onboarding gate.
  • NEW_PROJECT_SELECTORS (~line 142) and the SUBMIT_BUTTON_SELECTORS tail —
    localized-text fallbacks (Novo projeto / Nuevo proyecto /
    Nouveau projet; aria-label*="Create"). These work today only because the
    icon selector leads — maintenance debt and a silent-failure risk.

Why it works today

t2i is not broken — the icon-ligature selectors lead and match, and the
test accounts happen to be in covered languages. The risk is silent: onboarding
fails only for an uncovered language, with no obvious error.

Proposed fix (structural-first)

Flow is built on Radix UI + Material Symbols. Both expose developer
strings that are never translated:

  • Radix code tokens — role, aria-controls / aria-haspopup, radix-* ids.
  • Material Symbols icon ligatures — i.google-symbols:text('...').
  • Structural attrs — data-slate-editor, child order.

Rework ui_automation.py selectors to lead with structural anchors and keep at
most one localized-text entry as a last-resort diagnostic fallback (or drop it).
Priority: give ONBOARDING_SELECTORS a real structural anchor.

Scope / sequencing

  • Out of scope for the current video Phase 0 work. Do not change t2i
    selectors until this is planned as its own task — t2i works and must not
    regress.
  • The video spike (scripts/smoke_video_editor.py) is establishing the
    discovery pattern: _dump_region() dumps each control's structural
    fingerprint (role / aria-* / id / icon ligatures) so selectors are derived
    from real DOM evidence, not guessed. Once the video work proves the pattern,
    t2i adopts it.

References

  • docs/superpowers/specs/2026-05-18-ui-automation-video-generation-design.md
    §6, §10.4
  • scripts/smoke_video_editor.py — _dump_region, structural §6 selectors
  • src/gflow_cli/api/transports/ui_automation.py — ONBOARDING_SELECTORS,
    NEW_PROJECT_SELECTORS

Activity

  1. ffroliva
    added 5 commits that reference this issue on May 23, 2026

    docs(image): Phase 7 partial pass — race fix verified, count-tab sele…

    509f22a

    fix(image): locale-invariant count tab — position-first selectors (#24 …

    2a6de4d

    chore(image): add DOM dump diagnostic for count-tab discovery (#24)

    6181499

    fix(image): thread output_dir to client constructor so transport _out…

    a6d702f

    fix(image): filter count-tab selectors by 1x/xN text pattern (DOM-evi…

    38d391a

  2. ffroliva commented on May 23, 2026

    @ffroliva
    ffroliva
    on May 23, 2026
    OwnerAuthor

    Status update — partial fix landed, ONBOARDING_SELECTORS still pending.

    Done (count tabs):

    • _COUNT_TAB_TEXT_RE = ^(1x|x[2-4])$ in ui_automation.py:253 filters tabs by the digit+x text that Flow renders identically in every locale (numbers/symbols are not translated, per the issue body).
    • _set_count falls back to positional .nth(count - 1) when the read-back text is unrecognised — locale-invariant click. See ui_automation.py:1013.
    • Landed via commits 2a6de4d, 38d391a, c5c8d4a.

    Still pending (the issue's stated #1 priority):

    • ONBOARDING_SELECTORS (ui_automation.py:183-193) remains nine purely-localized button-text selectors with no structural anchor (Agree / Aceitar / I agree / Concordo / Accept / Create with Flow / Criar com o Flow / Get Started / Começar). An account whose Flow renders in an unlisted language (German, Japanese, ...) cannot pass onboarding.
    • NEW_PROJECT_SELECTORS localized fallbacks + SUBMIT_BUTTON_SELECTORS tail — icon-first selectors lead, so these work today; the localized fallbacks are maintenance debt per the issue body.

    KNOWN_ISSUES.md now carries an Open entry pointing at this issue. Recommend leaving this open until ONBOARDING_SELECTORS gets a structural anchor.

  3. ffroliva
    added 3 commits that reference this issue on May 23, 2026

    docs+test(image): polish — record mode-switch in debugging, surface W…

    3124d91

    fix(image): switch to Image mode in editor + live verify on ffroliva (#…

    db8fb7e

    Merge pull request #51 from ffroliva/feat/locale-agnostic-selectors

    718025e

  4. ffroliva commented on May 24, 2026

    @ffroliva
    ffroliva
    on May 24, 2026
    OwnerAuthor

    Status update — 2026-05-24, develop / post-v0.8.1 (unreleased)

    Phase 1 (env override) shipped via #51 — Playwright launch_persistent_context(locale=...) now reads GFLOW_CLI_LOCALE (default en-US). Documented in .env.template and docs/CONFIGURATION.md as of #53.

    Live-verified end-to-end with GFLOW_CLI_LOCALE=pt-BR uv run pytest -m e2e tests/e2e/test_video_t2v_e2e.py:

    Layer Result
    File written ✅ ~2.5 MB mp4
    Magic bytes ✅ ftypisom
    structlog invariants ✅ MEDIA_GENERATION_STATUS_SUCCESSFUL, video_saved bytes=2558991
    Generated under pt-BR ✅ confirmed via Playwright launch arg

    Still required for Phase 2 (drop --lang=en-US):

    • Invariant capture across ONBOARDING_SELECTORS, parts of NEW_PROJECT_SELECTORS / SUBMIT_BUTTON_SELECTORS, and I2V frame-slot labels via the hardened scripts/dev/capture_locale_invariants.py (filed separately).
    • Once invariants are confirmed across en/pt/es (and ideally a non-Latin locale like ja-JP), drop the --lang=en-US Chromium arg from UiAutomationTransport.setup.

    KNOWN_ISSUES.md § issue #24 now carries the same Progress block.

  5. ffroliva
    added a commit that references this issue on May 24, 2026

    fix(image): re-introduce i2i ref-attach (#50) with locale-agnostic me…

    822866d
  6. svasakorn
    added a commit that references this issue on May 24, 2026

    fix(image): locale-agnostic i2i media-dialog selectors + upload fallback

    50009f2

  7. ffroliva commented on May 24, 2026

    @ffroliva
    ffroliva
    on May 24, 2026
    OwnerAuthor

    Resolved (with one focused follow-up tracked separately)

    This umbrella issue is closed. The locale-agnostic selector work shipped across multiple PRs; the last narrow remaining gap has its own issue.

    What landed under this umbrella

    PR What
    #51 Phase 1: GFLOW_CLI_LOCALE env override (Playwright launch_persistent_context(locale=...)); locale-agnostic count-tabs (^(1x|x[2-4])$ regex on locale-stable digit-x text); locale-agnostic image-mode tab via aria-controls*='IMAGE' (structural).
    #53 Post-#51 docs council audit + drift fixes.
    #60 Phase 2: locale-agnostic _attach_references for i2i — replaced text-based has-text('Upload media') with Material Symbols icon ligature (button:has(i.google-symbols:text-is('upload'))) scoped to the open Radix popover. Added structural iconless selector for "Add to Prompt" and a fail-loud RuntimeError distinguishing Chrome profile language from Google account language. Resolved #56.
    #61 Test transport fix + reference_attached structlog assertion so the e2e can't silently pass on a text-only image again.

    Key learning (committed to agent memory as a durable rule)

    Flow renders dialog labels in the Chrome profile language preference — --lang=en-US Chromium launch arg does NOT override it (it's a server-side preference). The durable selector pattern across the codebase is now button:has(i.google-symbols:text('<ligature>')) with :text-is(...) exact-match when sibling icons would otherwise collide (e.g. upload vs drive_folder_upload).

    One follow-up tracked separately: #63

    FRAME_SLOT_BY_LABEL (div[aria-haspopup='dialog']:has-text('{label}')) at src/gflow_cli/api/transports/ui_automation_video.py:138 is still text-based and would hit the same hang on a non-EN profile for gflow video i2v Start/End frame attach. Spun off to #63 with the full repro shape, suggested fix (mirror PR #60), and test plan (mirror PR #61).

    Thanks

    Multi-PR multi-contributor effort — @svasakorn shipped the i2i half (PR #60 + #61) with proper fail-loud guards and the right framing of the icon-ligature vs text-label distinction; the council audit fixes (PR #53) made sure the documentation kept up.

    Closing.

  8. ffroliva
    closed this as completedon May 24, 2026
  9. ffroliva
    added a commit that references this issue on May 25, 2026

    chore(selectors): address PR #70 review gaps — tier counts, locale co…

    84ae89d

  10. 14 remaining items

  11. svasakorn
    added a commit that references this issue on May 30, 2026

    fix(transports): exit Agent mode before mode switch so the gen panel …

    e757e09
  12. 15990166686ll-crypto
    added 14 commits that reference this issue on Aug 24, 2026

    fix(image): switch to Image mode in editor + live verify on ffroliva (f…

    952fae3

    feat: locale-agnostic selectors (issue ffroliva#24) and multi-locale …

    68e7eee

    Merge pull request ffroliva#51 from ffroliva/feat/locale-agnostic-sel…

    cbb1840

    docs: post-PR-48/ffroliva#51 council audit fixes (T1+T2)

    294074d

    fix(image): re-introduce i2i ref-attach (ffroliva#50) with locale-agn…

    e491d46

    feat(selectors): structural-first cascade for onboarding + I2V frame …

    971ab74

    chore(selectors): address PR ffroliva#70 review gaps — tier counts, l…

    5cb88d2

    test(e2e): adapt t2v assertion to VideoResult wrapper + record de-DE …

    1300a0c

    Merge pull request ffroliva#70 from ffroliva/feature/issue-24-phase2-…

    e1c51a3

    chore(plan): refresh PLAN.md to develop ground truth

    97e31b7

    docs: record ffroliva#63 closure in KNOWN_ISSUES + CHANGELOG

    8934600

    chore(plan): record PR ffroliva#90 / ffroliva#63 closure in PLAN.md

    578e969

    fix(transports): locale-invariant NEW_PROJECT + SUBMIT selectors and …

    6ddc1f0

    fix(transports): exit Agent mode before mode switch so the gen panel …

    42a40de
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions