Skip to content

test: book-site - add Playwright frontend regression coverage #14

Open

Description

@ashleyshaw

Outcome

Complete reliable Playwright frontend regression coverage for lightspeedwp/ThisOneTimeOnAcidCom, including an explicitly configured, read-only Netlify Deploy Preview test mode.

GitHub issue: #14 — Playwright frontend regression coverage.

Reuse the existing Playwright foundation. Do not replace meaningful assertions with weaker checks, skip failing journeys silently, or treat a successful build as proof that the application works.

Repository instructions and baseline

Read the repository’s current AGENTS.md, CONTRIBUTING.md, relevant guidelines/, deployment documentation and branch-validation rules before implementation. Repository-specific instructions take precedence over general conventions.

The last inspected develop revision was 243c8670472a87c2f4379ec14c22aa68acf7bd76. Refresh the baseline before changing files and record the actual tested revision.

Existing infrastructure includes:

  • playwright.config.ts
  • e2e/routing.spec.ts
  • Headless, debug and UI package scripts
  • A custom production-output server
  • Five configured browser/device projects

PR #265 delivered this foundation alongside broader AI Ops work. It does not complete this issue’s acceptance criteria.

Earlier baseline execution produced 32 passes and 20 failures across Chromium/mobile Chromium. Production startup failed on unresolved React imports, while several checks passed against an empty application. These are historical findings to reproduce—not results to copy into a new validation report.

Branch instructions

  • Repository: lightspeedwp/ThisOneTimeOnAcidCom
  • Proposed branch: test/book-site-playwright-regression
  • Base: develop
  • GitHub PR classification: Test Coverage
  • GitHub PR label: type:test
  • Proposed PR title: test: book-site - Complete Playwright regression coverage
  • PR template: explicitly select the shared pr_test.md
  • Naming policy: repository docs/BRANCH_NAMING_LOCAL.md, scripts/validate-branch-name.js and the organisation branching strategy
  • Constraints: preserve unrelated changes; do not rename existing work, force-push, merge, trigger deployment or modify hosting settings

The proposed name passed the repository’s branch validator. Recheck existing work before creating it.

Routing conflict: the repository routes test/ to pr_test.md, while the shared routing configuration specifies pr_chore.md; the organisation strategy separately names pr_test.md. Use the outcome-specific Test template explicitly and disclose the discrepancy. Do not change governance routing in this work.

The test/ prefix fits the suite/harness outcome. Substantial runtime or hosting remediation must remain explicit prerequisite work rather than being hidden inside a testing PR.

1. Establish a working production baseline

Before extending coverage:

  1. Verify clean installation using repository-approved Node 24/npm tooling and the committed lockfile.
  2. Investigate the React/React DOM installation and production-bundling failures.
  3. Confirm the generated application actually renders meaningful content.
  4. Report prerequisite fixes separately, identifying their scope and approval requirements.
  5. Do not inject missing runtime dependencies, suppress console errors or alter application behaviour only inside test setup to manufacture passing results.

Preserve existing Jest, Bats, Spec Kit and accordion tests.

2. Implement two explicit execution modes

Local production mode

  • Build once and serve that exact output on a fixed loopback port.
  • Reuse the existing server only where its behaviour is suitable and documented.
  • Prevent accidental reuse of a stale or unrelated server.
  • Keep the development server separate from production-build acceptance.
  • Require route-specific rendered content—not merely HTTP 200, a retained URL or absence of “404”.
  • Collect page errors and required-resource failures.
  • Distinguish expected negative-case responses from unexpected failures.

The custom server does not reproduce Netlify configuration, headers or deployment processing. Document that limitation.

Netlify Deploy Preview mode

Introduce and document a proposed PLAYWRIGHT_BASE_URL option:

  • Accept an explicitly authorised HTTPS preview URL.
  • Disable local build/server startup when remote mode is selected.
  • Verify the deployment is ready and record its deployed Git SHA.
  • Do not silently substitute the production domain or localhost.
  • Keep deployed tests read-only and mark the eligible subset, for example @preview-safe.
  • Report authentication or access restrictions as blockers; do not bypass them.
  • Treat a missing preview URL or unknown deployment revision as incomplete deployed acceptance.

These are new requirements; the current configuration does not already support this mode.

3. Verify actual Netlify behaviour

The inspected repository has a nested src/app/netlify.toml, with older pnpm/Node settings that conflict with current npm/Node guidance. Do not assume that file is the effective deployed configuration.

Before claiming deployment compatibility, establish:

  • Effective base/package directory, build command and publish directory
  • Runtime/package-manager versions
  • Configuration file actually used by Netlify
  • Deploy Preview versus production overrides
  • Tested deployment URL and Git SHA

The current public/_redirects is a regular file containing the non-forced SPA fallback. Preserve real asset handling and specific redirects.

On the authorised preview, verify:

  • Direct loading and refresh of nested application routes
  • Route-specific rendered content
  • Existing article and unknown-article behaviour
  • Unknown-route recovery
  • /read-the-book alias behaviour
  • Required JavaScript/CSS responses, non-empty content and correct MIME types
  • Representative images and unexpected resource failures
  • Browser runtime errors
  • Observed response headers and caching against the approved deployment policy

Do not move configuration files, loosen CSP, introduce forced asset-catching redirects, cache HTML immutably, change DNS/domains or alter production settings as a testing workaround. Identify required hosting changes separately and coordinate with existing deployment work.

4. Complete relevant frontend coverage

Extend the configured e2e/ directory. Do not add tests under the originally suggested tests/e2e/ unless discovery is intentionally reconciled.

Routes and navigation

Use src/app/routes.ts as the active-route source.

Cover meaningful page content, direct loads, refreshes, unknown routes, homepage calls to action and footer destinations. Exclude unregistered legacy routes.

Test header menu:

  • Opening and aria-expanded
  • Initial focus and keyboard navigation
  • Escape dismissal and focus return
  • Scroll locking
  • Closing after navigation

Replace broad, ambiguous content selectors and conditional assertions that allow required journeys to pass without running.

Forms and journal

Use synthetic data locally:

  • Empty/malformed email validation
  • Ordinary draft submission to /thank-you
  • Waitlist confirmation
  • Special-email branch to /ebook, without exposing personal addresses
  • Contact required fields and current mock-success dialog
  • Journal index → existing article → index
  • Unknown-slug recovery

Do not claim real email delivery or provider capture. Do not invent category-filter acceptance where handlers do not exist.

Deployed form submissions are excluded by default. Testing Netlify Forms, CRM, email or other external writes requires separately authorised isolation.

Preferences and responsive behaviour

Cover:

  • Header theme toggle, keyboard operation and persistence
  • Mobile navigation and key forms
  • Reduced-motion and cursor preferences where implemented
  • Meaningful overflow/usability checks rather than viewport dimensions alone

Keep design review separate from automated visual approval.

Reader and search

Track prerequisite defects explicitly:

  • Reader content-loader paths versus actual manuscript location
  • Search destinations pointing at unregistered legacy routes

After remediation, cover:

  • Reader content, previous/next boundaries and chapter navigation
  • Keyboard controls, settings dismissal and saved position
  • Search query/URL synchronisation, no-results state, tabs and sorting
  • Successful result navigation

Avoid fixed reader page counts and ambiguous duplicate controls. Do not count skipped prerequisites as completed coverage.

5. Optimise execution and reporting

For the initial gate:

  • Use desktop Chromium and mobile-emulated Chromium.
  • Install Chromium and required host libraries reproducibly.
  • Build once per local-production job.
  • Use bounded workers, timeouts and retries.
  • Avoid arbitrary sleeps, unbounded retries and brittle load-time thresholds.
  • Cache dependencies appropriately without reusing stale build output.
  • Retain sanitised screenshots, traces and HTML reports on failure.
  • Ensure local failures can produce traces even without retries.

Firefox/WebKit acceptance remains follow-up scope unless explicitly expanded.

Add dedicated E2E CI only through the authorised workflow-management route. Do not silently modify centrally managed workflows.

6. Documentation and PR evidence

Document installation, browser prerequisites, both execution modes, one-test execution, trace debugging, fixture maintenance and adding scenarios.

After implementing the proposed remote mode, document commands such as:

# Local production mode
npm run test:e2e -- --project=chromium --project="Mobile Chrome"

# Authorised Netlify preview: read-only subset
PLAYWRIGHT_BASE_URL="<authorised-preview-url>" \
 npm run test:e2e -- \
  --project=chromium \
  --project="Mobile Chrome" \
  --grep @preview-safe

The PR description must identify:

  • Exact tested revision and deployed SHA where applicable
  • Node/npm/Playwright/browser versions
  • Execution mode and sanitised target URL
  • Commands and outcomes
  • Passed, failed, skipped and environment-blocked results separately
  • Evidence links
  • Remaining dependencies and agreed exclusions

Reference GitHub issue #14 without claiming it is complete before acceptance is met.

Acceptance criteria

  • Production startup succeeds from a reproducible clean installation.
  • Local production and Netlify preview modes are explicitly separated.
  • Remote mode uses an authorised preview and does not start localhost.
  • Core journeys require meaningful rendered content and mandatory assertions.
  • Runtime and required-resource failures cannot produce false-green results.
  • Desktop/mobile Chromium core suites pass locally and in authorised CI.
  • Read-only preview checks verify deployed routing/assets at a recorded SHA.
  • Forms cannot submit externally without separate authorisation.
  • Reader/search prerequisites are resolved or explicitly tracked; exclusions are not presented as passes.
  • Repeat runs demonstrate isolation.
  • Failure artefacts are actionable and sanitised.
  • Documentation and PR description match actual implementation and results.
  • Unrelated tests, hosting settings and governance remain intact.

Definition of Ready

  • Current repository instructions and baseline reviewed.
  • Existing work checked before creating the proposed branch.
  • Runtime prerequisites identified.
  • Authorised CI ownership confirmed.
  • Preview URL, deployed SHA and effective Netlify configuration available for deployed acceptance.

Definition of Done

  • Acceptance criteria supported by revision-specific evidence.
  • Required review findings dispositioned and approvals recorded.
  • Local, CI and preview outcomes distinguished accurately.
  • Applicable changelog requirement or authorised exemption satisfied.
  • Remaining work and exclusions documented.
  • Linked records updated only after approval; merge alone does not establish completion.

Audit appendix — supporting evidence and remaining findings

Baseline and evidence limits

Repository: lightspeedwp/ThisOneTimeOnAcidCom
Audited revision: develop at 243c8670472a87c2f4379ec14c22aa68acf7bd76.

Recheck each finding against the implementation revision before remediation. Distinguish reproduced failures, static findings and environment blocks. Do not present these results as PR #265 final-head execution or Netlify deployment acceptance.

Supporting execution results

  • Chromium and mobile Chromium: 32 passed, 20 failed.
  • Explicit production-mode Chromium repeat: 16 passed, 10 failed.
  • Firefox/WebKit/Mobile Safari: 78 browser-launch blocks caused by missing host libraries; no application verdict was established.
  • Build regression checks: 12/12 passed.
  • Targeted server unit tests: 26/26 passed across two suites.
  • Accordion test could not load React.
  • TypeScript checking stopped at TS5101, concerning deprecated baseUrl, before application checking.
  • Full Jest was not run; the audit avoided unrelated tests documented to write repository files.

Build and server-test success did not establish a working browser application.

Reproduced startup and false-positive failures

  • Production rendered an empty application root with Failed to resolve module specifier "react/jsx-runtime".
  • vite.config.ts externalised React-related modules without a browser import map. React and React DOM were also absent after clean installation, being declared optional peers.
  • Sixteen Chromium checks passed despite the blank application. URL-only refresh checks, conditional navigation assertions and insufficient positive asset expectations require strengthening.
  • Fix startup and assertions independently. Do not weaken assertions or extend timeouts to conceal the failure.

Static findings requiring follow-up

Test infrastructure

  • .github/workflows/tests.yml referenced missing scripts/validation/compare-jest-failures.cjs.
  • Jest discovery omitted tests/accordion.test.tsx and src/__tests__/redirects.test.ts; the inspected configuration lacked a TypeScript/TSX transform.
  • Local trace capture depended on retries, while local retries defaulted to zero.
  • The local server implemented its own fallback and did not evaluate Netlify configuration or response headers.

Semantics and application journeys

  • Root and journal pages duplicated main landmarks and main-content IDs. Broad content locators risk strict-mode ambiguity after startup is restored; the observed startup failures themselves returned zero matching elements.
  • Search indexed legacy destinations /blog/:slug and /portfolio/:slug, absent from the active router.
  • Footer theme controls used a provider fallback with a no-op setter, while the header maintained separate theme state.
  • Footer privacy/terms controls targeted unregistered routes.
  • Contact labels lacked explicit input associations.
  • Journal category elements had no filtering handlers; article bodies contained placeholder content.
  • Missing Figma assets were replaced by grey SVG fallbacks. Recover intended assets rather than treating placeholder responses as successful image validation.

These were static findings, not browser-validated downstream behaviour: application startup blocked that validation. Confirm intended prototype behaviour before classifying mock forms or placeholder content as delivery defects.

Scope and routing of follow-up

Evidence retention

Audit outputs were recorded at:

/tmp/toa-playwright-chromium.json
/tmp/toa-playwright-other.json
/tmp/toa-playwright-production.json
/tmp/toa-jest-server.json
/tmp/toa-build-bats.log
/tmp/toa-browser-deps.log
/tmp/toa-accordion.log
/tmp/toa-tsc.log

These are temporary execution paths, not durable attachments. Their continued availability is unverified. Preserve sanitised reports in durable PR/CI artefacts during the next validation; do not claim the files are attached to this issue.

Activity

  1. linear-code commented on Oct 3, 2026

    @linear-code
  2. self-assigned this
    on Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Fields

Priority

None yet

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions