[erikdarlingdata]

Claude posting for Erik Darling.

Fixes #5931.

Why

The Darling web UI smoke compared four page screenshots pixel for pixel and failed the job on any difference. One sidebar link (#5929) turned the check red on every open web-UI PR. Erik ruled on 2026-10-16 that the compare is advisory: a differing screenshot is reported, real breakage still fails.

What changes

tools/ui-smoke/shots.mjs

• runShots now returns { problems, shots }. shots has one { name, differing, pct } per compared page.
• A page whose screenshot differs prints NOTICE screenshot <page>: <n> pixels differ (<pct>%); baseline, actual and diff are in shot-failures, still writes the baseline, actual and diff images, and is not a problem.
• The update_baselines path (UI_SMOKE_SHOTS=update) is unchanged.

tools/ui-smoke/smoke.mjs (the caller, needed for the exit code and summary line)

• Prints screenshots: N matched, M differ (advisory), K problems after the PROBLEM lines (compare mode only).
• report.json is now { pages, screenshotProblems, screenshots }.
• Exit code is unchanged: only page problems and screenshotProblems give exit 1.

.github/workflows/darling-ui-smoke.yml

• New step "Screenshot compare summary" (if: always(), right after the run step) reads report.json and appends a table of per-page pixel counts, plus the sentence that the images are in the darling-ui-smoke artifact, to $GITHUB_STEP_SUMMARY. Nothing is parsed from the log.
• The darling-ui-smoke upload already had if: always(); I renamed it to "Upload logs, screenshot differences and traces". No new secrets, permissions or triggers. Paths filter and draft skip are untouched.

tools/ui-smoke/README.md: says the compare is advisory, lists what still fails the job, and how to refresh baselines (update_baselines=true dispatch, copy the PNGs from darling-ui-smoke-new-baselines, commit).

What fails the job now, and what only reports

Still fails (exit 1, as before):

• a page that does not load, a console or page error, a failed request;
• a panel or grid the page's expect requires that is missing or empty;
• a failed step (stage throws, seed or service failure, fleet never turns collection stale);
• no baseline for a page, a screenshot size that differs from its baseline (the viewport shot is always 1280 by 800, so this means a broken baseline, not a layout change), a page that never settles.

Only reports: a screenshot whose pixels differ from its baseline (NOTICE, images, summary row).

I left the existing Upload screenshot differences step (if: failure(), artifact darling-ui-smoke-screenshot-diffs) alone. The same images are in darling-ui-smoke on every run.

Test plan

• node --check tools/ui-smoke/shots.mjs and smoke.mjs pass. The script has no unit test or dry mode.
• Workflow YAML parses.
• Dispatched on this branch while the four baselines still differ on dev: run 38887769491 (job green, 34/34 pages clean) and, after the last commit, run 38888443595 (green).

  • NOTICE lines in the log: fleet 5789 (0.565%), ui-sql-overview 5494 (0.537%), ui-sql-queries 5494 (0.537%), admin-servers 6378 (0.623%).

  • screenshots: 0 matched, 4 differ (advisory), 0 problems.

  • "Screenshot compare summary" step ran green. I could not read the rendered job summary through the API, so I ran the step's own script locally on that run's report.json. It wrote:

    Page	Pixels that differ	Percent	Result
    fleet	5789	0.565	differs
    ui-sql-overview	5494	0.537	differs
    ui-sql-queries	5494	0.537	differs
    admin-servers	6378	0.623	differs

  • Artifact darling-ui-smoke was uploaded on the green run. Downloaded run 38887769491: ui-smoke/out/shot-failures holds baseline, actual and diff PNGs for all four pages, and ui-smoke/out/report.json has the screenshots array. The summary sentence names ui-smoke/out/shot-failures.

• Not run: a red-path proof (a real breakage still exits 1). The change touches only the screenshot-difference branch; the page-problem and screenshotProblems exit logic in smoke.mjs is unchanged.
• Not run: an update_baselines dispatch (that code path is untouched).

For the coordinator

• Web UI smoke: refresh the four screenshot baselines (#5929) #5930 (baseline PNGs) is independent. This PR does not touch tools/ui-smoke/baselines/.
• A drift nobody refreshes no longer turns anything red. The README says to read the NOTICE lines on a PR that changes a page.

CHANGELOG entry

SECTION: Fixed (the Unreleased section has Added and Fixed only, no Changed section)
ENTRY: - The web UI smoke check reports a screenshot difference instead of failing on it ([#5935]) - A page that differs from its screenshot baseline no longer fails the job. It prints a NOTICE with the pixel count, writes the baseline, actual and diff images to the darling-ui-smoke artifact, and adds a table to the job summary. A page that does not load, a console or page error and a missing panel still fail the job.
REF: [#5935]: #5935

[claude]

Verdict: LGTM

Claims

• VERIFIED — runShots now returns { problems, shots }, with one { name, differing, pct } per compared page — tools/ui-smoke/shots.mjs:135-240 (push at :226)
• VERIFIED — a differing screenshot prints a NOTICE, still writes baseline/actual/diff images, and is not pushed to problems — tools/ui-smoke/shots.mjs:227-230
• VERIFIED — the update_baselines path (UI_SMOKE_SHOTS=update) is untouched — tools/ui-smoke/shots.mjs:202-206 (unchanged branch)
• VERIFIED — smoke.mjs prints the screenshots: N matched, M differ (advisory), K problems line, compare mode only — tools/ui-smoke/smoke.mjs:232-236
• VERIFIED — report.json is now { pages, screenshotProblems, screenshots } — tools/ui-smoke/smoke.mjs:238
• VERIFIED — exit code logic unchanged: only page problems and screenshotProblems give exit 1 — tools/ui-smoke/smoke.mjs:245 (failed.length || shotProblems.length)
• VERIFIED — new "Screenshot compare summary" step is if: always(), runs right after the smoke step, and reads report.json directly (nothing parsed from the log) — .github/workflows/darling-ui-smoke.yml:139-154
• VERIFIED — the existing darling-ui-smoke upload step only had its display name changed; no new secrets/permissions/triggers, paths filter and draft skip untouched — diff shows only the name: line changed on that step, on:/permissions: blocks untouched
• VERIFIED — the pre-existing Upload screenshot differences step (if: failure(), artifact darling-ui-smoke-screenshot-diffs) was deliberately left alone, and its images are already covered by the always-on darling-ui-smoke artifact — confirmed no diff to that step (darling-ui-smoke.yml:189-196) and that ui-smoke\out\* (always-on upload) already includes shot-failures\*.png
• UNVERIFIED — the dispatched CI runs (38887769491, 38888443595) came back green with the four NOTICE lines and the quoted job-summary table — these are live GitHub Actions run logs/artifacts, not checkable from a read-only tree
• UNVERIFIED — node --check passes on both files — sandbox blocked running node in this session; the diff is small and type-safe-looking JS, but it was not executed

Riskiest lines

1. tools/ui-smoke/shots.mjs:227 — differing > MAX_DIFF_PIXELS (0) now only logs a NOTICE instead of failing the run; this is the crux of the behavior change — UNPINNED (no unit test exists for this tool; verified only by reading the code and the cited live CI run in the PR body)
2. tools/ui-smoke/smoke.mjs:238 — report.json shape change (screenshots field added) that the new workflow step depends on by field name — UNPINNED, but the workflow step is defensive (Test-Path, .Count -eq 0 guards) so a shape drift degrades to "nothing printed," not a crash
3. .github/workflows/darling-ui-smoke.yml:145 — $shots = @((...).screenshots) wraps the parsed JSON in @() to guard PowerShell's single-element-array unwrapping — UNPINNED, but correctly anticipates a real PowerShell/ConvertFrom-Json gotcha

Traps near this diff

• dorny/paths-filter v4 evaluates each pattern as an INDEPENDENT predicate... — does not apply — this diff adds no paths-filter patterns; darling-ui-smoke.yml's on.pull_request.paths block is untouched.
• Under set -euo pipefail, a bare $(gh api …) assignment that fails kills the step... — does not apply — the new step is pwsh, not bash, and makes no gh api calls; it reads report.json via Get-Content/ConvertFrom-Json with an explicit Test-Path guard (darling-ui-smoke.yml:144).
• gh api --paginate --jq runs the filter once PER PAGE... — does not apply — no gh api usage anywhere in this diff.
• A summary helper reading "$1" truncates every message whose echo wraps with a trailing backslash... — does not apply — the new summary step builds $lines as a PowerShell array and joins it with Add-Content, no echo/"$1" shell-argument pattern involved (darling-ui-smoke.yml:147-154).
• claude-code-action refuses to run when .github/workflows/claude-review.yml differs from the DEFAULT branch's copy... — does not apply — this PR does not touch claude-review.yml.
• Lite and Darling ship in parity... — does not apply — tools/ui-smoke is Darling-only web-UI tooling (there is no Lite web UI to keep in parity; Lite's own smoke tooling lives in tools/ui-smoke-wpf, untouched here).
• Ruling (#5833): tests are load-bearing — does not apply as a violation — no new test was added or needed to be; this tool has no existing unit-test suite (confirmed: no *.test.* under tools/ui-smoke/), and the PR body says so plainly ("The script has no unit test or dry mode").

Divergence (blind read vs PR body)

None — the body's account matches the diff on every code claim graded above. One thing I flagged independently in the blind read (the now-largely-redundant Upload screenshot differences / if: failure() step, since a screenshot diff can no longer fail the job) turned out to be a point the PR body addresses explicitly and by design ("I left the existing Upload screenshot differences step... alone. The same images are in darling-ui-smoke on every run.") — a documented tradeoff, not an omission, so it stays a comment rather than a finding.

Reviewer checklist

• Confirm the cited CI dispatch runs (38887769491, 38888443595) are actually green and that the job summary renders the table as shown in the PR body — not checkable from this read-only tree.