Job History: GROUP BY join replaces the per-job window function, tab off the 30s timer (#4229)

[erikdarlingdata]

Closes #4229.

Why

PR #4235 (merged) gave Job History's event-windowed read a collection_time floor so TimescaleDB skips out-of-window chunks. That left the window's own row cost. The per-job success average and max ran as AVG/MAX OVER (PARTITION BY server_id, job_id) against every step row the 24h window matched. That was all rows, not just the 2,000 the tab shows. Postgres sorted and aggregated the whole windowed set before the final ORDER BY ... LIMIT trimmed it. That ran every 30 seconds while the tab was open.

What changes

ViewerDataService.BuildJobHistorySql (Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.JobHistory.cs)

A new job_stats CTE computes the average and max once per (server_id, job_id). It uses a plain GROUP BY over the window's step_id-0 SUCCESS rows. base selects only the raw columns and runs ORDER BY run_datetime_utc DESC, instance_id DESC LIMIT with no aggregation. The planner picks a top-N sort instead of materializing the full windowed set. The final SELECT left-joins job_stats onto the rows base already picked.

Row selection is unchanged. The old ORDER BY/LIMIT never read the analytic columns. Trimming to the newest rows first and joining per-job stats after picks the same rows, order, and values. A job with no successful step-0 row in the window gets NULL avg and last-success from the left join. That matches what AVG/MAX returned for an all-NULL partition.

This is a two-scan shape now: job_stats and base both scan job_history, each independently floored. That is the same family as BlockingDurationStatsSql. The floor-count pin in EventWindowedReadsCarryTheFloorTests for BuildJobHistorySql moves from 1 to 2. The live chunk-reduction margin for job history drops from 3 to 2. Job history's oracle is single-scan on the old side. Doubling the scan count narrows the absolute reduction even though each scan is still floored.

30s fleet timer (MainWindow.xaml.cs, OnRefreshTimerTick)

Job History joins Recommendations, FinOps, and Overview in the tab-early-return. The mechanism matches #4227/#4240, applied independently here. The tab already refreshed on activation through the existing LoadVisibleTabAsync/JobHistoryContent.RefreshJobsAsync() wiring and has its own Refresh button. Only the 30s poll needed removing. Job history changes at job-run cadence, not every 30 seconds.

Lite parity (Lite/Services/LocalDataService.JobHistory.cs)

GetJobHistoryAsync carried the same window-function-over-every-row pattern (DuckDB, v_job_history) and got the same rewrite. Lite's Job History tab has no fleet-timer poll. Its only timer (_staleDataTimer, 10s) updates a "Refreshed Xs ago" label and never re-runs the query. No refresh-mechanism change was needed.

No MCP tool duplicates this per-job average computation. The only MCP job-history references are store-metrics evidence checks. Those are unrelated.

Test plan

Measured on a seeded probe database: 250,000 job_history rows across 13 daily chunks, 200,000 in the 24h window. This scales down from the issue's 21.7M-row/0.9M-window production numbers at the same ratios. EXPLAIN (ANALYZE, BUFFERS), fleet-wide, no server filter, LIMIT 2000:

	Before	After
Execution time	484-505 ms	299-300 ms
Planning time	16-28 ms	29 ms
Sort of the windowed set	external merge, disk (temp read=3750 written=3757, ~29 MB spilled)	quicksort, memory (343 kB)
Shared buffers	4,407 (+ disk spill)	8,713 (all in shared_buffers, no spill)

The old plan's WindowAgg forced a sort over all 200,000 windowed rows and spilled to disk. The new plan's HashAggregate scans only the step-0/success subset and never spills. The double scan of job_history raises shared-buffer touches but removes the disk-spilling sort and wins on wall time.

• Darling.Tests builds, 0 warnings.
• Lite.Tests builds, 0 warnings.
• New live test JobHistoryGroupedStatsMatchWindowFunctionLiveTests (two facts):
  Fact 1 drives GetJobHistoryAsync on a scoped server with 9 jobs tied on one run_datetime_utc.
  8 successes and 1 job whose only step-0 row failed (NULL average), limit = 5.
  Compares every field row-for-row against the pre-fix SQL from origin/dev.
  Crosses the LIMIT boundary through the tie.
  Confirms the NULL-average job's IsLongRunning stays false and LastSuccessfulRunUtc stays null.
  Fact 2 proves the join keys on (server_id, job_id), not job_id alone:
  two servers share a job_id with different durations on a fleet-wide unscoped call.
• Proved fact 2 red once: dropping the join's server_id half fanned one base row to both servers'
  job_stats. Assert.Equal(2, mine.Count) failed with Actual: 4. Restored the join, reran, green.
• Updated EventWindowedReadsCarryTheFloorTests' BuildJobHistorySql floor-count pin from 1 to 2
  and EventWindowedReadsAreBoundedLivePostgresTests' chunk-reduction margin for job history from 3 to 2.
  Both have an inline reason.
• Full Darling.Tests suite (13,845 tests) run once, after git merge origin/dev and a fresh darlingtest:
  4 failures on the first pass.
  Two (DocCommentHygieneTests.TheBoundedSetIsComparedForEquality, ...EveryCrefTargetResolvesOrIsBounded)
  were mine: a <see cref="Tied"> pointed at a local variable. Fixed to <c>tied</c>.
  The other two (ServerListAndSummaryPlanShapeTests, CaptureDownChunkOrderTests) are unrelated
  collection_log chunk-plan-shape tests that do not touch job history or MainWindow.
  Re-run alone on a freshly recreated darlingtest, all 81 passed. Shared-store cross-test pollution,
  not this change.
• Did not re-run the complete 13,845-test suite after the cref fix. The 81-test isolated re-run is
  green and is the evidence for this PR. CI decides whether a second full Darling run is needed.
• Lite gap closed: added Lite.Tests/JobHistoryGroupedStatsMatchWindowFunctionTests (2 facts, real DuckDB
  via SharedDuckDbFixture).
  GroupByJoin_MatchesTheOldWindowFunction_ThroughATieAndANullAverage_AtTheLimitBoundary compares the new
  job_stats statement against the pre-fix window-function text from origin/dev, field-for-field, in order.
  Seed: eight jobs tied on one run_datetime, one job with a NULL average, limit = 5, crosses the boundary.
  GroupByJoin_KeysOnServerIdAndJobId_NotJobIdAlone proves the join on (server_id, job_id), not job_id
  alone: two servers share a job_id with different durations on a fleet-wide call.
  Proved it red once: dropping server_id turned Assert.Equal(2, mine.Count) into Actual: 4.
  Full Lite.Tests suite: 5,341 tests, 0 failures.

CHANGELOG entry

SECTION: Fixed
ENTRY:

• Job History loads faster and stops polling every 30 seconds ([Job History: GROUP BY join replaces the per-job window function, tab off the 30s timer (#4229) #4256]): The tab used a window function
  over every job step in the 24h window. It then sorted and trimmed to 2,000 rows. On a large store this
  processed far more rows than the tab showed, every 30 seconds. The average and max are now computed once
  per job from its successful step outcomes. The result joins onto the newest rows after row selection.
  The tab now refreshes on open, focus, or user click. Lite's job history read had the same pattern and
  got the same fix.
  REF:
  [Job History: GROUP BY join replaces the per-job window function, tab off the 30s timer (#4229) #4256]: Job History: GROUP BY join replaces the per-job window function, tab off the 30s timer (#4229) #4256

Generated with Claude Code

https://claude.ai/code/session_01FVjn4PBJN71NQXdFo6ZxNQ