The Viewer's Job History reads each server's newest runs through an index (#4477)

[erikdarlingdata]

Refs #4477.

Why

The Viewer's Job History (ViewerDataService.BuildJobHistorySql) shows the newest 2,000 job runs across the fleet (or one server) in the chosen window.

• How today's read does it: it reads EVERY job_history row collected in the window plus a day of clock-skew allowance, applies each server's UTC offset, sorts them all, and keeps 2,000.
• Measured on a production store: that meant scanning 4.68 M rows in the 2-day floor. The store then held duplicate rows from a separate collector bug, since fixed; after that cleanup the same scan reads about 1.3 M rows, still for 2,000 shown.
• job_stats re-scans the same window for the long-running check.

What changes

• Each server's newest rows come from V150's idx_job_history_server_run (server_id, run_datetime DESC, instance_id DESC) as a per-server LATERAL … ORDER BY run_datetime DESC, instance_id DESC LIMIT n, then merge with today's exact order and tie-break (run_datetime_utc DESC, instance_id DESC).
• Why this is exact: a server's UTC offset is constant across its rows, so "newest by run_datetime − offset" per server is "newest by run_datetime", and the merged top-n is the same set.
• The per-server read is driven by the servers that HAVE rows in the window, not the servers registry. A server whose registry entry was removed still shows its history under its collected name, as the old LEFT JOIN servers did.
• job_stats (the average successful duration and last success, for the long-running flag) is computed only for the (server, job) pairs in the result.

Measurements

Measured on a local PostgreSQL 18 + TimescaleDB 2.30 store planted like a busy fleet: 43 servers × 72 hours, 2,015,496 job_history rows, the two oldest daily chunks compressed, after VACUUM ANALYZE. The parameters are the Viewer's defaults: the last 24 hours, all servers, the 2,000-row cap. EXPLAIN (ANALYZE, BUFFERS), run warm in both orders, and once after a container restart. The restart empties shared_buffers but not the OS page cache, so compare the "read" counts rather than the cold milliseconds.

	warm	after restart
before (fleet-wide scan)	573 ms, 63,687 shared hit	575 ms, 35,407 hit + 28,280 read
this PR (per-server top-N)	300 ms, 163,424 shared hit	381 ms, 157,931 hit + 5,482 read

The per-server read touches more pages in total: each of the 43 servers' index descents fetches up to the row cap before the fleet-wide merge. Nearly all of those pages are already in memory. It reads about 5× fewer blocks from outside shared_buffers, and it's about 47% faster warm. The same measurement on a production-sized store is planned after the next install, as a confirmation.

Test plan

• JobHistoryPerServerTopNLiveTests (live TimescaleDB rig):
  • NewSql_MatchesOldSql_ExactlyAcrossOffsetsTiesAndTheCapBoundary: 5 servers (offsets +60 / −300 / 0 / no server_properties row / deregistered), 3 days of runs with retries and failures, a cross-server tie on run_datetime_utc, a same-server tie on run_datetime, and a cap that cuts the plant. It matches dev's SQL row for row, fleet-wide and scoped, in both orders.
  • DeregisteredServersRow_StillSurfaces_LikeTheOldLeftJoin: a server removed from servers still returns its rows under its collected name.
  • ScopedRead_UsesTheIndex_WithNoExtraSort: the scoped read uses idx_job_history_server_run with no sort on run_datetime. The same assertion against dev's SQL on the same rig FAILS (no LATERAL).
• Mutations:
  • the instance_id DESC tie-break dropped from the fleet merge → the equivalence pin RED (a wrong server at row 19);
  • the registry-driven server set restored → the deregistered-server pin RED;
  • both reverted.
• Also green: JobHistoryGroupedStatsMatchWindowFunctionLiveTests, CollectionLogWatermarkAndJobHistoryIndexesRungTests, JobHistoryCapLabelTests, JobHistoryIdentityEpochTests and DocCommentHygieneTests. Darling.Tests builds with 0 warnings.

CHANGELOG

SECTION: Changed
ENTRY:

• The Viewer's Job History reads each server's newest runs through an index instead of sorting the whole fleet's window ([The Viewer's Job History reads each server's newest runs through an index (#4477) #4493]) - it shows the same rows in the same order, including a server whose registry entry has since been removed.
  REF:
  [The Viewer's Job History reads each server's newest runs through an index (#4477) #4493]: The Viewer's Job History reads each server's newest runs through an index (#4477) #4493