Store Checkpointer Pressure judges the average sync per checkpoint, not the interval's sum (#4037)

[erikdarlingdata]

Store Checkpointer Pressure now judges the average sync time per checkpoint in the interval, not the interval's summed sync time. The old rule compared an hourly sum (about twelve timed checkpoints on the default five-minute checkpoint_timeout) against a per-checkpoint bar (CheckpointSyncBarMs, 10 s, the MCP read deadline). So a healthy store whose checkpoints each sync in five to eight seconds breached every interval and never recovered: about four alerts a day at the 360-minute cooldown on one production store, and a second store sitting on the bar.

Closes #4037

What changed

• Schema rung V140 adds nullable collect.store_metrics.checkpoints_timed, filled on the object_kind = 'checkpointer' row only, with no default and no backfill. StorageVersion.SchemaVersion goes 139 → 140, and the viewer's schema probe recognises the new column. Fleet stores go from schema 139 (or 137) to 140 on the next install.
• StoreSelfMetrics writes the timed count from pg_stat_checkpointer.num_timed (PostgreSQL 17+) or pg_stat_bgwriter.checkpoints_timed (older).
• DarlingStoreMetricsReader: CheckpointerSample and CheckpointerDelta carry the timed count. CheckpointCount = timed + requested, and AverageSyncMsPerCheckpoint = sync / count. IsPressure is now requested > 0 (unchanged) or average > bar.
  • When either sample of the pair has no timed count (a row written before V140), the average arm is unmeasured. It never falls back to the old sum, so the alert can't fire from pre-rung rows.
  • An interval with zero checkpoints is not judged.
  • A timed count that goes backwards counts as a reset, like the other counters.
• DarlingSelfAlertEvaluator: the alert message, detail, short message and recovery message state the checkpoint count and the per-checkpoint average. The stale pointers to two fixes that already shipped are removed. The served text has no issue refs.
• get_store_metrics: the checkpointer block adds timed, cumulative_timed, checkpoint_count and average_sync_ms_per_checkpoint. No field is renamed, and sync_bar_ms stays. The reading note states the average and the count. No tool description changed.

Tests

• StoreToastAndCheckpointerTests (67): each case pinned:

  • a healthy store (many checkpoints, a low average, a sum far past the bar) → no pressure;
  • real pressure (a high average) → pressure;
  • the requested arm;
  • zero checkpoints in the interval;
  • a pre-rung null-timed pair → average arm unmeasured.

  The column-list and SQL assertions and the four CheckpointerPressure_* message assertions are also updated.

• New CheckpointsTimedRungTests (3) takes over the top-rung claims from PostmasterStartTimeRungTests (5), following the V138 → V139 hand-off pattern.

What was run (coordinator, on the branch head)

• Build: clean.
• Live PostgreSQL (TimescaleDB 2.28.1 on PostgreSQL 18, throwaway container):
  • CheckpointsTimedRungTests 3/3;
  • PostmasterStartTimeRungTests 5/5;
  • StoreSelfMetricsTests 33/33 (the insert writes the new column on PostgreSQL 18);
  • StoreToastAndCheckpointerTests 67/67;
  • QsCaptureModeRouteKnobToastRungTests 6/6.
• Full Darling suite without a live database, on macOS: 30 failures, all in Windows-only viewer, dialog and path-convention classes (PostgresAddServerDialogTests, ViewerWaitPickerLogicTests, DarlingManagedPostgresTests, AgTopologyCardsTests and similar). None is in a class this PR touches. I did not run a dev baseline on this Mac, so CI's full run is the real check.

Not verified

• The rule on a live production store. After the install, the alert should go quiet on a healthy store only after one full interval in which both samples carry the timed count. Until then the average arm is unmeasured, and only the requested arm can fire.
• PostgreSQL 16-and-older pg_stat_bgwriter.checkpoints_timed against a live server. The fixture runs 18.

CHANGELOG entry

• Store Checkpointer Pressure no longer fires permanently on a healthy store. It compared the interval's summed checkpoint sync time against a per-checkpoint bar, so a store whose checkpoints each synced in a few seconds breached every hour. It now judges the average sync per checkpoint (a WAL-forced checkpoint still fires it), and the alert and get_store_metrics report the checkpoint count and the average. Adds schema rung V140 (store_metrics.checkpoints_timed). (Store Checkpointer Pressure compares an hourly sync sum to a per-checkpoint bar, so it fires permanently on healthy stores #4037)