Store checkpointer alert: a requested checkpoint is no longer blamed on max_wal_size alone (#4758)

[erikdarlingdata]

Fixes #4758

Why

The Store Checkpointer Pressure alert fires on any requested checkpoint in an interval, next to a slow average sync. Its text called that checkpoint "WAL-forced" and said the store had written more WAL between checkpoints than max_wal_size allows, which steers the reader toward raising max_wal_size. But PostgreSQL also counts a base backup and a CHECKPOINT statement as requested, and its counters do not record which one happened. Only the log_checkpoints lines do. The alert is INFO and never pages, so the harm was a wrong remedy, not a false page.

The get_store_metrics checkpointer note made the same claim and closed with "The informational Store Checkpointer Pressure self-alert says the same", so both are reworded together. The tool description's requested field ("checkpoints forced by WAL volume reaching max_wal_size") made it a third time.

Diagnosis confirmed in the code: the pressure test is only Requested > 0 (unchanged, no threshold added), and the alert text asserted the WAL cause unconditionally.

Before, for one requested checkpoint with a short average sync (abridged; the test pins the current value and detail):

• current value: 1 WAL-forced checkpoint(s) over 2 checkpoint(s) in 60.0 min
• detail: "...REQUESTED - forced by WAL volume reaching max_wal_size rather than by checkpoint_timeout. A requested checkpoint means the store wrote more WAL between checkpoints than max_wal_size allows..."

After:

• current value: 1 requested checkpoint(s) over 2 checkpoint(s) in 60.0 min
• detail: "...REQUESTED - started by something other than checkpoint_timeout: WAL volume reaching max_wal_size, a base backup, or a CHECKPOINT statement. PostgreSQL's counters do not say which of those started a requested checkpoint; the log_checkpoints lines do. Raising max_wal_size is the remedy only for the first cause: when WAL volume did it, the store wrote more WAL between checkpoints than max_wal_size allows..."

What changes

• DarlingSelfAlertEvaluator.cs: the alert's current value and short message say N requested checkpoint(s) where they said N WAL-forced checkpoint(s). The detail names the three causes. When only the requested arm fired (the average sync is under the bar), it also says the counters do not say which, that log_checkpoints does, and that raising max_wal_size is the remedy only for the first cause. Two doc comments that said one requested checkpoint means the store outran max_wal_size are reworded, and a comment that quoted the old "WAL-forced" text now says requested-checkpoint pressure.
• DarlingMcpStoreMetricsTools.cs: the get_store_metrics checkpointer note says the same in its interval sentence and its pressure sentence, so its closing "the self-alert says the same" is true again. The requested clause in the tool description says the same in one clause.
• Tests: StoreToastAndCheckpointerTests (forced-arm case extended to require "base backup", "CHECKPOINT statement", log_checkpoints and the remedy sentence in the detail, and "requested checkpoint(s)" in the current value; the two restart-control pins and the MCP note pin updated to the new text), a new DarlingMcpStoreMetricsToolsTests pin on the description clause, and one assertion message in the live StoreSelfMetricsTests (the forced CHECKPOINT there is a statement, not WAL volume).

Unchanged on purpose: the firing conditions and threshold string (average sync > 10s per checkpoint, or any requested checkpoint), severity: null (INFO through the per-metric map), the sync-arm text, and the rule that the short message carries no "restart" wording. Comment-only mentions of "WAL-forced" in DarlingStoreMetricsReader.cs, StoreSelfMetrics.cs, PgMigrations.cs (an existing migration) and the README's migration table describe history or the counter and stay. The monitored-target advice, which grades a requested share, is a separate decision and is untouched.

Test plan

• RED first: with only the test edits, StoreToastAndCheckpointerTests failed 4 of 67 (the forced-arm case, the two restart controls that pin the current value, and the MCP note case).
• After the change: StoreToastAndCheckpointerTests, DarlingMcpStoreMetricsToolsTests, CheckpointsTimedRungTests, StoreMetricsSummaryFirstTests, McpToolsListBudgetTests and StoreSelfMetricsTests: 161 tests, 0 failed, 3 skipped.
• The new description pin fails on the old wording (dev's description holds "checkpoints forced by WAL volume") and passes now; DarlingMcpStoreMetricsToolsTests plus StoreToastAndCheckpointerTests after adding it: 110 tests, 0 failed.
• Darling.Tests builds with 0 warnings and 0 errors.
• Full Darling.Tests suite once, after merging origin/dev, without DARLING_TEST_PG: Total 17291, Failed 0, Skipped 1137 (live PostgreSQL classes), Not Run 1. That run was before the description pin was added; the pin's two classes were re-run afterwards (line above).
• Not run: the live PostgreSQL classes (StoreToastAndCheckpointerLivePostgresTests and the live method in StoreSelfMetricsTests); no database was available. The only live-test edit is one assertion message.

Worth a second look

• The get_store_metrics tool description changed by one clause, so anything that snapshots tool descriptions will see it. The tools-list budget test passes.
• The three-cause wording appears in the alert detail, the MCP note and the tool description. The alert detail and the note are asserted for the same key phrases so they cannot drift apart silently.

CHANGELOG

SECTION: Fixed
ENTRY:

• The store checkpointer alert no longer tells you to raise max_wal_size for every requested checkpoint ([The store checkpointer alert blames max_wal_size for any requested checkpoint #4758]) - The informational Store Checkpointer Pressure alert, and the checkpointer note in the MCP store metrics, called any requested checkpoint "WAL-forced" and said the store had written more WAL than max_wal_size allows. PostgreSQL also counts a base backup or a CHECKPOINT statement as requested, and its counters do not say which one happened, so the suggested remedy was sometimes wrong. The alert's text now says "requested checkpoint(s)". The alert detail and the note now list all three causes, and when a requested checkpoint is what fired them, they also point at the log_checkpoints lines that say which and suggest raising max_wal_size only for the WAL cause. The get_store_metrics tool description says the same about its requested field. The alert fires on the same conditions and stays informational.
  REF:
  [The store checkpointer alert blames max_wal_size for any requested checkpoint #4758]: The store checkpointer alert blames max_wal_size for any requested checkpoint #4758