A store with no readable log directory keeps its hourly collector-cost flush

[erikdarlingdata]

Closes #3971. Closes #3923.

Why

DarlingWorker.SweepStoreSelfMetricsAsync runs three passes on one connection every hour: the sizing pass
(StoreSelfMetrics.SweepAsync), the store-log capture (StoreLogSweep.SweepAsync), then the store-log
re-mask and the collector-cost flush together. On a store with no readable server-log directory — the Linux
compose store's shape, since the timescale/timescaledb image logs to stderr and carries no log directory
at all — the capture fails with 58P01, and Npgsql closes the connection outright rather than leaving it
idle-but-usable. The re-mask and the collector-cost flush then run on a dead connection and fail with
"Connection is not open," so get_collector_cost never gets a row on such a store, however long it runs.

Separately, the sizing pass shared the tick's OUTER catch with everything below it. #3918 hit this for one
specific statement (a catch-all census failing 42703 on TimescaleDB 2.29+): fixing that SQL left the next
statement a store permanently rejects (catalog drift on a bring-your-own TimescaleDB, a revoked privilege)
able to take the store-log census and the collector-cost flush down with it all over again.

What changes

Darling/PerformanceMonitor.Darling.Service/DarlingWorker.cs:

• The sizing pass gets its own narrow catch (Store self-metrics tick: a statement the store rejects in the sizing pass still skips the store-log census and collector-cost flush (#3918 follow-up) #3923), mirroring the store-log capture's catch beside it:
  catch (PostgresException ex) when (!PgBaselineProvider.IsCommandTimeout(ex) && connection.State == ConnectionState.Open). A command timeout and a broken connection still fall through to the outer catch
  unchanged; a server-rejected statement now leaves the tick's other two series alone.
• After the store-log capture's own catch, the connection is reopened if the capture closed it, before the
  re-mask and the collector-cost flush run (A store with no readable log directory loses the hourly collector-cost flush: the failed store-log capture leaves the shared connection closed #3971). Reopened rather than swapped for a fresh one: it came
  from the pool through _postgres, so OpenAsync pulls a new physical connection under the same object the
  await using already owns and disposes.

Tests:

• StoreLogSelfMonitoringStoreTests.TheSweepRidesTheHourlySelfMetricsTick now also pins the sizing pass's
  new catch by position (between the sizing call and the store-log capture) and by filter (both conditions
  in its when clause), and pins the reopen check's position between the store-log catch and the
  collector-cost flush — so a mutation that removes any of the three, or moves one past where it has to sit,
  reds here.
• AlertReadFailureSurfaceTests: the new sizing-pass catch is a swallowed-read census site, exempted with
  the same reason the outer catch already carries (a metrics write sweep; no alert is judged on its result).
  WorkerExemptSites 8 -> 9, the tree-wide totalExempt pin 27 -> 28, both with the comment naming the 28th
  exemption and its reason.
• New StoreSelfMetricsTickRecoveryLiveTests (own-store, #1776 own-store): against a real scratch
  database, forces the exact capture failure, asserts the connection really does close, reopens it the same
  way the worker does, then runs a real CollectorCostAccumulator.FlushAsync and asserts its row lands.

Lite has no store self-metrics tick (grep -rl "SweepStoreSelfMetricsAsync\|StoreSelfMetrics\|CollectorCostAccumulator\|StoreLogSweep" Lite/ returns nothing), so there is no Lite parity item.

Test plan

• Reproduced the Npgsql connection-close claim against a real PostgreSQL 18.4 / Npgsql 10.0.3 rig: a
  fresh cluster (logging_collector off, the default) has no log directory at all, so
  StoreLogSweep.SweepAsync fails with 58P01: could not open directory "log" and connection.State
  goes straight to Closed — no ALTER SYSTEM needed to force it, it is the rig's out-of-the-box state.
• StoreSelfMetricsTickRecoveryLiveTests proven to fail on the code before this PR: with the reopen
  disabled, the very next assertion reds (Expected: Open, Actual: Closed); with that assertion removed
  too, the run reaches CollectorCostAccumulator.FlushAsync and throws
  System.InvalidOperationException: Connection is not open — the exact class of failure A store with no readable log directory loses the hourly collector-cost flush: the failed store-log capture leaves the shared connection closed #3971 reports.
  Restored, the test passes.
• Targeted classes green throughout iteration: StoreLogSelfMonitoringStoreTests,
  AlertReadFailureSurfaceTests, StoreLogRemaskLiveTests, StoreSelfMetricsTickRecoveryLiveTests,
  LivePostgresCollectionHygieneTests (41 + 12, 0 failures).
• Gate census tests checked directly: DocCommentHygiene, StartupCommandTimeoutTests (14, 0 failures),
  McpPayloadContractCensusTests (67, 0 failures).
• Full Darling.Tests suite run twice against a dedicated throwaway PostgreSQL 18.4 + TimescaleDB rig
  with DARLING_TEST_PG set. First run: 12801 total, 2 failed. Both investigated:
  LivePostgresCollectionHygieneTests.EveryClassUsingTheSharedStore_IsSerializedOrDocumentsWhyNot was a
  real, self-inflicted miss — the new live test file was missing the #1776 own-store exemption comment
  the hygiene census requires for any class reaching the store through ScratchPostgres; fixed by adding
  the marker. CollectionHealthAggregateTests.OutageHole_PolicyRunAfterMissedRuns_JumpsNoGap_AgainstDevPostgres
  is untouched by this PR and passed 13/13 re-run alone, so it was a one-off flake in the full-suite run.
  Second full run after the hygiene fix: 12801 total, 0 failed, 24 skipped (env-gated), 0 errors.
• Installer.Tests and any monitored server (sql2022, sql2025) were not touched, per instructions.

Follow-up commit (coordinator)

• A capture failure the store will never clear (58P01 no log directory, 42501 no log read) warns once per process, then logs at Debug. get_store_log still reports the gap, and any other failure keeps the hourly Warning. Pinned in StoreLogSelfMonitoringStoreTests.
• The reopen closes the connection first, because Npgsql refuses to open one that is Broken rather than Closed.
• Pinned classes (StoreLogSelfMonitoringStoreTests, AlertReadFailureSurfaceTests, StartupCommandTimeoutTests, DocCommentHygiene): 130/130, 0 warnings.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GdmA4ND1wLSqA91ax1m4xv