Retry the alert pass's database-state maintenance once on a chunk-drop deadlock

[erikdarlingdata]

What happened

A deadlock (SQLSTATE 40P01) between a chunk drop and a concurrent statement hit three monitoring stores over 14 days. Two of the three were the TimescaleDB retention job on collection_log losing to a chunk drop (self-heals on the job's own schedule). The third was the alert pass's database-state maintenance losing to a drop_chunks call — the seed's INSERT picked as the deadlock victim — and that side had no retry, so it surfaced as an alert-pass error.

The fix

GetDatabaseStatesCoreAsync (Darling/PerformanceMonitor.Darling.Service/DarlingAlertReadAdapter.cs) runs four statements ahead of the deviation read: seed, heal, prune, clear-recovered. All four now run inside a new ExecuteDatabaseStateMaintenanceWithDeadlockRetryAsync, a delegate-seamed static helper with the same shape as the daily purge's ExecuteDropChunksWithDeadlockRetryAsync (DarlingRetention.cs): one immediate retry on PostgresErrorCodes.DeadlockDetected, logged once at Information (SQLSTATE + server id), give-up-and-surface on a second deadlock in a row, no retry for any other exception.

Why a new, narrower seam instead of widening the shared ExecuteWithOneRetryAsync: that seam serves twelve other reads across this type plus DarlingSelfAlertEvaluator, and its whole design deliberately excludes PostgresException — "an identical second attempt gets an identical answer" is true for most of those callers but not for this one sequence, whose four statements are each independently safe to repeat:

• the seed is INSERT ... ON CONFLICT DO NOTHING;
• the heal is an UPDATE gated on the newest snapshot, idempotent against its own output;
• the prune is a DELETE of rows absent from the newest snapshot, same idempotence;
• clear-recovered is an UPDATE to a fixed NULL.

Widening the shared seam would have put a PostgresException retry in front of eleven callers that have no such guarantee. The narrower wrap keeps the existing command-timeout retry untouched — this new retry sits entirely inside the read that seam wraps, so a 40P01 retried here is never also counted as a command timeout there.

The collection_log retention job's origin

Confirmed: no product code path creates a TimescaleDB retention policy on collection_log. TimescaleSupport.RetentionPolicies and its RawTierCoverage source, and AddRetentionPolicySql's only caller (EnsureRetentionPoliciesAsync's loop), never name collection_log. The only retention collection_log gets from the product is its own drop_chunks purge inside the daily retention sweep (DarlingRetention.cs ~694–712), which already carries the #2143 deadlock retry. So the TimescaleDB policy_retention job that failed against collection_log on two stores wasn't created by current code, and a search of the history for a rung that added one found none. Where it came from on those stores isn't known from the code. That job's failures self-heal on its own schedule, so this PR changes nothing on that side; the job's origin is worth a look on the stores themselves.

Pins

Darling/Darling.Tests/DarlingAlertReadAdapterTests.cs, same style as DarlingRetentionTests' DropChunksRetry_*:

• DatabaseStateMaintenanceRetry_OneDeadlock_RetriesOnce_AndSucceeds
• DatabaseStateMaintenanceRetry_TwoDeadlocks_GivesUpAndSurfaces
• DatabaseStateMaintenanceRetry_NonDeadlockPostgresException_DoesNotRetry
• DatabaseStateMaintenanceRetry_CleanRun_IsSingleShot

All four ran green on macOS via the in-process Darling.Tests.dll runner (no store needed — delegate-seamed like the purge's own pins). RED confirmed by mutation: changing the catch guard from attempt == 1 to false (removing the retry arm) failed DatabaseStateMaintenanceRetry_OneDeadlock_RetriesOnce_AndSucceeds with an unhandled PostgresException (40P01); reverted before the final build.

Also ran green after the fix (post-merge, dev sha 209aca7fe): DarlingAlertReadAdapterTests (11 total, 1 skipped — the live end-to-end feed test, needs DARLING_TEST_PG), DarlingRetentionTests (47, 3 skipped), AlertReadRetrySeamTests (17), AlertReadFailureSurfaceTests (27), DocCommentHygieneTests (77), CommentFilterAdoptionTests (4), McpPayloadContractCensusTests (68), McpServiceParameterDiSeatCensusTests (1). All 0 failures.

Build: dotnet build Darling/Darling.Tests/Darling.Tests.csproj -p:EnableWindowsTargeting=true — 0 warnings, 0 errors, both before and after the dev merge.

Not run: Lite.Tests — this change touches no Lite code (the database-state maintenance sequence and its retry live only in the Postgres-backed DarlingAlertReadAdapter; Lite's DuckDB read has its own path with no drop_chunks/deadlock exposure). Darling.Tests.dll/Lite.Tests.dll net10.0-windows target build-only per house rule; CI decides anything platform-specific — nothing in this diff touches WPF types.

Merge

Final step per house rule: git fetch origin && git merge origin/dev — merged clean, dev sha 209aca7fe ("Plan analysis: rule 38 reads the server's edition and MAXDOP at every entry point (#4601)"). Rebuilt and re-ran the affected classes after the merge (listed above); all green.

Refs #4606.

CHANGELOG

SECTION: Fixed
ENTRY: - A rare deadlock between a chunk drop and the alert pass's database-state check no longer fails that pass ([#4612]) - The alert pass retries its database-state statements once when PostgreSQL picks them as a deadlock victim, the way the daily purge already retries its chunk drop.
REF: [#4612]: #4612