Custom alert rules: an alert no channel delivered is sent again while its condition holds (#4795)

[erikdarlingdata]

Part of #4795 (custom alert rules; the PostgreSQL alerts follow separately).

Why

A custom alert rule has no cooldown. It fires once, on the rising edge, and the evaluator saved the incident as open and delivered whether or not the send worked. When every channel failed (a webhook answering HTTP 429 or 5xx, an unreachable mail server) the alert was lost until the condition cleared and came back. The resolve still went out, because it only checks that a severity was recorded, so an operator could get a "resolved" for an alert that never arrived. A severity change on an open incident used the same send and had the same gap.

What changes

All in Darling/PerformanceMonitor.Darling.Service/CustomAlertEvaluator.cs. No schema change, and FailedSendBackoff, AlertEngine, the persistence gate, ClassifySeverityChange, the saved-state record and the resolve message are untouched.

• DeliverFireAsync sends through DeliverAndReportAsync and returns the AlertDelivery, where it used to send through DeliverAsync and return nothing.
• The evaluator holds one FailedSendBackoff (family = the rule's Custom:<id> metric name, key = the server id as invariant text) and an in-memory table of when each (rule, server) may be sent again. FailedSendRetryCap is 15 minutes: a rule has no cooldown, so it is the cap on the 1, 2, 4 ... minute wait.
• Rising edge: when every channel failed, FiredSeverity stays null (the incident is open but not delivered), the failure is counted, the next attempt is set 1 minute out (then 2, 4, up to 15), and one Information line is logged with the rule id, server, failure count and wait. Otherwise the severity is recorded as before and the failure streak ends.
• While the condition holds on an open incident with no delivered severity (a failed send, or an incident saved before a restart), the fire is sent again at the severity the value has now, once its wait is over. After a restart nothing is recorded, so the first pass sends. That pass does not also run the severity-change branch.
• A severity change on a delivered incident that no channel took keeps FiredSeverity at the old band, so the next pass still sees the change, and it is sent again only when due. On success the new band is recorded and the streak ends.
• Resolve: still sent only when FiredSeverity is set, so an incident whose send failed gets no resolve. It also ends the failure streak and drops the wait, so a later incident starts again at 1 minute.
• A muted or throttled fire, a partial failure (one channel delivered) and a null report all read as delivered, as before.
• The decision half of one evaluation (persistence gate, send, state to save) moved into an internal ApplyValueAsync, so it can be driven pass by pass without a database. EvaluateRuleForServerAsync still loads the state, reads the value, calls it and saves what it returns. That move is its own commit and changes no behavior.

Test plan

• New CustomAlertFailedSendRetryTests (10 tests, no database): every channel fails on the rising edge (no severity saved, sent again after 1 minute, then 2); a delivered retry records the severity and the next pass sends nothing; the wait doubles 1, 2, 4, 8 and stops at 15; no resolve when the condition clears after a failed send, and the next rising edge starts again at 1 minute; a resolve does go out for an incident whose retry was delivered; a restart (a new evaluator over the saved firing state with no delivered severity) sends on its first pass, at the current severity; a failed severity change keeps the old band and is sent again when due; controls for a partial failure and a muted fire (severity recorded at once, never sent again).
• RED first: the tests and the behavior-preserving move were committed together, and run against that old switch body: 7 failed, 3 passed (the 3 are the partial-failure and muted controls and the resolve-after-a-delivered-retry test, which hold before and after). After the fix: 10 of 10 pass. The tests call the moved method, so they could not run against the literal pre-move file; the old switch body inside the move is that file's logic.
• Build of Darling.Tests: 0 warnings, 0 errors.
• Touched and census classes together (AlertMasterSwitchSurfaceTests, AlertReadFailureSurfaceTests, the CustomAlert* classes that run without a database, and the new class): 112 passed, 0 failed.
• Full Darling.Tests suite without DARLING_TEST_PG, after merging origin/dev (already up to date): Total 17587, Failed 0, Skipped 1145 (live classes that need a database), Not Run 1. The first full run had one failure, FleetIdentifierScrubTests.NoTrackedSourceFileNamesAServerByShortNameAndOrdinal, because the new tests named a server sql-retry-01; they now use alpha-01, and the second full run is clean.
• The live custom-alert classes (for example CustomAlertSeverityEscalationLiveTests) were not run: no PostgreSQL was available. They go through EvaluateRuleForServerAsync, which now calls ApplyValueAsync.
• Nothing in the PostgreSQL alert families is changed here; they follow in a separate PR.

Worth a second look

• The retry table is keyed (long RuleId, int ServerId), because CustomAlertRule.Id is a long.
• A retry is sent on the first pass at or after its time. Passes follow the rule's own cadence (30 seconds at the least), so a retry can land up to one cadence after the wait ends.
• A restart forgets the streak, so the first retry after a restart comes as early as a fresh failure would. An incident saved as firing with no delivered severity is sent on the first due pass after any restart.
• Disabling, deleting or re-scoping a rule does not clear its streak or wait. Nothing reads them before the next fire overwrites or drops them (only an incident that has failed since can consult the wait), so this is harmless, and the table stays as small as the number of (rule, server) pairs that ever failed.

CHANGELOG

SECTION: Fixed
ENTRY:

• A custom alert rule whose alert no channel delivered is sent again while its condition holds ([Custom alert rules: an alert no channel delivered is sent again while its condition holds (#4795) #4827]) - Before, the rule counted the alert as sent and stayed quiet until the condition cleared and came back. Now it tries again after 1 minute, then 2, 4 and so on, up to 15 minutes, and a resolve goes out only for an alert that was delivered. A severity change that no channel delivered is sent again the same way.
  REF:
  [Custom alert rules: an alert no channel delivered is sent again while its condition holds (#4795) #4827]: Custom alert rules: an alert no channel delivered is sent again while its condition holds (#4795) #4827