You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Custom alerts: surface broken/never-firing rules as service self-health #3304
Part of the custom-alerting feature (#3285). Follow-up slice from the design plan (Component 4 step 0 + Round-2 hardening).
Problem
A stored rule's spec is re-parsed against the live MeasureCatalog. If a measure key drifts (e.g. during the incremental pg_* buildout), the rule silently stops compiling. Nobody ever "opens" an alert rule the way they open a notebook, so a silent parse failure = an alert the operator believes is armed that never fires. Monitoring-integrity gap.
Fix
When CustomAlertEvaluator loads/refreshes rules, re-parse every ENABLED rule against the live catalog.
A rule that no longer compiles is surfaced as a service self-health alert via DarlingSelfAlertEvaluator - which is a closed set today, so this needs a new condition + AlertSeverityLevel arm + a fleet-global (not per-server) home.
Aggregate MANY broken rules into ONE self-alert (a pg_* rename could break many at once).
Also flag "armed but perpetually no-data for N sweeps": an always-NULL column (e.g. acu_utilization_percent on provisioned instances) or a 0-server tag scope both PASS the compile-check yet never fire.
Alert text = rule id/name + the catalog error, NEVER the compiled SQL.
Tests
Unit-cover: a drifted rule surfaces; N broken rules aggregate to one; the always-NULL / 0-server "armed but never fires" case is flagged. Run the FULL Darling.Tests + Lite.Tests suites (no -class filter) before the PR.
Addressed by PR #3311 (merged to dev; all 8 CI legs green, reviewed). Broken custom rules (re-parsed against the live catalog each health check) and 'armed but never fires' rules (0-server scope, or no data across every in-scope server for >20 min) now surface as ONE aggregated fleet-global service self-health alert via DarlingSelfAlertEvaluator (edge-triggered like disk pressure; text carries rule id/name + catalog error, never the compiled SQL; sanitized against the mute-pre-fill spoof). Fail-safe: a rule-load failure HOLDS the standing alert rather than resolving it. Ships in the next release.
Activity
Merge pull request #3311 from erikdarlingdata/feature/3304-custom-ale…
erikdarlingdata commentedon Sep 11, 2026
Addressed by PR #3311 (merged to dev; all 8 CI legs green, reviewed). Broken custom rules (re-parsed against the live catalog each health check) and 'armed but never fires' rules (0-server scope, or no data across every in-scope server for >20 min) now surface as ONE aggregated fleet-global service self-health alert via DarlingSelfAlertEvaluator (edge-triggered like disk pressure; text carries rule id/name + catalog error, never the compiled SQL; sanitized against the mute-pre-fill spoof). Fail-safe: a rule-load failure HOLDS the standing alert rather than resolving it. Ships in the next release.
Surface broken / never-firing custom alert rules as self-health (erik…
Add the custom-alert feature CHANGELOG entry (erikdarlingdata#3285)
Pin that every self-alert condition is reached, and on the right gate
Add tag scope to custom alert rules (erikdarlingdata#3350)