Skip to content

Custom alerts: surface broken/never-firing rules as service self-health #3304

Closed

Description

@erikdarlingdata

Part of the custom-alerting feature (#3285). Follow-up slice from the design plan (Component 4 step 0 + Round-2 hardening).

Problem

A stored rule's spec is re-parsed against the live MeasureCatalog. If a measure key drifts (e.g. during the incremental pg_* buildout), the rule silently stops compiling. Nobody ever "opens" an alert rule the way they open a notebook, so a silent parse failure = an alert the operator believes is armed that never fires. Monitoring-integrity gap.

Fix

  • When CustomAlertEvaluator loads/refreshes rules, re-parse every ENABLED rule against the live catalog.
  • A rule that no longer compiles is surfaced as a service self-health alert via DarlingSelfAlertEvaluator - which is a closed set today, so this needs a new condition + AlertSeverityLevel arm + a fleet-global (not per-server) home.
  • Aggregate MANY broken rules into ONE self-alert (a pg_* rename could break many at once).
  • Also flag "armed but perpetually no-data for N sweeps": an always-NULL column (e.g. acu_utilization_percent on provisioned instances) or a 0-server tag scope both PASS the compile-check yet never fire.
  • Alert text = rule id/name + the catalog error, NEVER the compiled SQL.

Tests

Unit-cover: a drifted rule surfaces; N broken rules aggregate to one; the always-NULL / 0-server "armed but never fires" case is flagged. Run the FULL Darling.Tests + Lite.Tests suites (no -class filter) before the PR.

Activity

  1. added a commit that references this issue on Sep 11, 2026
  2. erikdarlingdata commented on Sep 11, 2026

    @erikdarlingdata
    OwnerAuthor

    Addressed by PR #3311 (merged to dev; all 8 CI legs green, reviewed). Broken custom rules (re-parsed against the live catalog each health check) and 'armed but never fires' rules (0-server scope, or no data across every in-scope server for >20 min) now surface as ONE aggregated fleet-global service self-health alert via DarlingSelfAlertEvaluator (edge-triggered like disk pressure; text carries rule id/name + catalog error, never the compiled SQL; sanitized against the mute-pre-fill spoof). Fail-safe: a rule-load failure HOLDS the standing alert rather than resolving it. Ships in the next release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions