Skip to content

[FEATURE] Let a store's self-alerts name the store they came from, instead of the constant "Monitor Store" #3500

Closed

Description

@gotqn

Which component(s) does this affect?

  • Lite
  • Darling
  • Documentation

Problem Statement

Every self-alert a Darling store emits carries the same server label:

internal const string StoreServerLabel = "Monitor Store";

A constant, on all twelve of them - Retention Held, Store Disk Pressure,
Store Disk Pressure Resolved, Store Job Over Cadence, Store Runtime Upgrade,
Compression Job Stuck, Collector Cost Digest, Stale Mute Rules, Stale Mute Rules Cleared,
Custom Alert Rules Unhealthy, Custom Alert Rules Recovered, Fleet Sweep Rollup.

That is unambiguous with exactly one store, and the product does not assume there is only one. The
peers block in darling.json exists precisely to describe a multi-store estate - thisStoreCovers
is documented as "one sentence naming what THIS store monitors", and stores lists the siblings.
list_servers goes further and warns that a server missing from its list is not proof nobody
monitors it, because another store may. So multi-store is a supported shape. Self-alerts are the one
class of alert that cannot say which host they are about.

Two consequences, and the second is the one that bites.

On a shared channel, the alert is unattributable. We are about to run one store per data centre,
both feeding one Azure Logic App that fans out to Teams. A Store Disk Pressure arriving on the
monitoring-health channel says Monitor Store and nothing else. The operator has to go and check
both hosts to find out which one is filling up.

Downstream deduplication folds the two hosts together. AlertFingerprint hashes
(serverName, incidentType, objects). With serverName the same constant on every store, the same
self-alert raised on two different stores produces the same fingerprint. Our Logic App creates
and dedups one Azure DevOps work item per fingerprint, so the second store's disk pressure does not
open its own ticket - it lands on the first store's as a recurrence. The alert is not merely
ambiguous; one of the two hosts is silently lost.

A transport-level workaround does not fix this. We can distinguish the source by giving each store a
different query parameter on the Logic App's trigger URL, and that is enough to label the Teams card
correctly - but the fingerprint is computed inside the product, so the work-item collision remains.

Proposed Solution

Derive the self-alert server label from the store's own identity rather than from a constant, using
something that already exists rather than adding a new knob:

  • Reuse peers.thisStoreCovers when it is set, or add a short peers.storeName beside it for a
    label rather than a sentence - the peers block is already the "which store am I" block, is
    file-only, and is already documented as deployment topology that a peer's Viewer must not edit.
  • Fall back to the machine host name when nothing is declared, which is unambiguous without
    configuration and is what an operator would write anyway.

Make it opt-in, with the current constant as the default. Changing the label changes the
fingerprint, and a changed fingerprint orphans every open work item keyed on the old one and starts
creating duplicates beside them - silently. Existing single-store installs must see no change unless
they ask for it. An operator who sets the new field is accepting the re-key knowingly, and a line in
the release notes saying so is enough.

Minimal viable fallback: emit the store label as an additional field on the alert payload,
leaving the Server value and the fingerprint untouched. That solves the display problem for
everyone and leaves the deduplication problem to whoever opts into the full change.

Use Case

We are standing up a second Darling installation, one per data center, both delivering to a single
Logic App, a single Teams team and a single Azure DevOps board. Everything else about that topology
works: monitored servers have distinct display names, so their fingerprints are distinct and the
board deduplicates correctly across both installations. The stores themselves are the only thing that
collides, and they collide on every one of their twelve self-alert families.

With peers.storeName set to the host name on each store, a Store Disk Pressure names the host in
the Teams card and opens its own work item - the same behavior every monitored server already gets.

Alternatives Considered

  • Deriving the host from {{triage_url}} - the generic webhook offers fourteen tokens
    (metric, server, value, threshold, severity, context, timestamp, context_json,
    incidents_json, dedup_key, resource_name, database, triage_url, detail) and
    triage_url is the only one that names the host, because it is built from web.publicBaseUrl.
    Three problems: it is empty whenever no valid base URL is configured, so it cannot be relied on;
    parsing an identity out of a URL is fragile; and it does not touch the fingerprint, so the work
    items still merge.
  • A query parameter per store on the webhook URL - labels the Teams card correctly, but the
    fingerprint is computed inside the product, so the work items still merge.
  • One Logic App per store - duplicates the routing rules, the card format and the work-item logic,
    which then drift apart. It also does not help: the two stores would still write colliding
    fingerprints onto the same board.
  • A separate Azure DevOps project per store - isolates the collision rather than fixing it, and
    gives up the single board that is the point of centralising.

Additional Context

No response

Activity

  1. erikdarlingdata commented on Sep 17, 2026

    @erikdarlingdata
    Owner

    Accepted — and the fingerprint-collision analysis is exactly right, including the part most requests miss: that the fix's own deployment is the hazard, since a changed label re-keys every open work item downstream. The shape being built follows your proposal with one deliberate deviation, resolved in favor of your own bolded requirement: peers.storeName (a short label beside thisStoreCovers, file-only like its siblings), used by all twelve self-alert families when set — flowing into the fingerprint, history, and delivery exactly as a monitored server's display name does, so two stores produce distinct fingerprints and your board opens two work items — and the constant, byte-identical to today, when unset. The machine-hostname fallback is NOT taken: it would re-key every existing install on upgrade without anyone asking for it, which is precisely the silent orphaning you called out. Setting the field is the opt-in; the field's own documentation carries the re-key warning, plus one coupling worth knowing at deploy time: self-alert mute rules match the alert row's server label, so an opted-in store's existing self-alert mutes need re-targeting to the new name. The display-only extra-field fallback you offered is deliberately deferred — the full change subsumes its purpose. It lands in the cycle after the release currently cutting.

  2. erikdarlingdata commented on Sep 17, 2026

    @erikdarlingdata
    Owner

    Closed by #3506 (merged to dev).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions