Repository navigation
A store's self-alerts can name the store they came from, opt-in via peers.storeName (#3500) - #3506
Merged
Merged
Conversation
erikdarlingdata
enabled auto-merge (squash)
September 17, 2026 12:31
…eers.storeName (#3500) Every fleet-level self-alert a Darling store emits -- Store Disk Pressure, Retention Held, the Collector Cost Digest, the Fleet Sweep Rollup, all twelve families and their resolution edges -- fired under the constant "Monitor Store". Unambiguous with exactly one store, and the product does not assume there is only one: the peers block exists precisely to describe a multi-store estate. The reporter is standing up one store per data centre, both feeding one Logic App, one Teams channel and one Azure DevOps board, and named both halves of the cost with precision: on the shared channel the alert cannot say which host it is about, and -- the half that bites -- the delivery fingerprint downstream automation dedups on is computed inside the product from the same constant identity on both stores, so the second store's disk pressure never opens its own work item; it lands on the first store's as a recurrence. One host's incident is silently lost, and no transport-level workaround can reach a fingerprint computed inside the product. The new file-only peers.storeName -- a short LABEL beside thisStoreCovers, which stays the sentence it is -- opts a store in. Set, it becomes the server label on every self-alert family through ONE seam: EffectiveStoreLabel resolves the configured name once at construction (the peers block is restart-only, so a Func here would claim a liveness the config cannot deliver), every fire site reads _storeLabel, and StoreKey() prefixes the label onto the family key beside it. The key half is load-bearing rather than cosmetic: a no-incident alert's dedup identity is {serverKey}:{metric} (DerivePagerDutyDedupKey's fallback), and the server NAME is not in it -- relabel the card without qualifying the key and two stores' work items keep colliding, the invisible half of the bug. The family key stays inside the qualified form because the per-object families (retention, compression, job cadence) rely on it to keep per-object cooldowns apart, and history storage is untouched either way: every non-numeric key already collapses into the write-only server_id 0 bucket (#3456), so qualifying re-keys nothing but the fingerprint -- which is the re-key the operator accepted by setting the field. Unset is byte-identical to today, pinned by comparing fired outcomes as whole records. That promise is also why the reporter's suggested machine-hostname fallback is deliberately NOT taken: it contradicts their own (correct, bolded) opt-in requirement by performing the fingerprint re-key on every existing install at upgrade, silently -- the exact harm they name. Only setting the field opts in, and a storeName spelled exactly as the constant resolves TO the constant: same bytes, no re-key, rather than an opt-in that changes nothing but the key shape. The couplings ride along rather than being discovered later. Mute rules match the alert row's server spelling, so an opted-in store's self-alert mutes must target the label -- documented at the field, in the sample, the README, the MCP tool descriptions and the CHANGELOG, and held by tests in both directions on the stale-mute alert's explicit-rule scan (the one mute path that does not go through the shared seam) plus a capture at the shared seam itself. The triage endpoint recognises BOTH spellings as fleet-level -- the label is by design not in the registry, and channels still hold links minted before the opt-in -- read through the ambient DarlingPeerDirectory publish the peers block already has, rather than a second config load. The label gets the peers block's credential-shaped-text refusal too, unconditional and before any early return, because it rides out on every delivery channel, a wider broadcast than the MCP-only siblings. And storeName alone does NOT summon the Fleet Coverage disclosure: it names the store on its own alerts, it does not declare a fleet split, so IsEmpty deliberately ignores it. A label that shadows a monitored server's display name is a stated non-goal to police: the registry is store-authoritative after seeding, so config validation cannot see it -- the field's comment and the docs say not to reuse one, and what it would cost (the two become indistinguishable on shared surfaces, including the triage page's fleet-level read). The resolution prose that hardcoded "Monitor Store:" now speaks through the label for the same reason the rows do, which is also what lets the census hold: the literal appears in exactly one string in the evaluator (the const), every fire site flows through the seam (held from source, so a thirteenth family cannot quietly hardcode the constant back in), and everywhere the label travels as a server name, the key beside it goes through StoreKey(). The reporter's "minimal viable fallback" (a display-only payload field leaving fingerprints untouched) is deliberately not in this change: the full opt-in subsumes its purpose, and a fifteenth webhook token has census reach across both apps' Settings help text. If the display-only field is still wanted beside the opt-in, it is its own small follow-up. Credit to the report: the estate shape, the fingerprint mechanics, the enumeration of the twelve families and the opt-in requirement arrived already worked out. The issue was the design document. Closes #3500.
erikdarlingdata
force-pushed
the
feat/3500-store-self-alert-label
branch
from
September 17, 2026 13:54
2e79cb0 to
f6fb3ba
Compare
This was referenced Sep 17, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #3500 — ceremonially; the watcher closes it at merge.
The request, and the discovery that reshaped it
A multi-store estate's self-alerts all say "Monitor Store" — unattributable on a shared channel, and colliding downstream where dedup keys open one work item for two hosts. The external report proposed the fix shape (an opt-in label in the peers block, constant-by-default because a changed label re-keys every open work item) and its analysis was right about everything except one mechanism — and that one mattered: for the twelve self-alert families, the delivery fingerprint is not
AlertFingerprintat all. With no incidents attached, the dedup key falls back to{serverKey}:{metric}, and the server name is not in it. Relabeling the name alone — the literal reading of the request — would have fixed the Teams card and left both stores' work items still colliding onstore:Store Disk Pressure, an apparent fix hiding the original defect.The seam
One construction-time label (
peers.storeName, trimmed; blank is not an opt-in; a storeName spelled exactly "Monitor Store" resolves to the constant so a no-op opt-in cannot re-key anything), one key qualifier (StoreKeyprefixes the label onto the family's dedup key —dc1:store— when and only when opted in), and every one of the nineteen fire and resolution sites across the twelve families speaking through both. The qualification is safe because the non-numeric-key bucketing already collapses these families' history writes to the store's own row — only the delivery fingerprint re-keys, which is precisely the re-key the operator accepted by setting the field. Unset renders byte-identical to today, pinned by whole-outcome-record equality.The reporter's hostname fallback is deliberately not taken — it would re-key every existing install on upgrade, the exact silent orphaning their own opt-in requirement forbids — and the display-only payload field is deliberately deferred, since the full change subsumes it and a fifteenth webhook token has census reach.
The couplings, handled rather than footnoted
Mute rules match the alert row's label, so an opted-in store's self-alert mutes need re-targeting: documented at the field, in the sample, and in both alert-settings tool descriptions — and the stale-mute scanner itself now matches on the label. The triage endpoint recognizes both spellings, so links minted before the opt-in keep resolving. The peers validation refuses credential-shaped tokens in the label unconditionally, because it broadcasts to every delivery channel. Setting the label does not summon the fleet-coverage disclosure — a name is not a topology statement.
Tests
Thirteen across three suites, the load-bearing ones being: the unset arm's byte-identical whole-record pin (the opt-in promise), the two-stores-distinct-fingerprints arm (the reporter's acceptance, verbatim), the mute-rule coupling in both directions, and the from-source census that every fleet-level fire site flows through the one seam — so a thirteenth self-alert family cannot quietly hardcode the constant back. Tests cannot run on macOS; verified by full-solution build and static census replication. CI is the first execution, review-dark per the known window.