Skip to content

Every service restart re-announces the daily digest and sweep rollup: the once-a-day gate lives in process memory, so it asks 'did I send' when the channel asks 'was it delivered' #3580

Closed

Description

@erikdarlingdata

Summary

Every service restart re-delivers the day's collector-cost digest and fleet-sweep rollup, because their once-a-day gates live in process memory. On the v3.8.0 install night, three stores restarting once each produced six re-announcements among ~23 overnight channel posts — a quarter of the channel was the same two documents, twice.

Mechanism

DarlingSelfAlertEvaluator.EvaluateFleetSweepRollupAsync gates on _lastSweepRollup — a ConcurrentDictionary<string, DateTime> under one fixed key, explicitly "the _lastCostDigest idiom" — and compares now - lastSent < FleetSweepRollupInterval (1 day). The digest gates the same way. A fresh process has an empty dictionary, so the first tick after any start delivers both, whatever was delivered an hour earlier by the previous process. The gate answers "has THIS PROCESS sent one today", and the reader's question is "has one been delivered today".

The case that must survive

The install night also showed the opposite case working correctly: a digest/rollup pair whose delivery FAILED (a delivery fault, not a suppression) was re-attempted after the restart and landed — the restart was the recovery. Any fix must keep that: a failed delivery is not a delivery, and the next opportunity should retry it.

Fix shape (the monitoring seat's design, agreed)

Gate on delivered-today, recorded in the store, not fired-today in memory: on a SUCCESSFUL delivery, persist the delivery stamp under the document's fixed key (the self-alert state tables the fleet-sweep rungs introduced are the natural home — no new rung if an existing key/value state table fits; if a rung is needed, it queues behind the current ladder); the gate reads that stamp and skips when it is inside the interval. A failed delivery writes nothing, so the next tick — restart or not — retries. The in-memory dictionary can remain as a same-process fast path, but it is no longer the authority.

Secondary consideration for the lane: the first tick after a restart is also when a store's alert-read cache is coldest; the re-announce shares that instant with the collection-health read excursions that another lane is examining. Not a dependency — a note that "first tick after restart" is doing a lot of things at once.

Evidence: the dogfood channel's overnight post census (monitoring seat), three stores, one restart each, 2 documents × 3 = 6 re-announcements; the digest's and rollup's gate code as cited.

Activity

  1. erikdarlingdata commented on Sep 18, 2026

    @erikdarlingdata
    OwnerAuthor

    Closed by the merge of #3633 (its body names this issue with a closing keyword; keywords do not fire against dev, so the watcher closes it).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions