get_fleet_overview bands healthy servers Offline when query_store stretches the sweep past 15 minutes

[erikdarlingdata]

Summary

get_fleet_overview intermittently bands healthy, actively-collecting servers as Offline - no recent collection. The servers are not offline and nothing is failing. query_store's multi-minute runs stretch the per-server sweep interval past the hardcoded 15-minute OfflineThreshold, so the freshness classifier calls a server dark while it is mid-cycle.

This is a display-only defect — the alert engine uses a different, more conservative window and did not fire. But it is actively misleading: it is what sent an investigation chasing five phantom-dark production shards today.

Evidence

Server names are synthetic slugs; the same slug always denotes the same server.

At 20:49Z on 2026-09-02, the use1 store reported 5 offline servers. At 21:07Z, with no intervention, it reported 0.

server	20:49Z band	20:49Z last_collection	21:07Z band	21:07Z last_collection
shard-alpha	Offline	20:31:40	Healthy	21:05:24
shard-beta	Offline	20:32:32	Healthy	21:06:17
shard-gamma	Offline	20:34:30	Healthy	21:06:06
shard-delta	Offline	20:34:20	Healthy	21:07:40
shard-epsilon	Offline	20:34:17	Warning	20:58:04

Staleness at the 20:49Z snapshot was 15.4–18.2 min for the five. The next-oldest server was 13.9 min stale and banded Critical, not Offline — bracketing OfflineThreshold (15 min) precisely. Nothing else distinguishes the five.

Mechanism

On shard-alpha, over 18:14–21:05, every gap longer than 5 minutes is bounded by a query_store run — 11 of 12 begin the moment query_store completes:

GAP  19.3 min   18:56:19 (query_store, dur=144201ms) -> 19:15:38
GAP  18.9 min   20:31:40 (query_store, dur=  4157ms) -> 20:50:33
GAP  16.5 min   19:53:40 (query_store, dur= 14420ms) -> 20:10:10
GAP  16.3 min   18:36:37 (plan_correction)           -> 18:52:57
GAP  15.2 min   19:20:20 (query_store, dur= 56531ms) -> 19:35:33

query_store durations on that server in the window ranged 4,157 ms to 182,222 ms. The sweep does not relaunch promptly after a long query_store cycle, so no collector — including the one-minute ones — records for 6.6–19.3 minutes.

That directly falsifies the premise the band structure rests on, in ServerHealthThresholds:

The fastest scheduled collector's cadence (wait_stats / cpu_utilization / memory_stats etc. all run every minute), so MAX(collection_time) tracks a one-minute rhythm on a healthy server. Freshness bands are multiples of this.

A healthy server here goes 19 minutes. And OfflineThreshold is not a multiple of the cadence — it is a bare 15-minute constant:

public static readonly TimeSpan OfflineThreshold = TimeSpan.FromMinutes(15);

ClassifyFreshness bands purely on age > OfflineThreshold, with no signal for whether a collection is in flight or whether any collector has actually failed.

Blast radius: display only

The self-alert path uses a different and more conservative threshold — DarlingSelfAlertEvaluator.StaleWindow = 30 minutes, configurable via CollectionStaleMinutes. Since the observed gaps peaked at 19.3 min, no alert should have fired, and none did: get_alert_history over 6 hours returns 20 alerts, all deadlocks / CPU / collector-cost regressions, and zero Collection Stopped.

So one condition has two definitions that disagree — the display calls a server dark at 15 minutes while the alert engine has deliberately decided it is not stopped until 30, and only the alert side is configurable.

Prior form

[#2699] fixed this same symptom from a different cause (an unbounded GROUP BY server_id plus a default(DateTime) fallthrough making live servers read as ancient). shard-beta appears in that CHANGELOG's list of affected servers and in today's five. The 48-hour bound and null propagation from #2699 are working correctly — these last_collection values are real, not artifacts. This is a genuine 15-minute gap being misclassified, which #2699 did not address.

Recommended fix — and why it should wait for a measurement

The upstream cause is query_store duration. [#2792] is merged but not yet deployed, and takes the plan fetch from ~60,000 ms CPU to 0.508 s by forcing a hash join where a fixed TVF cardinality estimate was driving QUERY_STORE_PLAN_IN_MEM to be re-executed once per candidate plan_id. That is the thing stretching these sweeps.

Changing OfflineThreshold now and deploying #2792 means we cannot tell which removed the symptom. Deploy #2792 first, then re-measure the sweep gaps. If they no longer cross 15 minutes, no classifier change is warranted.

If gaps still cross 15 minutes after #2792, the principled fix is not to raise a second hardcoded constant but to make the display's Offline decision agree with the alert engine's configurable CollectionStaleMinutes, so one condition has one definition. ServerHealthThresholds exists precisely to stop this class of drift ("Freshness thresholds ... used to live twice at numerically equal but independently-editable values — a drift risk", #1562); the alert engine's window was never folded in.

Separately worth considering regardless: Offline is a strong claim — the enum documents it as "nothing has been collected for long enough to call the server dark" and it renders a red overlay. A server that is merely between sweeps has a band already: Stale.

[erikdarlingdata]

Mechanism confirmed from the app log (was inferred above, now direct)

The service logs the relaunch skip explicitly:

2026-09-02 21:08:40.788 [WARN] [DarlingWorker] [shard-nu] collection body has not completed after 63s of execution - skipping relaunch
2026-09-02 21:11:10.861 [WARN] [DarlingWorker] [shard-xi] collection body has not completed after 73s of execution - skipping relaunch
2026-09-02 21:11:10.862 [WARN] [DarlingWorker] [shard-omicron] collection body has not completed after 62s of execution - skipping relaunch

So the sweep is on a ~60 s cadence and declines to relaunch while the previous body is still running. When query_store runs long — up to its PerDatabaseWallClockBudget of 10 minutes across a server's databases — the body stays busy, no new collector starts, and nothing writes to collection_log. Collectors that already finished have logged; the remainder of the body is one slow collector. That produces exactly the observed 15–19 minute holes with no failure recorded anywhere.

skipping relaunch mentions per server today on use1 (top 10):

52 shard-iota   50 shard-theta   50 shard-mu   48 shard-zeta   45 omega-01
44 shard-epsilon   44 shard-delta   43 shard-eta   43 shard-lambda   42 shard-kappa

Note this list is not the same as the five that read Offline at 20:49Z — shard-iota, shard-theta and shard-mu skip relaunches more often than shard-delta/shard-epsilon and were not banded Offline. That is expected and worth stating, because it rules out "these five are special": the band depends on where the snapshot happens to land relative to each server's current hole, which is why the set rotates. It is an oscillating condition sampled at an instant, not a stable property of five servers — the same rotating-set behaviour [#2699] described for its own different cause.

[erikdarlingdata]

Fixed by #2885, merged to dev as dfff4080e67aa89b9d92fc7e13c9f699239de145.

What changed. OfflineThreshold was a bare TimeSpan.FromMinutes(15) standing on its own, while the alert engine had already decided collection is not "stopped" until 30 minutes. One condition, two definitions. It now derives from a single shared constant, ServerHealthThresholds.CollectionStoppedMinutesDefault = 30 in PerformanceMonitor.Common/ServerHealthBands.cs, which is also what every other spelling of the same claim now reads:

• DarlingSelfAlertEvaluator.StaleWindow
• AlertsConfig.CollectionStaleMinutes's default (DarlingConfig.cs)
• DarlingWorker.PgLongRunningQueryRecencyMinutes
• Lite's AppAlertEngineSettings.CollectionStaleMinutes
• Lite's AgentStatusRow.StaleWindow (LocalDataService.JobHistory.cs), whose comment already claimed to "mirror" the service's window from a numerically-equal but independently-editable copy

Effect on the case in this report. A server between stretched sweeps now bands Stale — amber, "collection has lagged" — instead of Offline red. Every surface follows, because the banding reads all resolve through the one ServerHealthClassifier.ClassifyFreshness (#2473). Genuinely dark servers run hours, so they still band Offline.

One asymmetry worth knowing. collectionStaleMinutes still widens the alert window per deployment; the display band stays pinned at the shared default, because the display has no live settings on every surface. That is the conservative direction — the band can only be tighter than the alert, never looser.

CollectionStoppedThresholdAgreementTests pins the definitions together by value on both apps, and was proven red against the pre-change tree (15m vs 30m, and the measured stretch banding Offline).

Not addressed here, deliberately. This widens the window; it does not add an in-flight signal. ClassifyFreshness still cannot distinguish "quiet because the sweep body is mid-cycle" from "quiet because collection died" — it only bands on age. If sweep stretch ever legitimately exceeds 30 minutes, this returns as the same defect at a larger constant. The durable fix for that is a signal for whether a collection is actually in flight, which is not in this change.