You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Darling: a heavy query_store run stalls the whole per-server collection body (BODY_OVERRUN cascades to real staleness) #2700
`RunDueCollectorsAsync` runs every due collector for a server sequentially, in one `foreach`, inside `ProcessServerSweepAsync` — and the outer launch loop will not relaunch a server's sweep body while the previous one is still running (INV-2, "one body per server").
`query_store` is watermark-driven and its per-run duration is bimodal: mean ~5-35s, but a heavy batch (large plan/text volume) runs 100-230+ seconds. When a heavy run lands on the cycle where every collector's cadence comes due together, `get_collection_health`'s sweep-pressure diagnostic already detects and explains this as `BODY_OVERRUN` — the body cannot finish inside its cadence and its relaunch is skipped.
The consequence is worse than a slow query_store row: because query_store is awaited inline in the same sequential foreach as every other due collector (query_stats, procedure_stats, wait_stats, all 1-minute cadence), and because the whole body can't relaunch until it returns, ALL of that server's collection is blocked for the full 100-230+ seconds — not just query_store's own slot. On observed servers this pushed `last_collection` past the fleet's 15-minute Offline threshold, producing a real (not classifier-bug) "Offline" band on an otherwise fully healthy server with zero actual collector failures.
Observed today (2026-08-30, use1)
11 servers hit this in one hourly check (multi-03, rho-01, multi-14, multi-16, multi-25, multi-27, multi-37, multi-43, multi-44, multi-48, phi-01), each confirmed via `get_collection_health`: `peak_cycle_risk: BODY_OVERRUN`, `peak_collector: query_store`, peak run 83,903-232,815 ms against a 60,000 ms sweep budget, `failed_collector_count: 0` on all of them (this is a scheduling overrun, not a collector failure). Root cause of the heavy query_store batches is a known upstream application issue (application-side decimal-parameter-type instability generating excess distinct query_ids), tracked separately — but PerformanceMonitor itself should not let one collector's slow run stall a server's entire collection body regardless of why that collector is slow.
Fix
Split `query_store` off the sequential body: fire it detached rather than awaiting it inline in `RunDueCollectorsAsync`, reusing the existing per-server `QueryStoreServerGate` (#2165) for mutual exclusion so a still-running previous tick skips rather than overlapping. Every other due collector in the foreach keeps running on schedule regardless of how long query_store takes, and the body itself returns promptly so the outer launch loop keeps relaunching normally.
Activity
erikdarlingdata commentedon Sep 1, 2026
Fixed by #2701 (query_store split off the sequential per-server body). Merged to
dev; ships in the next release.Darling: split query_store off the sequential per-server collection b…
Pin the sweep-body detach policy and the invariant that makes it safe…