Skip to content

Darling: a heavy query_store run stalls the whole per-server collection body (BODY_OVERRUN cascades to real staleness) #2700

Closed

Description

@erikdarlingdata

What's happening

`RunDueCollectorsAsync` runs every due collector for a server sequentially, in one `foreach`, inside `ProcessServerSweepAsync` — and the outer launch loop will not relaunch a server's sweep body while the previous one is still running (INV-2, "one body per server").

`query_store` is watermark-driven and its per-run duration is bimodal: mean ~5-35s, but a heavy batch (large plan/text volume) runs 100-230+ seconds. When a heavy run lands on the cycle where every collector's cadence comes due together, `get_collection_health`'s sweep-pressure diagnostic already detects and explains this as `BODY_OVERRUN` — the body cannot finish inside its cadence and its relaunch is skipped.

The consequence is worse than a slow query_store row: because query_store is awaited inline in the same sequential foreach as every other due collector (query_stats, procedure_stats, wait_stats, all 1-minute cadence), and because the whole body can't relaunch until it returns, ALL of that server's collection is blocked for the full 100-230+ seconds — not just query_store's own slot. On observed servers this pushed `last_collection` past the fleet's 15-minute Offline threshold, producing a real (not classifier-bug) "Offline" band on an otherwise fully healthy server with zero actual collector failures.

Observed today (2026-08-30, use1)

11 servers hit this in one hourly check (multi-03, rho-01, multi-14, multi-16, multi-25, multi-27, multi-37, multi-43, multi-44, multi-48, phi-01), each confirmed via `get_collection_health`: `peak_cycle_risk: BODY_OVERRUN`, `peak_collector: query_store`, peak run 83,903-232,815 ms against a 60,000 ms sweep budget, `failed_collector_count: 0` on all of them (this is a scheduling overrun, not a collector failure). Root cause of the heavy query_store batches is a known upstream application issue (application-side decimal-parameter-type instability generating excess distinct query_ids), tracked separately — but PerformanceMonitor itself should not let one collector's slow run stall a server's entire collection body regardless of why that collector is slow.

Fix

Split `query_store` off the sequential body: fire it detached rather than awaiting it inline in `RunDueCollectorsAsync`, reusing the existing per-server `QueryStoreServerGate` (#2165) for mutual exclusion so a still-running previous tick skips rather than overlapping. Every other due collector in the foreach keeps running on schedule regardless of how long query_store takes, and the body itself returns promptly so the outer launch loop keeps relaunching normally.

Activity

  1. erikdarlingdata commented on Sep 1, 2026

    @erikdarlingdata
    OwnerAuthor

    Fixed by #2701 (query_store split off the sequential per-server body). Merged to dev; ships in the next release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions