Skip to content

query_store's normal hourly source cadence no longer reads as "produced then stopped" (#4473) - #4483

Merged
erikdarlingdata merged 1 commit into
devfrom
fix/4473-interval-sourced-zero-streak
Sep 27, 2026
Merged

erikdarlingdata merged 1 commit into
devfrom
fix/4473-interval-sourced-zero-streak

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner

Fixes #4473.

Why

CollectorHealthClassifier.ProducedThenStopped flags a collector as regressed once it has three
consecutive SUCCESS runs storing zero rows after having been productive. That rule is right for a
collector whose source advances on every cycle. query_store runs every five minutes, but its
source (Query Store's runtime-stats view) only gains a new row when the engine-side Query Store
interval closes — commonly an hour, and configurable up to a day (INTERVAL_LENGTH_MINUTES accepts
1/5/10/15/30/60/1440). So ten or eleven zero-row cycles followed by one productive cycle is
query_store's ordinary hourly shape, and the plain streak rule flagged it as "produced then
stopped" on every health surface that reads it.

What changes

  • A new name set, IntervalSourcedCollectorNames, with IsIntervalSourcedCollector(name), next
    to the existing event-collector and on-load-collector sets in ServerHealthBands.cs. Members:
    query_store — it reads sys.query_store_runtime_stats per interval
    (install/09_collect_query_store.sql:748,753).
    No other collector's source advances per Query Store interval today: plan_correction reads
    sys.dm_db_tuning_recommendations joined to sys.query_store_plan by id (a live catalog lookup
    keyed on the recommendation set, not a per-interval aggregate —
    PerformanceMonitor.Collectors/PlanCorrectionCollector.cs), and query_store_health is a
    per-database configuration snapshot off sys.database_query_store_options, not a runtime-stats
    read. Both were checked and excluded with their evidence rather than added on a guess.
  • A named constant, QueryStoreLongestIntervalSlack = TimeSpan.FromMinutes(1440 + 15). 1440 is
    the longest legal INTERVAL_LENGTH_MINUTES; 15 minutes is cadence slack on top of it.
  • A new clock-aware overload, ProducedThenStopped(collectorName, trailingZeroRowSuccessRuns, lastProductiveTimeUtc, nowUtc). For an interval-sourced collector it flags only when the streak is
    at least 3 and nowUtc - lastProductiveTimeUtc is strictly greater than
    QueryStoreLongestIntervalSlack. Every other collector keeps the existing streak-only rule,
    unchanged. The existing three-argument overload is kept and now forwards to the new one with
    DateTime.UtcNow, so every existing caller (Darling's per-server MCP read and Lite's local health
    read) keeps the same UTC-now clock semantics it had before this change — neither caller needed to
    change.

Why not exclude it outright

A real Query Store stop still needs a backstop: Query Store can genuinely go quiet (for example when
it flips to READ_ONLY), and that is a real regression an operator should see eventually.
get_query_store_health's state check catches the common READ_ONLY case earlier and more
specifically; this arm is the fallback for whatever that check misses, so it keeps a roughly
day-plus-slack backstop rather than exempting query_store from this class of finding entirely.

Test plan

All in Darling/Darling.Tests/ProductiveZeroBandingTests.cs (extended), run on macOS via
Darling.Tests.dll in-process (Microsoft.WindowsDesktop.App stripped from the runtimeconfig).

  • RED against dev, through the pre-fix 3-argument overload: an isolated probe test asserting
    query_store with 11 zero-row runs and a productive instant 55 minutes back is NOT regressed,
    built and run against origin/dev in a detached worktree — FAILED (the pre-fix rule flags it).
    The same fact passes on this branch.
  • query_store, 1,900 zero runs, last productive 14 days ago → regressed (job_history dark on 41 of 43 servers for up to two weeks: the numeric instance_id watermark outlived a weekly msdb reseed, so every run is SUCCESS with zero rows — add an identity-epoch check that falls back to the bounded window and bands the regression #3885's own shape, on an
    interval-sourced collector).
  • A 1440-minute-interval collector: 23 hours since productive → not regressed; 25 hours → regressed.
  • The slack boundary itself: exactly QueryStoreLongestIntervalSlack → NOT regressed (strictly
    greater); one tick past it → regressed.
  • A non-interval-sourced collector (wait_stats) with a short streak still flags immediately —
    unchanged.
  • The set itself: query_store is interval-sourced; wait_stats, job_history and null are not.
  • All of the pre-existing facts in the file (event/on-load exclusions, the finding text, the fleet
    estimate, the wiring pins) stayed green with no changes to their bodies.
  • Two real mutations, each rebuilt and reverted: IsIntervalSourcedCollector forced to always
    return false → 4 facts went RED (the hourly-shape fact, the 1440-minute fact, the boundary fact,
    and the set-membership fact); > changed to >= on the clock comparison → the boundary fact went
    RED. Both reverted to GREEN.
  • Ran: ProductiveZeroBandingTests + DocCommentHygieneTests (no other test class in
    Darling.Tests/Lite.Tests references ProducedThenStopped or ProductiveZeroRunStreak) — 93/93
    passed on macOS.
  • Build: Darling.Tests and Lite.Tests, Release, -p:EnableWindowsTargeting=true — 0 warnings,
    0 errors on both. Lite.Tests is build-only on macOS (discovery needs Windows); its caller keeps
    the same UTC-now semantics as Darling's and gets the fix through the shared classifier, with no
    code change of its own needed. CI decides the Lite runtime pins.

CHANGELOG

SECTION: Fixed
ENTRY:

…ed then stopped" (#4473)

query_store polls its target every five minutes, but its source (Query
Store's runtime-stats view) only gains a new row once per interval,
which is commonly an hour and can be configured up to a day. Ten or
eleven zero-row cycles between productive ones is that collector's
ordinary shape, and the streak-only rule flagged it as regressed.

Adds an interval-sourced collector name set (query_store today), a
named slack constant for the longest legal Query Store interval plus
cadence slack, and a clock-aware overload of ProducedThenStopped that
gives an interval-sourced collector a day-plus-slack backstop instead
of the plain streak count. Every other collector's rule is unchanged.
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review September 27, 2026 17:58
@erikdarlingdata
erikdarlingdata merged commit 0abba99 into dev Sep 27, 2026
16 of 18 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/4473-interval-sourced-zero-streak branch September 27, 2026 17:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant