Repository navigation
query_store's normal hourly source cadence no longer reads as "produced then stopped" (#4473) - #4483
Merged
Merged
Conversation
…ed then stopped" (#4473) query_store polls its target every five minutes, but its source (Query Store's runtime-stats view) only gains a new row once per interval, which is commonly an hour and can be configured up to a day. Ten or eleven zero-row cycles between productive ones is that collector's ordinary shape, and the streak-only rule flagged it as regressed. Adds an interval-sourced collector name set (query_store today), a named slack constant for the longest legal Query Store interval plus cadence slack, and a clock-aware overload of ProducedThenStopped that gives an interval-sourced collector a day-plus-slack backstop instead of the plain streak count. Every other collector's rule is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4473.
Why
CollectorHealthClassifier.ProducedThenStoppedflags a collector as regressed once it has threeconsecutive SUCCESS runs storing zero rows after having been productive. That rule is right for a
collector whose source advances on every cycle.
query_storeruns every five minutes, but itssource (Query Store's runtime-stats view) only gains a new row when the engine-side Query Store
interval closes — commonly an hour, and configurable up to a day (
INTERVAL_LENGTH_MINUTESaccepts1/5/10/15/30/60/1440). So ten or eleven zero-row cycles followed by one productive cycle is
query_store's ordinary hourly shape, and the plain streak rule flagged it as "produced thenstopped" on every health surface that reads it.
What changes
IntervalSourcedCollectorNames, withIsIntervalSourcedCollector(name), nextto the existing event-collector and on-load-collector sets in
ServerHealthBands.cs. Members:query_store— it readssys.query_store_runtime_statsper interval(
install/09_collect_query_store.sql:748,753).No other collector's source advances per Query Store interval today:
plan_correctionreadssys.dm_db_tuning_recommendationsjoined tosys.query_store_planby id (a live catalog lookupkeyed on the recommendation set, not a per-interval aggregate —
PerformanceMonitor.Collectors/PlanCorrectionCollector.cs), andquery_store_healthis aper-database configuration snapshot off
sys.database_query_store_options, not a runtime-statsread. Both were checked and excluded with their evidence rather than added on a guess.
QueryStoreLongestIntervalSlack = TimeSpan.FromMinutes(1440 + 15). 1440 isthe longest legal
INTERVAL_LENGTH_MINUTES; 15 minutes is cadence slack on top of it.ProducedThenStopped(collectorName, trailingZeroRowSuccessRuns, lastProductiveTimeUtc, nowUtc). For an interval-sourced collector it flags only when the streak isat least 3 and
nowUtc - lastProductiveTimeUtcis strictly greater thanQueryStoreLongestIntervalSlack. Every other collector keeps the existing streak-only rule,unchanged. The existing three-argument overload is kept and now forwards to the new one with
DateTime.UtcNow, so every existing caller (Darling's per-server MCP read and Lite's local healthread) keeps the same UTC-now clock semantics it had before this change — neither caller needed to
change.
Why not exclude it outright
A real Query Store stop still needs a backstop: Query Store can genuinely go quiet (for example when
it flips to
READ_ONLY), and that is a real regression an operator should see eventually.get_query_store_health's state check catches the commonREAD_ONLYcase earlier and morespecifically; this arm is the fallback for whatever that check misses, so it keeps a roughly
day-plus-slack backstop rather than exempting
query_storefrom this class of finding entirely.Test plan
All in
Darling/Darling.Tests/ProductiveZeroBandingTests.cs(extended), run on macOS viaDarling.Tests.dllin-process (Microsoft.WindowsDesktop.Appstripped from the runtimeconfig).query_storewith 11 zero-row runs and a productive instant 55 minutes back is NOT regressed,built and run against
origin/devin a detached worktree — FAILED (the pre-fix rule flags it).The same fact passes on this branch.
query_store, 1,900 zero runs, last productive 14 days ago → regressed (job_history dark on 41 of 43 servers for up to two weeks: the numeric instance_id watermark outlived a weekly msdb reseed, so every run is SUCCESS with zero rows — add an identity-epoch check that falls back to the bounded window and bands the regression #3885's own shape, on aninterval-sourced collector).
QueryStoreLongestIntervalSlack→ NOT regressed (strictlygreater); one tick past it → regressed.
wait_stats) with a short streak still flags immediately —unchanged.
query_storeis interval-sourced;wait_stats,job_historyand null are not.estimate, the wiring pins) stayed green with no changes to their bodies.
IsIntervalSourcedCollectorforced to alwaysreturn
false→ 4 facts went RED (the hourly-shape fact, the 1440-minute fact, the boundary fact,and the set-membership fact);
>changed to>=on the clock comparison → the boundary fact wentRED. Both reverted to GREEN.
ProductiveZeroBandingTests+DocCommentHygieneTests(no other test class inDarling.Tests/Lite.TestsreferencesProducedThenStoppedorProductiveZeroRunStreak) — 93/93passed on macOS.
Darling.TestsandLite.Tests, Release,-p:EnableWindowsTargeting=true— 0 warnings,0 errors on both.
Lite.Testsis build-only on macOS (discovery needs Windows); its caller keepsthe same UTC-now semantics as Darling's and gets the fix through the shared classifier, with no
code change of its own needed. CI decides the Lite runtime pins.
CHANGELOG
SECTION: Fixed
ENTRY:
query_storeno longer reads as "produced then stopped" between Query Store intervals ([query_store's normal hourly source cadence no longer reads as "produced then stopped" (#4473) #4483]) - it runs every 5 minutes, but its source advances once per interval (up to a day). It's flagged only after more than a day plus 15 minutes with nothing new.REF:
[query_store's normal hourly source cadence no longer reads as "produced then stopped" (#4473) #4483]: query_store's normal hourly source cadence no longer reads as "produced then stopped" (#4473) #4483