Skip to content

Stop query_stats from recording a false zero CPU when only the CPU delta is unknowable (#4394) - #4423

Merged
erikdarlingdata merged 1 commit into
devfrom
fix/4394-query-stats-unknown-cpu
Sep 26, 2026
Merged

erikdarlingdata merged 1 commit into
devfrom
fix/4394-query-stats-unknown-cpu

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Sep 26, 2026 •

Copy link
Copy Markdown
Owner

Refs #4394.

Why

QueryStatsCollector.WritePayload makes eight independent delta calls per row, one per counter family, and keeps only the worker (CPU) call's interval as sample_interval_seconds. When the worker family's delta call takes the no-delta path (a first sighting, a counter reset, or a gap past the policy) it reports interval 0 — the honest signal for "this counter's delta is not knowable." But the exec-count and elapsed-time families are keyed and evaluated independently, and in the field they sometimes DO have a real, knowable delta in the same pass where CPU does not. The row was written with CPU 0 and sample_interval_seconds 0 regardless, and the interval-honest filter (sample_interval_seconds IS DISTINCT FROM 0) then discarded that row's real executions and elapsed duration along with the fabricated CPU zero.

Observed on two production SQL Server stores over 24 hours: 183 and 8,844 rows respectively (up to 0.43% of fleet-wide elapsed time), up to 100% of a rare query's day on one store, one row that carried up to 50 seconds of elapsed time, and on one store 86% of the affected rows showing zero reads with a long tail of nonzero ones.

What changes

Not changed

  • A real unchanged CPU counter keeps its 0. A worker delta of 0 over a REAL interval (e.g. a query that ran the whole window but burned no measurable CPU, such as one that spent it all blocked) is a measured zero, not an unknowable one, and is left untouched with its own real interval. This is pinned explicitly (WorkerZeroOverARealInterval_KeepsTheMeasuredZero).
  • A row where every counter is unknowable (first sighting/reset/gap across the board) stays exactly as it is today: CPU 0, interval 0.
  • The other five counters' deltas (reads, writes, physical reads, rows, spills) are untouched.
  • Rows already written are not rewritten by this change.
  • The per-database hourly continuous aggregate (query_stats_interval_hourly) still filters on sample_interval_seconds IS DISTINCT FROM 0, not on delta_worker_time, so a NULL-CPU row with a real interval is included in the rollup's executions/elapsed sums and simply skipped by sum()/min()/max() for the worker column — no continuous-aggregate change was needed or made.
  • ICollectorRowWriter.Value(long?) already exists on both the PostgreSQL (Darling) and DuckDB (Lite) writers, and delta_worker_time is already nullable in both stores, so this needed no migration.

Test plan

  • Pure helper (Darling/Darling.Tests/QueryStatsUnknownCpuTests.cs): five cases against ResolveWorkerDelta — worker unknown + exec known, worker+exec unknown + elapsed known, the stakeholder's real-zero case, all-unknown, and a normal delta. RED on dev: the helper doesn't exist there, so this is a compile-time RED (confirmed on a detached worktree at dev's tip, 15 compile errors, all CS0117/CS8130 pointing at the missing member).
  • Through the collector's payload (same file, one more [Fact]): drives a real CollectorDeltaCalculator where the exec and elapsed families already know the row's key from a prior pass but the worker family sees it for the first time, and a recording ICollectorRowWriter. Asserts delta_worker_time is NULL and sample_interval_seconds comes from the exec interval. Included in the same RED-on-dev compile failure above.
  • Live (Darling/Darling.Tests/QueryStatsUnknownCpuLiveTests.cs, own scratch database per #1776): after PgMigrations.MigrateAsync, plants one normal row and one NULL-CPU row (real interval, real executions/elapsed) for the same query hash, refreshes query_stats_interval_hourly, and asserts the hourly rollup's execution-count and elapsed sums include both rows while the worker sum equals the normal row alone. A second assertion runs the Viewer's worker_time_per_second expression (CAST(delta_worker_time AS double precision) / NULLIF(sample_interval_seconds, 0) / 1000.0) directly against both rows: it returns a number for the normal row and NULL — not an error — for the unknown one.
  • Ran on this Mac against a throwaway timescale/timescaledb:2.30.1-pg18 container (removed after the run): all new tests green, plus the existing DeltaSeriesAgeTests class (25 total, 0 failures).
  • Built Darling.Tests and Lite.Tests on macOS with EnableWindowsTargeting=true: 0 warnings/errors on both. Did not find an existing CollectorPayloadArityTests/QueryStatsCollectorDefinitionTests assertion that pins delta_worker_time's nullability or arity in a way this breaks; Lite.Tests built clean unmodified.
  • Did not run the full existing Darling.Tests/Lite.Tests suites end to end given the time remaining in this pass — only the touched/related classes above. CI will run the full suites on Windows.

CHANGELOG entry

SECTION: Fixed
ENTRY:

  • Query statistics no longer record a false zero CPU when only the CPU counter's delta is unknowable; those rows keep their executions and duration ([Stop query_stats from recording a false zero CPU when only the CPU delta is unknowable (#4394) #4423]) - a query_stats row whose CPU (worker time) counter reset or was seen for the first time, while its execution-count or elapsed-time counters had a real delta in the same pass, previously wrote CPU as a false 0 over a fabricated zero interval — which then caused the row's real executions and duration to be dropped by the interval-honest filter along with the fake CPU. That row now records CPU as unknown (NULL) and keeps its real interval, executions, and duration. A CPU counter that is genuinely unchanged over a real measured interval still records 0, unchanged.

REF:
[#4423]: #4423

@erikdarlingdata
erikdarlingdata marked this pull request as ready for review September 26, 2026 14:14
@erikdarlingdata
erikdarlingdata merged commit 485e160 into dev Sep 26, 2026
16 of 18 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/4394-query-stats-unknown-cpu branch September 26, 2026 14:14
erikdarlingdata added a commit that referenced this pull request Sep 26, 2026
The MCP raw top-N reads apply the hourly rollups' zero-interval filter, so raw and hourly agree over the same window.

TopQueriesSql, TopQueriesByHostObjectSql and TopProceduresSql in DarlingDataReader take TimescaleSupport.IntervalHonestSourceFilter, the filter the interval-honest rollups already use. A zero-interval row carries zero deltas (#4423 now records a real interval when execution or elapsed time is knowable), so no figure changes in real data.

Tests: a SQL-shape pin, and TopQueriesHourlyRoutingLiveTests extended with a zero-interval row, where raw and hourly totals now match through GetTopQueriesByCpuRoutedAsync.

Refs #4394
erikdarlingdata added a commit that referenced this pull request Sep 26, 2026
…ts (#4431)

When a cached plan's counters restart, query_stats now decides the whole row at once, instead of each counter on its own.

- ICollectorDeltaCalculator.DecideRow is new. Its default implementation is a no-op, so existing implementers keep compiling. It peeks at every counter family under the key without updating anything, and reports whether any family would reset.
- CollectorDeltaCalculator.DecideRow is the shared core both hosts run. Any decrease makes the whole row a reset. The #2235 series-age test decides what the row becomes. If the restart falls inside the gap since the previous pass, every counter reads as its current value over the real interval. Otherwise the whole row is unknown, (0, 0).
- QueryStatsCollector.WritePayload calls DecideRow once per row, before the eight per-family delta calls, which still update their own baselines. It then overrides all eight deltas and the interval inputs when the row restarted. A row with no reset is unaffected. The ResolveWorkerDelta fallback from #4423 still applies afterward.
- It applies to query_stats only.
- Tests: QueryStatsRowCoherentResetTests goes through the real WritePayload and calculator. It covers two field-shaped restarts, a row with no reset, an unchanged CPU over a real interval, and a restart the age test can't place.

Refs #4428
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant