Repository navigation
Top-N raw reads apply the hourly rollups' zero-interval filter, so both tiers agree by construction (#4394) - #4414
Merged
Conversation
…g the hourly rollup (#4394) query_stats' zero-interval first-collection row can carry a delta_worker_time cumulative since plan creation rather than a true per-interval delta. The hourly successor already excludes this row from its CREATE (#3653's IntervalHonestSourceFilter); TopQueriesSql, TopQueriesByHostObjectSql and TopProceduresSql did not, so the same window ranked differently raw vs hourly. Adds the same filter to all three raw consts, through the one shared constant. Refs #4394.
…st' into fix/4394-raw-top-n-interval-honest
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #4394.
Why
The hourly successors exclude zero-interval rows (
sample_interval_seconds = 0): theirCREATEbakes inTimescaleSupport.IntervalHonestSourceFilter(sample_interval_seconds IS DISTINCT FROM 0). The raw top-N reads did not carry the same filter, so the two tiers answered the same window through different row sets.In real data that difference is empty. The collector writes a zero-interval row with zero deltas:
CollectorDeltaCalculatorreturns (delta 0, interval 0) for a first sighting, a counter reset, and a gap past the delta policy, and a first sighting that did real work (#2235) carries its real interval. A 24-hour count on two production SQL Server stores founddelta_worker_timesumming to exactly 0 over every zero-interval row. So this change moves no CPU figure. It applies the filter so that a raw-served and an hourly-served read agree by construction, instead of by the collector's invariant (see #4394's close-out).What changes
Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingDataReader.cs:TopQueriesSql,TopQueriesByHostObjectSqlandTopProceduresSqleach gainAND {TimescaleSupport.IntervalHonestSourceFilter}in theirWHEREclause, through the oneshared constant (matching the file's existing pattern of interpolating Storage constants into
$"""raw strings) — never a copied literal.TopQueriesHourlySqlis untouched; it alreadyinherits the filter from its successor's
CREATE.Test plan
dev(the constsdon't emit it there); GREEN on the branch. Build:
Darling.Tests0 warnings / 0 errors,Lite.Tests0/0 (build-only on macOS; both targetnet10.0-windows, CI runs them).TopQueriesHourlyRoutingLiveTests.cs:GetTopQueriesByCpuRoutedAsync_RoutesToHourlyOnceRawIsPurged_AndTotalsAgreenow seeds azero-interval row for a group that also has ordinary rows (
0xTOPQ1), on top of theexisting zero-interval-only group (
0xZEROINTERVAL). Both are routed throughGetTopQueriesByCpuRoutedAsync(raw, then hourly with a freshNpgsqlDataSourcefor the5-minute coverage cache), never by reading either table's SQL directly.
dev(pre-fix, run in-process on macOS against a throwaway TimescaleDB container,timescale/timescaledb:2.30.1-pg18): raw's0xTOPQ1total (17 executions / 1,500,000CPU-us) exceeds hourly's (16 executions / 800,000 CPU-us) by exactly the zero-interval row's
delta —
[FAIL]at the assertion"a hash whose only rows are zero-interval is excluded from raw's own read by #4394's fix"(line 107 of the pinned test).Darling.Tests.TopQueriesHourlyRoutingLiveTests—Total: 2, Errors: 0, Failed: 0.TopProceduresSqlhas no routed read yet (its hourly routing is Route get_top_procedures_by_cpu and the Top Procedures grid to the hourly rollup (#4231) #4413, not yet on dev), so it's covered only by the unit shape pin above; a routed live pin for proceduresis follow-on work once Route get_top_procedures_by_cpu and the Top Procedures grid to the hourly rollup (#4231) #4413 (At "Last 7 days" the WPF Queries grids silently show ~4 days: Top Queries, Top Procedures and Query Store read raw tables dropped at 4 days, with no floor disclosure (get_query_store_top got one in #2364; the viewer and the query-stats MCP tools didn't) #4231's procedure routing) is on dev.
TopQueriesHourlyRoutingTests,IntervalHonestHourlyRollupTests,LiveCleanupConversionRatchetTests,StorageCommandTimeoutTests,StartupCommandTimeoutTests,DocCommentHygiene,AlertReadFailureSurfaceTests.LivePostgresCollectionHygieneTests.EveryClassUsingTheSharedStore_IsSerializedOrDocumentsWhyNotfails on macOS only, with
PresentationFrameworkmissing after the runtimeconfig strip neededto run
Darling.Tests.dllin-process here (a reflection-over-all-types scan hits a WPF-typedclass); pre-existing and unrelated to this change — CI decides it on Windows.
Ranking on the test seed
The seed plants a zero-interval row with a nonzero delta, a shape the collector does not write, so that the filter is observable:
Window
WindowStart..WindowStart+1d,0xTOPQ1carries an ordinary row (500,000 CPU-us / 10executions), a second ordinary row (300,000 CPU-us / 6 executions), and one zero-interval row
(700,000 CPU-us / 1 execution):
0xTOPQ1total0xTOPQ1total (unchanged either side)Before: raw and hourly disagreed by the planted row's delta. After: raw's own read
now excludes that row the same way the hourly successor's
CREATEalways has, so the two tiersagree on the same window.
CHANGELOG entry
SECTION: None
ENTRY: None: raw and hourly top-N reads now apply the same row filter; no figure changes in real data (#4394).