Skip to content

Query Store reads on the stored interval tables no longer miss an interval that began just before the table's floor (#5541) - #5543

Merged
erikdarlingdata merged 3 commits into
devfrom
fix/5541-interval-table-floor-margin
Oct 8, 2026
Merged

erikdarlingdata merged 3 commits into
devfrom
fix/5541-interval-table-floor-margin

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

Fixes #5541

Why

The Query Store read gates for the plan regression / drill-down reads (QueryStoreIntervalLatest.UseTable) and the grid / MCP top reads (QueryStoreIntervalWide.UseTable) took the stored interval table when raw's floor R was at or above the table's floor H. Raw purges a snapshot by collection_time; the interval tables purge an interval by first_execution_time. So raw could still hold a snapshot (collection_time >= R) of an interval that began below H and that the table had dropped. With R >= H the read used the table and left that interval out.

What changes

  1. Clause 3a is now R >= H + QueryStoreIntervalWide.PurgeEdgeMargin in both rules. PurgeEdgeMargin (26 h = one interval span of 1 day + the slowest below-floor cadence of 60 min + the collector catch-up cap of 1 h) is the bound the code already documents: the collector keeps only intervals with end_time > cutoff, the cutoff is at most MaxCatchup before the row's collection_time, backfill slices are capped, so first_execution_time > collection_time - (1 day + MaxCatchup). The Wide below-floor read already uses H + PurgeEdgeMargin for the same reason. Both rules share the constant.
  2. Round-1 review, finding 1 (Wide clause 3b): 3b used S - IntervalSpanMargin (1 day). The Wide raw read filters on collection_time only, so a snapshot collected at or after the window start S can belong to an interval that began down to S - 26 h, which left a one-hour hole. 3b now uses S - PurgeEdgeMargin. This case can happen on default retention (raw keeps 30 days, so 3a never passes there and 3b decides): it needs a 1440-minute Query Store interval and a window start in a one-hour band just above the table's oldest data (the review's input: H = S - 24 h). The Latest rule's 3b is unchanged and sound: its raw read also filters last_execution_time >= $2, which keeps every counted interval's first execution at or above B. A comment in that rule's doc now says so.
  3. Round-1 review, finding 2 (table floor): neither interval table is a hypertable in production (retention purges both with a sliced DELETE), and a catalog chunk start would sit far below the real floor, because a DELETE leaves emptied chunks listed, which moves the rule toward the table. H now ALWAYS comes from the per-server MIN(first_execution_time) (PlainTableFloorSql) in both rules, including the Wide cached-inputs path. The chunk-catalog read (ChunkFloorsSql, renamed RawChunkFloorSql) returns only raw's floor; the table_is_hypertable arm and the cached table-floor fields are gone. The earlier "Observed: the hypertable is by_range(first_execution_time)" claim came from test fixtures that convert the table by hand, not from production, and is withdrawn.
  4. Round-1 review, finding 3 (docs): the "+1 day" wording in QueryStoreIntervalWide (the ExactBelowFloorStart doc, the clamp note and the IntervalSpanMargin doc) and in DarlingRetention (two comments) now says 26 h / the purge-edge margin. Test comments that named the old one-day floor were updated too.

Test plan

  • New pure tests in QueryStoreIntervalWideGateTests: Clause3_TableFloorShortOfThePurgeEdgeMargin_ReadsRaw, Clause3_ReviewFailingInput_... (the review's input), Clause3_TableFloorAtOrBeyondThePurgeEdgeMargin_ReadsTheTable. The old one-day pins are replaced; the 3a tests from round 1 stay.
  • RED-plant: with clause 3b put back to S - IntervalSpanMargin (the previous head's rule) and the new tests in place, Clause3_ReviewFailingInput_... and Clause3_TableFloorShortOfThePurgeEdgeMargin_ReadsRaw fail (Total 33, Failed 2); with the fix they pass.
  • Live classes on a local PostgreSQL 18 + TimescaleDB rig (UTC): every class that names UseTable, PlainTableFloorSql, the floor SQL, PurgeEdgeMargin or ResolveReadAsync, plus *QueryStoreIntervalWide*, *PlanRegression*, *QueryStoreTop*, *RawWindowFloor*, *ReadScope*, *DarlingMcpQueryStoreHistory*, *QueryStoreIntervalLatest*: Total 619, Failed 0, Skipped 1.
  • *Census*, *Ratchet*, *Hygiene*, *Pin*, *Guard*, *Scan*, *DocComment*, RepoFileAdoptionTests, FleetIdentifierScrubTests, RegisteredServerSettingDriftTests, TsqlConventionGuardTests: Total 1690, Failed 0, Skipped 2.
  • The wide fleet-eligibility live tests count the floor read by its raw_floor text only now (the hypertable column is gone from that statement); they still count one read per check.
  • Darling.Tests build: 0 warnings, 0 errors.
  • Full suite not run (CI runs it).

Measurements

None from a store. On the default store (raw 30 days, table 9 to 15 days) 3a never passes, so the Wide read decides on 3b, which is where the one-hour hole was.

Changelog

SECTION: Fixed
ENTRY:

…and the table floor always comes from the per-server minimum (#5541)

The Wide raw read counts a snapshot by collection_time alone, so a snapshot collected at or after the window start can belong to an interval that began down to S - 26 h. Clause 3b used S - 1 day and left a one-hour hole; it now uses S - PurgeEdgeMargin.

Neither interval table is a hypertable in production (retention purges both with a sliced DELETE), and a catalog chunk floor would sit far below the real one because emptied chunks stay listed. Both rules now always read H as the per-server MIN(first_execution_time); the chunk-catalog read keeps only raw's floor. Stale one-day wording in the docs now says 26 h.
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review October 8, 2026 07:03
@erikdarlingdata
erikdarlingdata merged commit b532e34 into dev Oct 8, 2026
21 of 22 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/5541-interval-table-floor-margin branch October 8, 2026 08:45
erikdarlingdata added a commit that referenced this pull request Oct 8, 2026
Adds the changelog entries for the 3.10.0 PRs that merged after the first splice: #5528, #5532, #5537, #5542, #5543, #5552 and #5553. Each full entry goes into docs/changelog/3.10.md and its title-only line into the CHANGELOG.md index. The archive census row for 3.10.0 moves from 257 to 265 entries.

Changelog entries for the 3.10.0 PRs merged after the first splice.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant