Read the interval table below the raw floor, down to the exact bound it can prove, and say how far back it reaches (#4689)

[erikdarlingdata]

Fixes #4689.

Summary

query_store_interval_wide reads were clamped at raw's chunk floor, so a window longer than the raw tier lost its older part even though the table still held it. Reads now go below the floor down to L = max(window start, filled_since, table floor + 1 day) and no further. Each surface reports the start it actually served. GetQueryStoreTop (MCP tool) now returns history_source (raw or interval_table) and effective_start, and it builds a truncation_note that names which bound set the start. The reason sentences are:

• FilledSince: "The interval table began keeping complete history for this server at {effective_start}."
• TablePurgeEdge: "The interval table keeps 9 days, and intervals that began before its purge edge are not read."
• RawFloor / Window: "The read is clamped at the raw tier's retention floor: nothing older than it can be shown exactly."

A WideStartBound enum names the bound (ties prefer FilledSince, then TablePurgeEdge, then Window). ExactBelowFloorStart computes the start and ResolveReadAsync returns it as ReadStart.

Fixed here: ResolveReadAsync bound Kind=Utc DateTimes to timestamp parameters. Npgsql refuses that, the gate's catch fell back to raw, and the MCP tool (which resolves its window from DateTime.UtcNow) never read the table on the unreleased #4341 route; each such call logged a warning and read raw. The gate now normalizes its window arguments to Unspecified.

Hazards (why L is what it is)

• H1: rows below filled_since are outside the coverage claim.
• H2: retention deletes row by row on first_execution_time, so the oldest intervals can be partly purged (hence table floor + 1 day).
• H3: pending batches whose raw chunk is gone can never replay (the gate already refuses while any is pending).
• H4: legacy rows (interval_start_time_utc IS NULL) differ from raw.
• H5: late backfill rows land below filled_since (covered by H1).
• H6: no materialization job touches this table.
• H7: Compose read below the floor unclamped (fixed in the Viewer/Compose section).

Design notes

• Coverage-based disclosure, no per-server probe: the only secondary index on the table leads with first_execution_time.
• Duration trend: stays on the clamp; legacy rows are real (QueryStoreCollector.cs:805).
• Lite: no equivalent (Lite has no interval table; grep of Lite/ finds no query_store_interval_wide).
• Watch item, pre-existing: PlainTableFloorSql is MIN(first_execution_time) WHERE server_id on that index. It is cheap when the server has old rows and a full index walk for a server with coverage but no rows.
• Docs updated in present tense: DarlingRetention.cs (9-day constant and the purge comment) and the QueryStoreIntervalWide.cs section comment, ClampedStart doc (the clamp is the bound at and above raw's floor; ResolveReadAsync extends it below).
• The history note and the grid's short reason come from one builder in Storage (QueryStoreIntervalWide.HistoryNote / BannerReason), pinned to occur once.
• A visibility gap this doesn't change: every production caller of the gate (the MCP read, Compose, and the Viewer grid and trend) passes logger: null. A gate fault falls back to raw silently, which is why the Kind defect above left no trace in the service log. Wiring a logger through those static readers is a separate change.

Pins and REDs

Live class QueryStoreIntervalWideBelowFloorLiveTests (7 tests, local TimescaleDB rig, 7/7 green). Each mutation was a temporary code change, reverted after:

• (a) ExactBelowFloorStart ignoring filledSince: RED, 3 fail (FilledSinceBound_IsLoadBearing line 138/150, Boundary_IsInclusive, Disclosure_EffectiveStart…).
• (b) ignoring tableFloor + IntervalSpanMargin: RED, 2 fail (TablePurgeEdge_Bound line 215/220, Disclosure_EffectiveStart…).
• (c) > instead of >= on collection_time in the MCP table read: RED, 2 fail (Boundary_IsInclusive line 193/196, PlainStore_ReadStartIsWindowStart line 234).
  Unit: QueryStoreIntervalWideGateTests 23/23. Also added history_source to the DarlingMcpDataToolsTests payload pin (raw), and a QueryStoreTopWindowTests pin that the raw floor probe sits after the interval-table branch.

Census and neighbours run on the rig (745 tests, QueryStoreTop*, QueryStoreIntervalWide*, RawWindowFloor*, DarlingMcpDataToolsTests-adjacent pins, McpToolsListBudgetTests, McpReadCommandTimeoutTests, every *Census*, *AdoptionTests, timeout/deadline classes, DocCommentHygieneTests): 741 pass, 4 fail. The 4 are RawWindowFloorViewerPortTests.UpdateTruncationBanner_*, which need WPF (STA) and cannot run on macOS. Darling.Tests and Viewer builds: 0 warnings, 0 errors. DarlingMcpDataToolsTests itself was not run as a class (Windows-only DPAPI cases); its one edit is a one-line assertion.

• QueryStoreIntervalWideHistoryNoteTests (14): exact HistoryNote strings for every bound and both scopes, exact BannerReason, the Compose forwarder, and a census that the purge-edge phrase occurs once in product source (in QueryStoreIntervalWide.cs). RED: HistoryNote saying "this server" when manyServers is true fails 4 of 14; restored, 14/14. DarlingMcpDataToolsSurfaceAndSqlTests 64/64.

Rig EXPLAIN (local TimescaleDB rig, not production numbers)

MCP table read (QueryStoreTopTableSql) at a below-floor start, 2,000,000 _wide rows over 40 servers, one server, 6.5-day window: Index Scan using ux_query_store_interval_wide returned 37,036 rows; Buffers: shared hit=50835 for the scan, 50,847 total; small sort spill (temp read 482); Execution Time 72 ms.

Viewer grid and Compose

Queries grid. The Query Store grid now calls ResolveReadAsync and binds the plan's ReadStart (new GetQueryStoreTopQueriesWithReachAsync returns rows plus the plan; the old method delegates). When the interval table served, the banner reads Showing since <EffectiveStart> plus (interval table complete from then) or (interval table keeps 9 days), and appends · slicer since <raw floor> when raw's floor is truncated (the slicer stays raw). The raw route's banner is unchanged.

Duration trend. No behaviour change: a doc line at its ReadsTableAsync call says it stays on the clamp, because _wide can hold legacy NULL interval_start_time_utc rows that make the table differ from raw, and below raw's floor they cannot be checked without a probe that walks every server's rows.

Compose. The #4617 Compose path read the interval table below the raw floor unclamped and was not exact there; this binds the exact start. ResolveQueryStoreWideEligibleAsync now calls ResolveReadAsync per server and returns the latest ReadStart as one common WideStart (plus the bound of the server that set it). ComposeRunContext.QueryStoreWideStart carries it; the compiler binds it as its own parameter on the table read's collection_time predicate (collection_time is query_store_stats' time column). The response adds query_store_history_starts and query_store_history_note when the table served from a start later than the window's.

Pins. DarlingComposeTests (SQL carries the wide-start bound when set, not when null/earlier; note text), RawWindowFloorViewerPortTests (banner branches, wiring and trend doc-line source pins; the banner test needs WPF and cannot run on macOS), and live ComposeQueryStoreWideStartLiveTests (filled_since forced to S+12h, backdated table rows at S+1h excluded). RED: binding no wide-start parameter makes the live test fail (earliest bucket is below filled_since); restored and green.

The purge-edge margin and the cadence gate

• Reads below raw's floor require the server's effective query_store cadence to be 60 minutes or less (the per-server override, else the fleet override, else the 5-minute default, from config.config_collector_schedules) AND the largest gap between successful (SUCCESS) query_store collection-log rows over [the table floor's UTC day, that day + 3 days] to be 60 minutes or less, with the two range edges counted as boundary rows so a lead-in or tail gap counts. No rows in that range counts as unknown and stays clamped. Otherwise the read is clamped at raw's floor with the reason RawFloorSlowCadence, and the banner says "slow Query Store cadence".
• The purge-edge margin is 1 day + 60 minutes + the 1-hour catch-up cap (26 h): a purged interval's last snapshot can land up to one interval length (at most 1,440 minutes on the monitored server), one cadence and one catch-up after its first execution. The schema caps the cadence only at frequency_minutes >= 0, which is why the gate exists.
• collection_log is kept 60 days, so a 9-day-old edge is always covered. On a local rig (20 servers x 2 collectors x 30 days at 5 minutes) the gap read is an index scan on idx_collection_log_watermark with the server, collector and time range in the index condition: 576 rows, shared hit=216 read=6, 0.52 ms.
• The Viewer banner says "Slicer since (the grid shows the full window)" when only the slicer is cut, and "Showing since … (reason) · slicer since …" when both are. Compose's note names the server that set the common start, and the payload carries query_store_history_set_by.
• The duration trend stays at raw's reach (legacy rows with no interval start are real); the note says so.
• Pins: a late-snapshot positive purge-edge pin (raw snapshotted before both purges, EXCEPT ALL both ways against the table read at ReadStart; RED on a 1-day margin, where the read starts 02:00 instead of 04:00 and the late interval is missing); cadence 61 stays clamped and 60 reads below; a 3-hour log hole at a 5-minute cadence stays clamped; an empty log stays clamped; a late first or early last collection stays clamped (RED without the boundary rows); the Viewer grid and a compiled Compose panel each get the snapshot → drop → EXCEPT ALL pin.

The collection-gap verdict cache

• On a large store the gap read cost about 65 ms and about 480 buffers per server per decision, because it reads 7-9-day-old compressed collection_log chunks; a 43-server Compose panel paid about 2.7 s per render. The verdict is now cached.
• Key: (store host:port/database, server_id, the table floor's UTC day, the effective query_store cadence). The window is [floor day, floor day + 3 days], derived from the key only: a superset of [floor, floor + 2 days] for every floor in that day, so it can only clamp more. One entry per (store, server); storing a new floor day or cadence drops that server's older entry. A null or failed read is not cached, and a restart starts empty. The cadence is resolved on every decision.
• Settled only: a window is read and cached only when floor day + 3 days <= now - the 1-hour catch-up cap - a 5-minute settle margin. Until then (only a young table, whose oldest interval is under about 3 days old) the read is clamped before the gap read with the reason "the collection log does not yet cover the interval table's purge edge".
• Never backdated: the only writer of collect.collection_log is DarlingObservability.InsertCollectionLogSql (callers at DarlingObservability.cs lines 264, 397 and 514), which writes collection_time = DateTime.UtcNow. No import, backfill or replay path inserts into it, so a settled window's answer cannot change. The gap read counts SUCCESS rows only (SKIPPED is written only by Lite, which has no interval table).
• A failing cadence probe gives up only the below-floor part; the at-floor table read still serves.
• Not batched: Compose still resolves each server in its own round trips, so after a restart (and once per floor day) the first Compose render pays about N x 65 ms on that store; later renders are cache hits. The MCP and grid single-server reads are unchanged.
• Measured on a large production store (43 servers, read-only, final SQL with the 3-day window and SUCCESS only): median 38.9 ms and 700 buffers per server (max 45.3 ms, 728 buffers), 1,638 ms summed; the earlier 2-day read measured about 65 ms and 480 buffers per server on colder chunks. So a 43-server Compose panel pays about 1.6-2.7 s on its first render after each floor-day rollover per service process, then cache hits. Largest collection gaps on that store were 13.1-24.7 minutes, so none of its 43 servers is clamped by the gate.
• Pins (QueryStoreIntervalWideGapCacheLiveTests): a second decision issues no read; a cadence change recomputes; a moved floor day recomputes (a day-D verdict never serves D+1); a window still filling is not cached; a gap past floor + 2 days now clamps (the conservative trade of the day-wide window). REDs: a key without the cadence, never caching, and no settle rule each fail a pin. The below-floor, gap-cache and Viewer floor classes share one serial xunit collection because they share the cache's clock and read counter.
• The young-table reason has its own pin, the late-snapshot seed is a possible interval (it ends 5 minutes before a day after it began; its snapshot lands 55 minutes later) and still fails on a 1-day margin (read start 02:00 instead of 04:00) and passes on 26 h.

CHANGELOG

SECTION: Changed
ENTRY: - Query Store views over windows longer than the raw tier now show the full window from the interval table, and say how far back it reaches ([#4700]) - The Queries grid, the get_query_store_top MCP tool and Compose Query Store panels read the per-interval table below the raw tier's floor, down to the exact point where it provably holds the same rows raw held. The MCP tool reports history_source and effective_start, and its note says which bound set the start.
REF: [#4700]: #4700