Field data confirms the interval-table fix removes the spill, ship it (#3953)

[erikdarlingdata]

Part of #3953.

Why

#3953 was built, tested, and shelved on 2026-09-23. The shelving measurement was on a small rig. It found a
full 14-day pass only went from about 6.5s to 5.5s. The drill-down got 4x slower (71ms to 286ms). That did
not justify three new tables, a writer inside the Query Store COPY transaction, a replay path, and a
compression-band conflict. The ruling was to reopen if field data from the busy client-site server showed
PLAN_REGRESSION dominating.

It now does. The store ingests 43 SQL Servers' Query Store data (build 3.8.0-nightly.20260924.467, about 5
hours of steady-state passes). On it, PlanRegressionSql, the raw whole-slice dedup this issue targets, is
the store's single heaviest statement:

measure	value
share of owner-role store time	22.5% (5,164s of 22,914s)
calls	344 (about one per server per pass)
mean / max time	15.0s / 58.5s
shared blocks read	19.7M (about 162GB)
temp blocks written	10.35M (about 85GB in about 5 hours), 91% of all temp writes among the top 40 statements

At that rate the spill alone is roughly 400GB/day of temp writes. That is exactly the write-amplification
input the original ruling named as a first-class factor. The small rig's 14-day pass never spilled to temp,
which is why it missed this.

What changes

This is the branch built for the original ruling (design v2, adversarial-reviewed before code). A
collector-written table, collect.query_store_interval_latest, holds the latest Regular snapshot per Query
Store interval, with a 15-day horizon. PLAN_REGRESSION and its drill-down aggregate the table instead of
deduplicating raw. Each server and pass reads the table only when coverage matches the raw read's scope
and there are no pending (failed-apply) batches. Otherwise the read falls back to the shipped
raw SQL, unchanged. Full design is in the issue's 2026-09-23 comments.

This lane's own work was the merge and the re-measurement:

• Merged origin/dev. The branch's migration was V140. Dev had since grown its own V140
  (checkpointer-timed-count, Store Checkpointer Pressure compares an hourly sync sum to a per-checkpoint bar, so it fires permanently on healthy stores #4037) and V141 (collection-caveats, PostgreSQL-target analysis engine: v2/v3 slices and the v1 lanes' residue (from #3542) #3691). Renumbered this branch's
  migration to V143, the next free version. Updated every version-bump obligation: StorageVersion.SchemaVersion,
  StoreSchemaProbeSql sentinel (arm and parameter), the GetBoolean call
  list, and the rung test files' handoff chain. QueryStoreIntervalLatestRungTests now claims the top rung.
  CollectionCaveatsRungTests does not.
• Fixed three tests the renumbering broke. CollectionCaveatsRungTests still asserted it was the top
  rung. QueryStoreIntervalLatestWriterTests had one hardcoded Scripts.Single(m => m.Version == 140).
  ViewerCollectionCaveatsGateTests built its "V141 absent" argument array as i < arity - 1, which silently
  stopped meaning "V141's own ordinal" the moment V143 appended a parameter after it. Fixed by finding the
  parameter by name via reflection instead.
• Fixed a line-ending drift in the two files I hand-resolved through a merge-conflict splice. The repo's
  .gitattributes pins .cs files to CRLF. A Python read/write round-trip during conflict resolution had
  silently flattened both files to LF-only on disk. Git's own clean filter had already fixed the committed
  blob, which is why only the working copy needed the fix. OversizedPlanBacklogPins caught it, since it
  pins a literal CRLF sequence in the migration SQL text.
• Re-measured with a field-shaped load (below).

Re-measurement

The small rig never spilled. This time the seed was sized to make the OLD read spill. Seed: one server, 14
days, 500 queries x 2 plans x 336 hourly Query Store intervals x 4 re-collections per open
interval. That is 1,344,000 raw rows, deduplicating to 336,000 latest-interval rows, against a plain
PostgreSQL 18 + TimescaleDB rig at default work_mem (4MB). EXPLAIN (ANALYZE, BUFFERS) on both statements,
same parameters, same snapshots:

	raw dedup (old)	interval table (new)	change
execution time	3,428.95 ms	297.76 ms	11.5x faster
shared blocks read	32,109 (~250.9 MB)	3,810 (~29.8 MB)	8.4x less
temp blocks written	42,295 (~330.4 MB)	4,308 (~33.7 MB)	9.8x less

The old plan's Sort node reads Sort Method: external merge Disk: 57888kB per parallel worker. That spills
the full 1.34M-row slice three times over, once per worker, on top of two more external-merge spills at the
GroupAggregate stages above it. The new plan reads the pre-deduplicated 336,000-row table directly. Its
worker-level sort still spills modestly (about 34MB total), but nothing upstream of it does.

This reproduces the field mechanism: heavy temp writes from sorting the whole raw slice. The scale is about
1/40th of one server's share of the real store's roughly 5-hour window. The new shape removes the great
majority of it there. Given the lane's time budget, I did not additionally scale up to the full field row
count. The ratios above are squarely consistent with the field numbers' 22.5%-of-store-time and
91%-of-temp-writes shares.

Result identity is proven at real scale through the real write path. An md5
oracle covers the fact and drill-down, table vs. raw, including a purge-boundary case
(PlanRegressionIntervalTableEquivalenceTests). It was re-run live against this lane's rig and passed,
unchanged by the merge.

Lite parity

S1's claim above ("Lite... I did not change it") is incomplete: this branch's diff against origin/dev already touches 8 files outside Darling. They predate this lane, inherited from the branch's 2026-09-23 revival, not written by S1. Per this lane's brief, they are accounted for here rather than reverted. The PR tender decides what to do with them:

file	what the hunk does	relation to #3953
Lite/Analysis/DuckDbFactCollector.QueryPerf.cs (+28)	Carries best_last_exec through the fact's compared CTE and SELECT. Computes best_plan_age_days metadata and a plan_regression_source flag (0 = Lite's raw read, 1 = Darling's interval table) on the PLAN_REGRESSION fact.	Direct: the fields the rest of this list consumes.
Lite/Analysis/DrillDownCollector.Queries.cs (+11/-2)	Appends best_plan_last_seen to the drill-down's compared CTE and output row.	Direct: same age data, for the drill-down's own display.
PerformanceMonitor.Analysis/AnalysisContext.cs (+9)	Adds PlanRegressionReadsIntervalTable (nullable bool): which source Darling's fact read this pass. The drill-down reads the same source, so it never shows a best plan the interval table still holds but raw has already purged.	Direct: Darling-only disclosure the interval-table read needs. Lite always leaves it null.
PerformanceMonitor.Analysis/FactAdvice.cs (+7/-1)	Appends "(it last ran N days ago)" to the PLAN_REGRESSION advice sentence when best_plan_age_days is present.	Direct: the interval table (Darling) and Lite's already-real 14-day window both mean the best plan on record can be up to two weeks old. The advice text now says so, instead of implying it is recent.
PerformanceMonitor.Analysis/FactRemediation.cs (+15/-1)	Parses best_plan_last_seen off the drill-down row into a new BestPlanLastSeenUtc on ForcePlanTarget.	Direct: plumbing for the bot gate below.
PerformanceMonitor.Analysis/RemediationAction.cs (+7/-1)	Adds BestPlanLastSeenUtc to the ForcePlanTarget record, defaulted for wire compatibility.	Direct: same plumbing.
PerformanceMonitor.Analysis/ForcePlanBotPolicy.cs (+23)	Adds ReasonBestPlanStale and MaxBestPlanAgeDays = 4. The unattended force-plan bot now blocks on a target whose best plan last ran more than 4 days before the pass.	Direct: raw Query Store's 4-day retention used to cap "best plan" age automatically. Both the interval table and Lite's real 14-day window remove that cap. The bot's own automation scope is pinned back to the 4-day regime it was scored in. The advisory (display-only) surface still shows an older best plan, with its age, to a human.
PerformanceMonitor.Collectors/CollectorContext.cs (+9)	Adds QueryStoreIntervalMisses, a per-cycle count of batches that stored raw rows but missed Darling's interval table and were queued for replay.	Direct: Darling-only collector-side observability for the interval table's coverage gaps. Lite leaves it at 0, since it has no such table.

All eight are #3953-scoped. None looked unrelated enough to flag for reversion. None are mentioned in the ## CHANGELOG entry below, which as written only covers the Darling interval-table read. Whether the advice-text and force-plan-bot behavior changes need their own CHANGELOG line is a coordinator call before this leaves draft.

One correction to S1's description while verifying it: the Lite dedup CTE does not use array_agg(...). Like the text above, it is ROW_NUMBER() OVER (... ORDER BY collection_time DESC, execution_count DESC) filtered to rn = 1, with the same effect through a different construct. The description was off. The scale claim next to it is the one this lane was sent to actually measure.

Measuring Lite's twin read at the top of its scale

Seeded Lite's own query_store_stats DuckDB table through the real DuckDbInitializer.InitializeAsync() / CreateArchiveViewsAsync() (production schema, not hand-transcribed), on the same connection string Lite itself uses (memory_limit=1GB;checkpoint_threshold=1GB). The seed:

• 5 servers x 3 databases x 100 queries (Lite's query_store collector caps at the top 100 queries per database).
• x 2 plans per query.
• x 336 hourly Query Store intervals (14 days).
• x 12 re-collections per open interval. Lite's default query_store schedule polls every 5 minutes. A Query Store interval is 60 minutes, so 60 / 5 = 12.

That is 12,096,000 raw rows, of which 2,419,200 are the one server's slice a single server_id = $1 call reads. Same shape as S1's re-measurement above, scaled down to Lite's real ceiling of a handful of servers instead of 43.

EXPLAIN ANALYZE on both reads against that seed, server 1, params matching production exactly, such as context.TimeRangeStart.AddDays(-14):

read	wall time	rows scanned	spill
FACT: CollectPlanRegressionFactsAsync (DuckDbFactCollector.QueryPerf.cs), unconditional every pass, Lite's actual parity twin of PlanRegressionSql (4 runs)	530-981 ms	2,419,200 (TABLE_SCAN, confirmed in the plan)	none: no SPILL / external / temp-file text anywhere in the plan
DRILLDOWN: CollectRegressedQueries (DrillDownCollector.Queries.cs), filtered to the fact's own offenders, the only path the running app actually takes (see below) (1 run)	373 ms (0.308s per EXPLAIN ANALYZE's own reported total)	same scan, filtered early	none
DRILLDOWN, unfiltered: the SQL's $5 IS NULL branch (4 runs)	1,347-1,622 ms	2,419,200, no early filter	none

The unfiltered branch is real SQL, but its only call site never reaches it. DrillDownCollector.cs gates the call on pathKeys.Contains("PLAN_REGRESSION"). The fact sets context.PlanRegressionOffenders to that pass's own (<=20) results before deciding whether to create the finding the drill-down keys off. Every real invocation already carries a non-empty offender list, so in the running app $5/$6 are never actually null.

The 1.3-1.6s number is real, but it is dead code today, not the steady-state cost. I did not change that: it is out of scope for this lane. Noted here for whoever next touches this file.

Peak memory needed a different check. duckdb_memory(), polled every 40ms during each run, read 0.0 in every case, including runs known to use real memory (below). It is not a trustworthy instrument for this workload on this version, so I did not rely on it.

Instead, a calibration sweep throttled memory_limit on the identical seeded data and the identical FACT query. This is not Lite's real 1GB setting. It completed at 300MB (2 runs, 1,673-1,886 ms, still no spill). It hard-failed at both 150MB and 75MB (2 runs each): a DuckDBException: Out of Memory Error, not a graceful spill, every time at "143.0-144.1 MiB used". That pins this read's actual peak working set between about 143MB and 300MB. Lite's real budget, 1GB, sits roughly 3.4x-7x above that range.

Decision: do not port

Both of Lite's real code paths finish in under a second: FACT at 530-981ms, the drill-down's only reachable path at 373ms. Neither spills or comes close to Lite's 1GB ceiling, even in the unreachable worst case. That is short of the brief's "over about 1s, or a large spill" bar for porting.

It also matches the structural reason S1 gave without a number behind it. Every Lite call reads exactly one server's slice under a fixed 1GB budget on a vectorized columnar engine. It never reads several servers' slices competing for one shared 4MB work_mem, the way 43 concurrent Postgres passes do on Darling's store. The conclusion is unchanged from S1's. It now rests on a measurement instead of an assertion about scale.

Test plan

• dotnet build Darling/Darling.Tests/Darling.Tests.csproj: 0 Warning(s), 0 Error(s).
• Every touched test class run individually against the rig: QueryStoreIntervalLatestRungTests,
  CollectionCaveatsRungTests, CheckpointsTimedRungTests, PostmasterStartTimeRungTests,
  ViewerSchemaVersionGateTests, QueryStoreIntervalLatestWriterTests,
  PlanRegressionIntervalTableEquivalenceTests, PgFactCollectorTests, DarlingAnalysisPipelineTests,
  ForcePlanBotPolicyTests, PlanRegressionReadShapeTests, ViewerCollectionCaveatsGateTests,
  OversizedPlanBacklogPins. All pass, including the live (DARLING_TEST_PG) cases.
• Full Darling.Tests suite, run once: 13,763 total, 2 failed, 696 skipped (no live Postgres or
  pg-runtime for that pass). Both failures were the line-ending drift and the ViewerCollectionCaveatsGateTests
  arity assumption above. Both were fixed and individually reverified passing afterward.
• Did not re-run the complete 13,763-test suite a second time after the two fixes, given the lane's time
  budget. The affected classes were reverified individually. The rest of the suite did not
  depend on either fix (a line-ending pin and one test's own argument construction).
• Field-shaped EXPLAIN (ANALYZE, BUFFERS) re-measurement above, on a throwaway probe database, not
  the suite's darlingtest.
• Rig at port 55961, C:\GitHub\worktrees\rig-s1, stopped and deleted after use.

CHANGELOG entry

SECTION: Fixed
ENTRY:
- **PLAN_REGRESSION no longer deduplicates the whole raw Query Store slice every pass** ([#4208]). On a store ingesting many servers' Query Store data, the plan-regression check was the single heaviest statement in the store. It spent most of its time sorting and spilling the server's entire raw Query Store history to temp files. It now reads a collector-maintained table holding only each interval's latest snapshot. It falls back to the old full read only when coverage is incomplete for a server. Findings are unchanged: the same regressions, the same snapshots.
- **Force-plan bot no longer acts on a best plan older than 4 days** ([#4208]). The unattended force-plan bot now gates on `ForcePlanBotPolicy.MaxBestPlanAgeDays` = 4. A target whose best plan has not been seen in more than 4 days is blocked with reason `best_plan_stale`. This prevents a force on a plan the server evicted. The gate value is pinned by `ForcePlanBotPolicyTests`.
REF:
[#4208]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/4208