The start-up hole scan probes each bucket once instead of joining the whole materialization and the whole source table (#3933)

[erikdarlingdata]

Closes #3933.

Why

TimescaleSupport.MaterializationHoleScanSql is the scan behind the start-up materialization-hole repair (#3731). RepairMaterializationHolesAsync runs it for every one of the 23 registered aggregates on every service start, and again after each repair.

Its doc said the two correlated probes are "each an index range ... so cost follows the number of buckets scanned and not the size of either table". The planner did something else. Written as a bare NOT EXISTS and EXISTS, both are pulled up into joins, and neither join can push the per-bucket bound into its scan:

• The materialization side became an anti-join over the WHOLE materialization hypertable: every server, its whole retention, with every compressed chunk decompressed and sorted by bucket. On DARLING01 this was a merge anti-join, read once. On the rig the planner chose a nested-loop anti-join that re-read the materialization once per bucket: 92 × 776K rows.
• The source side became a semi-join over a Materialize of the WHOLE source table, read through once per bucket with no materialized row.
  • Every outage hour inside the scan span is such a bucket, and it holds no source row, so each one reads the entire Materialize without a match.
  • The first one builds it, spilling past work_mem.
  • This is the Web viewer server page: the two daily-summary reads are ~26 of the Overview tab's ~28 s on a busy production primary, one at the viewer role's 15 s timeout — raw CTEs over the whole window plus a per-call rollup-coverage probe #3905 defect in the daily summary's not-carried probes (fixed in The daily summary's month read probes each day once and hashes its query count, and its coverage gate is cached, so the server page's Overview stops waiting on it (#3905) #3950), but here without a server_id, so the whole fleet's rows are read, not one server's.

Confirmed on DARLING01, a real store with a few outage hours in its 35-day baseline span. The wait-stats baseline's scan materialized all 9,785,051 wait_stats rows to disk (21,501 temp blocks written, 64,503 read) and read them four times: 4.5 s for one aggregate. The whole start-up scan across 23 aggregates was 5.9 s. On a 43-server store with 30 days of raw data, the same shape is several times the rows on every start.

What changes

• The fence: each probe carries OFFSET 0. PostgreSQL's simplify_EXISTS_query refuses to pull a sub-select with an OFFSET up into a join ("OFFSET 0 ... traditionally is used as an optimization fence"). So each probe runs as a SubPlan, once per bucket, with the chunk it needs picked at run time.
• Probe order: the candidate buckets are fenced the same way, so the source is probed only for the buckets the materialization probe found empty.
• Results are unchanged: the same buckets in the same order, checked against the old text as an oracle, below.
• Docs: the method doc records the finding, and the type summary's "both are index probes" now points at the fence that makes it true.
• Scope: no store migration, and no change to the pass's logic, cap, logging or tally.

Test plan

• DARLING01, read-only as -U admin with default_transaction_read_only = on. Every one of the 23 targets over the span the pass itself would scan, old text vs new, EXPLAIN (ANALYZE, BUFFERS, TIMING OFF), median of 2. Same buckets on every target.

  aggregate	before	after
  wait_stats_interval_baseline	4,662 ms (64,503 temp blocks read)	22.5 ms
  perfmon_interval_baseline	999 ms (401,386 buffers)	23.8 ms
  memory_baseline	76.8 ms	8.4 ms
  deadlock_baseline	61.2 ms	4.3 ms
  session_stats_baseline	41.7 ms	6.2 ms
  query_store_stats_interval_daily	14.2 ms	0.1 ms
  the other 17	≤ 6 ms each	≤ 1.4 ms each
  all 23: one start's scans	5,876 ms	75 ms

• Rig, one busy server (571K raw query_stats rows, 807K hourly rollup rows) with a six-hour outage and a two-hour tail planted: query_stats_hourly's scan went from 3,886 ms to 6.3 ms, finding the same two buckets.

• New tests:

  • MaterializationHoleScanShapeSqlTests: for every target, three OFFSET 0, with the materialization probe inside the fenced candidate subquery and the source probe filtering its output.
  • MaterializationHoleScanShapeLiveTests (rig), on a two-day seed with a tail, an outage, a restart-only hour, covered hours and a day the daily skipped:
    • oracle: the pre-The start-up materialization-hole scan plans its per-bucket probes as a join over the whole source span, so a post-outage start rescans every source row once per empty bucket #3933 text against the new one on the legacy hourly, the interval-honest successor, the query baseline and the hierarchical daily. Identical buckets, plus the expected ones as values.
    • plan: no Materialize and no nested-loop, merge or hash join. The materialization probe runs 48 times (the span) and the source probe 6 times (the tail plus the outage).

• Red on the old shape: with the product file reverted, the fence pin and the plan test fail (the old plan's nested-loop joins). The oracle test passes trivially then, as it must.

• Existing pins updated deliberately (text, not intent): MaterializationHoleRepairTests.TheSourceFilter_IsTheAggregatesOwnWhere_Verbatim_OrEmpty. Both repair live tests (tail, outage, forced-refresh hole, successor filter, cap and deferral tally) pass on the new text.

• Full Darling.Tests against the rig with DARLING_TEST_PG: 12,703 total, 0 failed, 24 skipped (env-gated managed-runtime, PostgreSQL-target, live SQL Server and symlink tests), on dev db93e292; after rebasing onto 8395bece (A restart no longer raises a false Store Checkpointer Pressure warning or reads as a WAL-forced checkpoint on a monitored PostgreSQL server (#3955) #3969, unrelated), the hole-repair, store-convergence, not-carried and doc-hygiene classes were re-run (110 tests, 0 failed)

🤖 Generated with Claude Code

https://claude.ai/code/session_018Z4T51J4E6L8fjriPHpsif