issue-3653 A6 lane LB: the 3 interval-honest successor dailies + backfill runbook

[erikdarlingdata]

Part of #3653 (A6, lane LB).

What this adds

This change adds three continuous aggregates that roll up by day:

• query_stats_interval_daily, built from query_stats_interval_hourly
• procedure_stats_interval_daily, built from procedure_stats_interval_hourly
• query_stats_db_interval_daily, built from query_stats_db_interval_hourly

Each one reads an interval-honest hourly successor (#3653 Q12), not the legacy hourly rollup. A daily built from the legacy hourly inherits the restart-row contamination of the legacy at the day grain. These three let the daily tier stop inheriting it in a later lane. Each one is created WITH NO DATA. Its columns are the SUM, MIN and MAX re-aggregation of the legacy daily, plus sum(sample_interval_seconds_sum).

This change does not touch legacy refresh or compression policies, readers, retention gates or raw-purge gates. Lanes LA and LC own those.

Registration

• DailyAggregates, the list that the creation sweep walks, grows from 7 to 10.
• RollupViews, the list behind the coverage probe and the --backfill-rollups plan, grows from 16 to 19. Each new row names the successor hourly as Source, uses bucket as SourceTimeColumn, and uses the DailyBucket width. RollupBackfill.Targets derives from this list and orders by source depth, so it needed no change.
• The availability probe (RollupProbeSql, DetectRollupsAsync and RollupAvailability) gets three new flags and a WithoutIntervalDailies shorthand for tests. The probe checks existence only. There is no migration and no version gate.
• SupersededDailyRollups is a new registry with one (LegacyDaily, SuccessorDaily, SuccessorHourly) row for each pair. No reader uses it yet. Lane A6-LA-8 replaces s_supersededDailyRollupsStub from issue-3653 A6 lane LA: stitched rollup reads, every hourly reader routes legacy→successor at the successor floor #4182 with it.

Compression

The three new dailies get a refresh policy but no compression policy. CompressionDeferredUntilFreeze holds their names, and AggregateCompressionTargets leaves them out. The daily compression band is full at 23 members, so the target count stays at 23 (9 hourly + 10 daily + 7 baseline - 3). The three dailies come after the other dailies in the list, so no existing aggregate changes its compression slot. The freeze in lane LC retires the legacy dailies from the band, and then LC removes this set.

Runbook

docs/runbooks/a6-successor-daily-backfill.md tells an operator how to backfill the three dailies. The backfill is restricted to these three views. Each daily starts at date_trunc('day', successorHourlyFloor). The first day is partial by design, because a daily cannot hold more history than its hourly source. The runbook includes the verification query from the design.

Tests

• The count pins now derive their values. RollupViews.Length is 19, DailyAggregates.Length is 10, and MaterializationHoleRepairTests expects 26 targets.
• TimescaleAggregateCompressionTests asserts CompressionDeferredUntilFreeze.Count == 3. Its end-to-end test expects the sweep to create HourlyAggregates + DailyAggregates + BaselineAggregates + OffGridAggregates aggregates, which is 27 on a fresh store.
• SuccessorDailyLiveTests is new. It has three live tests:
  1. The ensure sweep creates the three dailies with one refresh policy each and no compression policy. The sweep must log no Warning. A second sweep changes nothing.
  2. The test seeds about four days of raw rows with no restart rows. For each key and each day, each successor daily has the same sums as its legacy daily.
  3. The test backfills the three successor hourlies, then the three successor dailies. The verification query of the runbook then holds (daily_floor <= date_trunc('day', hourly_floor)).

Windows-only CI failures (fixed in the tests)

Two live tests failed only on Windows CI. Both causes were in the test setup. The product code did not change.

1. SuccessorDailyLiveTests did not call TimescaleSupport.EnsureCollectionLogHypertableAsync before the ensure sweep. So collect.collection_log stayed a plain table, and TimescaleDB rejected collection_health_hourly with 0A000: invalid continuous aggregate view. The sweep then built 26 of 27 aggregates. The worker calls ConvertToHypertablesAsync, then EnsureCollectionLogHypertableAsync, then EnsureContinuousAggregatesAsync. The three test setups now make the same calls in the same order.
2. IntervalHonestHourlyRollupLiveTests simulates an upgraded store. It drops query_stats_interval_hourly with DROP MATERIALIZED VIEW ... CASCADE. The CASCADE also drops query_stats_interval_daily, because that daily reads from the hourly. The test recreated only the hourly, so the coverage probe failed with 42P01: relation "collect.query_stats_interval_daily" does not exist. The test now recreates the daily too. Product code never drops the successor hourlies.

The earlier passes (LB-3, LB-4 and LB-5) did not reproduce these failures. The timescale/timescaledb docker image has the timescaledb extension in template1, so every scratch database on that rig started with the extension. The bundled runtime that Windows CI uses does not do that. When LB-6 dropped the extension from template1, both failures reproduced on the first local run.

The theories in the earlier notes and in the LB-5 comment are wrong and need no follow-up. They are a TimescaleDB 2.30.1 validation change, a catalog visibility race, a Windows-only continuous aggregate defect, and the INTERVAL '1 hour' spelling in collection_health_hourly. The last commit removes the diagnostic output that LB-4 added to IntervalHonestHourlyRollupLiveTests. The lane-by-lane notes are in the edit history of this description.

Verification

• The client-site loop seat checked 993640bcf before it armed this PR. It ran 294 live tests with 0 failures on a TimescaleDB 2.30.1 rig with the extension dropped from template1. On that rig, the full suite had 214 failures, the same set as the dev base.
• CI on 993640bcf passed every check except review. The three Windows Darling PG tests shards ran with DARLING_TEST_PG set: 13,705 tests, 0 failed.
• A local session rebased the branch onto dev after issue-3653 A6 lane LA: stitched rollup reads, every hourly reader routes legacy→successor at the successor floor #4182 merged (b6bd7630), with no conflicts. On the rebased tree, the Darling test classes that issue-3653 A6 lane LB: the 3 interval-honest successor dailies + backfill runbook #4181 or issue-3653 A6 lane LA: stitched rollup reads, every hourly reader routes legacy→successor at the successor floor #4182 touch passed without a live store. The result was 316 tests, 0 failed and 24 live tests skipped.
• CI on the rebased head c7a95a142 passed every check except review. The three Windows Darling PG tests shards ran with DARLING_TEST_PG set: 13,733 tests, 0 failed. The four Lite tests shards ran 5,311 tests, 0 failed.

The review check fails on every PR to dev. It started to fail on 2026-09-19. The claude-review.yml on dev differs from the copy on main, the default branch, so the review action does not run. The client-site loop seat ignores this check until a release syncs main.

Handoff note

The client-site agents stopped at 23:40Z on 2026-09-24. The client-site loop seat had armed this PR on three conditions: a clean rebase, green CI and no new red. Erik handed the PR to a local session, which did the rebase, the cleanup commit and this description. The cleanup commit is outside the arm of the loop seat, and Erik approved it before auto-merge was turned on.

CHANGELOG entry

Section: Added

• Three daily rollups built from the interval-honest hourly rollups ([Brains-review campaign: deferred structural residue (from #3538 / #3539 / #3540 / #3541) #3653]) - The store now creates query_stats_interval_daily, procedure_stats_interval_daily and query_stats_db_interval_daily. Each one reads its interval-honest hourly rollup, which does not count restart rows, instead of the legacy hourly rollup. They start empty, and no reader uses them yet. They have a refresh policy but no compression policy until a later change retires the legacy dailies. docs/runbooks/a6-successor-daily-backfill.md tells an operator how to backfill them. A store now has three more continuous aggregates.

[erikdarlingdata]

Lane A6-LB-5 handoff (ran out of time budget mid-investigation; no code changes made)

Status: investigation only, no commits. This lane found the Step-A answer with certainty, made progress on Step B, but hit the context/time wall before writing the fix. Filing this as a comment (branch fix/3653-a6-successor-dailies has no new commits from me) so the next lane doesn't redo the legwork.

(2) The 42P01 coverage-probe failure — test-connection finding

The coordinator's hypothesis is WRONG. I read IntervalHonestHourlyRollupLiveTests.RestartRow_… end to end. Both the ensure sweep (TimescaleSupport.EnsureContinuousAggregatesAsync(connection, …)) and the coverage probe (TimescaleSupport.DetectRollupCoverageAsync(dataSource, …)) run against the same scratch database: connection is opened on scratch.ConnectionString, and dataSource is NpgsqlDataSource.Create(scratch.ConnectionString) — the identical connection string, same scratch DB, not the shared DARLING_TEST_PG database. This is not a connection-string/test bug.

The LB-4 diagnostics (already on the branch) prove it from the CI run itself (the Windows PG-tests shard-1 job on PR #4181's latest run): RollupAvailability.AllPresent = True (so the probe's own to_regclass check says query_stats_interval_daily exists) and timescaledb_information.continuous_aggregates for schema collect lists query_stats_interval_daily in the catalog. Yet the very next statement — RollupCoverageProbeSql, which does (SELECT min(bucket) FROM collect.query_stats_interval_daily) — throws 42P01 "relation ... does not exist" at byte position 1027, which I hand-verified is exactly the query_stats_interval_daily clause in the generated SQL text (confirmed by reconstructing RollupCoverageProbeSql's column list in Python: the substring starts at offset 1034, i.e. inside the same multi-column SELECT at the position Postgres reported).

So: the relation is visible to to_regclass and to the catalog view, but a bare FROM collect.query_stats_interval_daily in the SAME session, moments later, 42P01s. That is NOT a stale-schema/wrong-database story — to_regclass and information_schema/timescaledb_information reads are catalog-cache-coherent with a subsequent SELECT ... FROM in the same transaction/session on every PostgreSQL I've ever seen. The much more likely explanation, given item (1) below, is a Windows-only continuous-aggregate creation defect: on 2.30.1/Windows, CREATE MATERIALIZED VIEW ... WITH (timescaledb.continuous) for the hierarchical interval-honest daily successors may be registering the view in the TimescaleDB catalogs (so to_regclass/continuous_aggregates sees it) without the underlying relation being fully materializable/visible for a plain SELECT, OR the daily's underlying materialization hypertable failed under the hood while the wrapper view got created. I did not get to prove this precisely — see "what I did NOT verify" below.

Next step for whoever picks this up: add SELECT * FROM collect.query_stats_interval_daily LIMIT 0 (not just to_regclass) as a THIRD diagnostic line right after the two LB-4 already added, and also print pg_class.relkind/relispartition and any error from \d collect.query_stats_interval_daily equivalent (e.g. information_schema.views and pg_matviews) for this view specifically, on Windows CI, before the coverage probe runs. That will show whether the daily successor's CREATE partially failed in a way EnsureContinuousAggregatesAsync's try/catch treats as success (i.e., the CREATE statement returns OK but a hierarchical/internal step it depends on did not).

(1) collection_health_hourly — rejected clause, NOT YET ISOLATED to one expression, but reproduction is close

I stood up the CORRECT rig per the A6 rules (timescale/timescaledb:2.30.1-pg18, PG 18.6 base image, TimescaleDB 2.30.1 confirmed via SELECT extversion), built the schema by hand (collect.collection_log hypertable), and ran the exact, fully-expanded CreateCollectionHealthHourlySql text (with EnumeratedCollectorDriver.AbandonedByNotePredicateSql, AbandonedRunPredicateSql, and CollectorRuntimePrecondition.NamedSkipStatusSqlList substituted verbatim) against it on Linux (my Mac's docker is linux/arm64 TimescaleDB) — it succeeded. This matches the brief's statement that Linux docker 2.30.1 passes both failures; the defect is Windows-specific, so a Linux repro (even on the correct TimescaleDB version) cannot surface it. I did not get to push a temporary [Fact] diagnostic to CI (Step B) before the time/context wall, so the rejected clause is not yet isolated — that step still needs to run on Windows CI, per the brief's Step B plan (2 rounds of variant CREATEs with one suspect clause removed each: the time_bucket(INTERVAL '1 hour', ...) form specifically — note this aggregate uses time_bucket(INTERVAL '1 hour', collection_time), spelled with INTERVAL '1 hour' rather than the bare '1 hour' string literal every OTHER aggregate in this file uses (checked via grep -n "time_bucket(" — every other CAGG in TimescaleSupport.cs uses time_bucket('1 hour', ...); only this one, line 8593, uses time_bucket(INTERVAL '1 hour', ...)). That textual divergence is the most promising single lead for the next lane's Step B — it is worth trying the bare-string-literal spelling as the FIRST variant, since it would make this CREATE consistent with every other aggregate in the file and is a one-line, semantically-identical fix if it's the culprit.

What I did NOT verify

• Did not run on Windows CI at all (no diagnostic test pushed — ran out of budget before Step B).
• Did not isolate which single clause 2.30.1/Windows rejects for (1); the INTERVAL '1 hour' vs '1 hour' spelling difference is a lead, not a confirmed cause.
• Did not confirm my hypothesis for (2) (partial-success CREATE) with direct evidence; only ruled out the coordinator's original same-database/different-connection-string hypothesis with high confidence by reading the test source and the LB-4 CI output.
• No code changes were made; branch fix/3653-a6-successor-dailies is unchanged by this lane.
• Docker rig used locally (lb5-pg, PG 18.6 + TimescaleDB 2.30.1) was torn down; nothing left running.

Part of #3653 (A6, lane LB-5). No PR opened per brief (continuing existing PR #4181); this is a comment, not a body edit, to avoid clobbering other lanes' content under time pressure.