Repository navigation
#4186 gate follow-ups: release needs a second start after an outage, plus three smaller holds #4300
Description
Found by the round-3 review of #4186 (M2, L1, L2 and L3). None of these loses rows. Each one holds a purge or a retention job longer than it needs to, or tells the operator to act when nothing needs doing.
1. After an outage of more than a day across the upgrade, the gate holds until the second start (M2)
Where:
TimescaleSupport.MaterializationHoles.csnear lines 507-512: the walk skips an aggregate with no materialized bucket.DarlingWorker.csnear line 1940: the walk runs once per start.TimescaleSupport.csnear line 6059: the doc says the gate "releases automatically, with no manual step".
- A 3.8.0 store is stopped for 2 days, then takes the Legacy trio off the refresh grid (#3653) #4186 build.
- At the first start, the successors are new and empty, so the walk skips them, and the gate reads Short.
- After the successor's first refresh, the seam holds the last 2 hours or so before the stop. Each hourly check reads Short, and nothing repairs the seam until the next start.
Until then, the raw purge for query_stats and procedure_stats stays held, and raw grows by a day of collection each day. A store that seldom restarts needs a manual step: a restart, or --backfill-rollups.
Fix options: when the probe finds a seam, run only the seam window, with the same cap, from the hourly retention check. Or run the walk once more after the successor's first materialization. If neither, change the doc to say the release needs a second start.
2. Every upgrade from 3.8.0 holds the raw purge again, with a warning to backfill (L1)
Where: TimescaleSupport.cs near line 6110 (COALESCE(s.mn, 'infinity'::timestamp)) and the warning near line 6676.
At the upgrade start the successors are empty, so the probe window runs from the legacy's last bucket plus 1 hour to infinity. It always finds rows there, so the gate reads Short, holds the purge, and logs "RE-HELD ... Backfill that consumer". The successor's first refresh clears it within about 2 hours. But the warning sends the operator to a heavy command for a state that clears by itself.
Fix: when s.mn is NULL, end the probe window at now() - HourlyRefreshStartOffset. The successor's first policy window reaches every newer row, and the purge drops only chunks older than 4 days.
3. A repaired seam older than 3 days never reaches the successor daily (L2)
Where: TimescaleSupport.MaterializationHoles.cs near lines 525-537 (LegacyOf maps only the hourly successors) and TimescaleSupport.cs near lines 6213-6215 (each successor hourly's retention waits on its successor daily).
After a 5-day outage, the second start repairs the seam into query_stats_interval_hourly. Those hours are outside the daily's 3-day policy window and below the daily's floor, so query_stats_interval_daily never gets them. The hourly's oldest bucket is then older than the daily's. So the hourly's 90-day retention reads Short, and it stays held, with a warning, until someone runs --backfill-rollups. The hourly keeps the rows.
Fix: give each successor daily a seam window that starts at its hourly's floor. Or make the seam repair also refresh the dependent daily over the repaired days.
4. The probe filter for the db-grain slot is narrower than that rollup's WHERE (L3)
Where: TimescaleSupport.cs near line 6111 uses IntervalHonestSourceFilter for every slot. query_stats_db_interval_hourly (near lines 2147-2148) also requires delta_worker_time IS NOT NULL, and the walk copies that WHERE.
The probe finds a seam row with a NULL delta and a non-zero interval, but the walk never treats it as a hole. So the purge holds forever. That is the safe direction. The current collector always writes a delta, so only old rows can do this.
Fix: build each slot's filter from MaterializationHoleSourceFilterFor over that rollup's CREATE, so the probe and the walk cannot differ.
Activity
Seam repair walks newest-first and stops at the first failed range (#…
Restore the summary opening in the raw-purge gate doc (#4300)
Don't re-hold the raw purge for an empty successor mid-upgrade (#4406)
erikdarlingdata commentedon Sep 26, 2026
Claude posting for Erik Darling.
#4406 is merged (item L1). An upgrade with no outage no longer holds the raw purge again while the successor rollup is still empty; a real outage in that window still holds it.
Fix the seam-only repair test fixture, the single-start pin's RED and…
Add live pins for the raw purge gate's per-slot source floor (#4300)
Give the hourly seam repair its own budget, skip it while the start-p…
Pin a deferred seam keeping the gate held, with no repair epoch stamp…
Keep the interval-honest source filter to the two relations whose suc…
After an outage across an upgrade, the raw purge gate releases within…
erikdarlingdata commentedon Sep 26, 2026
Claude posting for Erik Darling.
Item 1 is done: #4430 is merged (dc2027b). After an outage across an upgrade, the hourly retention pass repairs the legacy/successor seam, so the raw purge gate can release within the hour on a running store, with no second start. The repair has its own 2-minute budget, so it never starves the sweep or the purge trigger. It skips while the start-path repair is running, and it never stamps the repair epoch.
erikdarlingdata commentedon Sep 26, 2026
Claude posting for Erik Darling.
Item 1 (release needs a second start after an outage) is on dev in #4430. The hourly retention evaluation now runs a seam-only repair (legacy-paired targets only, the same cap and filter) before it re-judges coverage. So after an outage across the upgrade, the raw purge gate releases within the hour on a running store, with no second start. Item 3 (repaired days reaching the successor daily) is in progress, and item 4 (a per-rollup source floor) is in #4432.
The raw purge gate judges each rollup against the rows that rollup ca…
erikdarlingdata commentedon Sep 26, 2026
Claude posting for Erik Darling.
Item 4 is done: #4432 is merged (8caf5a0). The raw purge gate now judges each query_stats and procedure_stats rollup against the oldest raw rows its own filter admits, so an hour of CPU-unknown rows at raw's floor can no longer hold the purge forever. The query-grain rollup still holds until it covers that hour. Every other relation keeps its unfiltered floor.
Add DailyBucketsChained to the hourly seam-repair log line and retry …
Guard the daily chase against a still-holed hourly (#4300)
Add live pin for the cap-split daily-chase defer guard (#4300)
Pin the daily chase's hole scan on the catch-up path (#4300)
After a long outage across an upgrade, the seam repair also refreshes…
erikdarlingdata commentedon Sep 26, 2026
Claude posting for Erik Darling.
All four items are done, so this issue is closing: