Skip to content

#4186 gate follow-ups: release needs a second start after an outage, plus three smaller holds #4300

Closed

Description

@erikdarlingdata

Found by the round-3 review of #4186 (M2, L1, L2 and L3). None of these loses rows. Each one holds a purge or a retention job longer than it needs to, or tells the operator to act when nothing needs doing.

1. After an outage of more than a day across the upgrade, the gate holds until the second start (M2)

Where:

  • TimescaleSupport.MaterializationHoles.cs near lines 507-512: the walk skips an aggregate with no materialized bucket.
  • DarlingWorker.cs near line 1940: the walk runs once per start.
  • TimescaleSupport.cs near line 6059: the doc says the gate "releases automatically, with no manual step".
  1. A 3.8.0 store is stopped for 2 days, then takes the Legacy trio off the refresh grid (#3653) #4186 build.
  2. At the first start, the successors are new and empty, so the walk skips them, and the gate reads Short.
  3. After the successor's first refresh, the seam holds the last 2 hours or so before the stop. Each hourly check reads Short, and nothing repairs the seam until the next start.

Until then, the raw purge for query_stats and procedure_stats stays held, and raw grows by a day of collection each day. A store that seldom restarts needs a manual step: a restart, or --backfill-rollups.

Fix options: when the probe finds a seam, run only the seam window, with the same cap, from the hourly retention check. Or run the walk once more after the successor's first materialization. If neither, change the doc to say the release needs a second start.

2. Every upgrade from 3.8.0 holds the raw purge again, with a warning to backfill (L1)

Where: TimescaleSupport.cs near line 6110 (COALESCE(s.mn, 'infinity'::timestamp)) and the warning near line 6676.

At the upgrade start the successors are empty, so the probe window runs from the legacy's last bucket plus 1 hour to infinity. It always finds rows there, so the gate reads Short, holds the purge, and logs "RE-HELD ... Backfill that consumer". The successor's first refresh clears it within about 2 hours. But the warning sends the operator to a heavy command for a state that clears by itself.

Fix: when s.mn is NULL, end the probe window at now() - HourlyRefreshStartOffset. The successor's first policy window reaches every newer row, and the purge drops only chunks older than 4 days.

3. A repaired seam older than 3 days never reaches the successor daily (L2)

Where: TimescaleSupport.MaterializationHoles.cs near lines 525-537 (LegacyOf maps only the hourly successors) and TimescaleSupport.cs near lines 6213-6215 (each successor hourly's retention waits on its successor daily).

After a 5-day outage, the second start repairs the seam into query_stats_interval_hourly. Those hours are outside the daily's 3-day policy window and below the daily's floor, so query_stats_interval_daily never gets them. The hourly's oldest bucket is then older than the daily's. So the hourly's 90-day retention reads Short, and it stays held, with a warning, until someone runs --backfill-rollups. The hourly keeps the rows.

Fix: give each successor daily a seam window that starts at its hourly's floor. Or make the seam repair also refresh the dependent daily over the repaired days.

4. The probe filter for the db-grain slot is narrower than that rollup's WHERE (L3)

Where: TimescaleSupport.cs near line 6111 uses IntervalHonestSourceFilter for every slot. query_stats_db_interval_hourly (near lines 2147-2148) also requires delta_worker_time IS NOT NULL, and the walk copies that WHERE.

The probe finds a seam row with a NULL delta and a non-zero interval, but the walk never treats it as a hole. So the purge holds forever. That is the safe direction. The current collector always writes a delta, so only old rows can do this.

Fix: build each slot's filter from MaterializationHoleSourceFilterFor over that rollup's CREATE, so the probe and the walk cannot differ.

Activity

  1. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    #4406 is merged (item L1). An upgrade with no outage no longer holds the raw purge again while the successor rollup is still empty; a real outage in that window still holds it.

  2. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    Item 1 is done: #4430 is merged (dc2027b). After an outage across an upgrade, the hourly retention pass repairs the legacy/successor seam, so the raw purge gate can release within the hour on a running store, with no second start. The repair has its own 2-minute budget, so it never starves the sweep or the purge trigger. It skips while the start-path repair is running, and it never stamps the repair epoch.

  3. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    Item 1 (release needs a second start after an outage) is on dev in #4430. The hourly retention evaluation now runs a seam-only repair (legacy-paired targets only, the same cap and filter) before it re-judges coverage. So after an outage across the upgrade, the raw purge gate releases within the hour on a running store, with no second start. Item 3 (repaired days reaching the successor daily) is in progress, and item 4 (a per-rollup source floor) is in #4432.

  4. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    Item 4 is done: #4432 is merged (8caf5a0). The raw purge gate now judges each query_stats and procedure_stats rollup against the oldest raw rows its own filter admits, so an hour of CPU-unknown rows at raw's floor can no longer hold the purge forever. The query-grain rollup still holds until it covers that hour. Every other relation keeps its unfiltered floor.

  5. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    All four items are done, so this issue is closing:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions