Skip to content

Live compression tests have two ~200-second midnight blind windows from now()-relative chunk seeding #1972

Closed

Description

@erikdarlingdata

Three live compression tests seed chunks with now()-relative timestamps, which makes chunk PLACEMENT depend on the time of day the suite runs. There are two ~200-second windows around midnight UTC where the tests fail deterministically - not flaky in the reused-database sense, just blind windows the clock walks through every day.

Evidence (caught live tonight)

EndToEnd_CompressionRun_TakesEligibleChunksAndLeavesYoungOnes_AgainstDevPostgres - Assert.Equal() expected 1, actual 2, at TimescaleSupportTests.cs:740 (the UNcompressed chunk count). A rerun outside the window passed with no code change.

Mechanism

The tick tables are day-aligned 1-day chunks. The young-chunk seed is:

INSERT ... SELECT now()::timestamp - (g || ' seconds')::interval ... generate_series(1, 200)

which writes rows spanning [now-200s, now-1s]. Run between 00:00:00 and 00:03:20 UTC, that span straddles the day boundary, so the "one young chunk" is actually TWO chunks (yesterday's tail + today's). Total chunks = 3, uncompressed = 2, and line 740 fails exactly as observed.

The old-chunk seeds have the symmetric problem in the other direction:

INSERT ... SELECT now()::timestamp - INTERVAL 'N days' + (g || ' seconds')::interval ...

spans [now-Nd+1s, now-Nd+200s]. Run between 23:56:40 and 23:59:59 UTC, the +200s end crosses midnight N days back, splitting the old data into two chunks - both closed, both past the eligibility delay, so the policy compresses BOTH and the compressed-count assert (line 739) fails instead.

Affected sites (all in Darling/Darling.Tests/TimescaleSupportTests.cs)

Line Test Window
735 EndToEnd_CompressionRun_TakesEligibleChunksAndLeavesYoungOnes 00:00:00-00:03:20 (hit tonight)
733 same test, old-chunk seed 23:56:40-23:59:59
798 EndToEnd_CompressionRun_OneUncompressibleChunkDoesNotBlockTheRest (parameterized daysBack) 23:56:40-23:59:59
986 EndToEnd_CompressionActivity_ReportsBacklogAndClearsAfterARun 23:56:40-23:59:59

Nightly scheduled darling-pg runs and evening PR pushes can both land in these windows; tonight's did.

Fix direction

Anchor the seeds to midday instead of now():

date_trunc('day', now()) - INTERVAL 'N days' + INTERVAL '12 hours' + (g || ' seconds')::interval

A +/-200s span around midday can never cross a day boundary, chunk placement becomes deterministic for any run time, and no assertion changes are needed. The young chunk keeps its "still open, still young" property (its chunk is today's) - worth one careful look at the range_end > now() - CompressAfterDays assert at line 749 while touching it, but midday-today satisfies it with hours to spare.

Mechanical fix, four lines plus comments; the only care point is keeping the young/old semantics each test pins.

Activity

  1. added 2 commits that reference this issue on Aug 1, 2026
  2. erikdarlingdata commented on Aug 1, 2026

    @erikdarlingdata
    OwnerAuthor

    Shipped in #1976. All four seeds now anchor to midday in naive-timestamp space: date_trunc('day', now()::timestamp) plus 12 hours, minus the day offset - deliberately NOT the timestamptz form the issue sketched, because the column is naive timestamp and the timestamptz form would reintroduce session-zone and DST dependence on the way back in. Proven on a live TimescaleDB rig by entering both windows on demand with literal anchors: the old forms reproduce expected-1-actual-2 in both directions, the midday forms give 1 and 1, and a sweep across all 1440 minutes of the day splits on 3 minutes for each now()-relative form and zero for midday. The young chunk keeps its pinned still-open-still-young property at every hour (range_end = tomorrow midnight). Gated-live suite run with ZERO skips locally to prove the tests actually executed. The other chunk-count test files were checked and are safe by construction (fixed literal dates in one, before/after same-seed comparison in the other). Note for readers: green CI on this PR is necessary but not the proof - CI ran outside the windows, where the buggy code was also green; the rig sweep is the proof.

  3. added 2 commits that reference this issue on Aug 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions