Skip to content

EnsureRetentionPolicies live test discards all 17 exception messages, so its intermittent 'expected 17, got 0' cannot be triaged #2818

Description

@erikdarlingdata

The observation

TimescaleSupportTests.EnsureRetentionPolicies_ConvergesAnOldHorizon_PreservingScheduledStateAndNextStart_AgainstDevPostgres failed once on a nightly with expected 17, got 0, then passed on re-run — on the same commit that had passed the same job ~30 minutes earlier on PR #2812.

It was called a flake. I could not determine the root cause, and I am not going to assert one. What follows is what I ruled out with evidence, and the reason the next occurrence will be just as undiagnosable unless one thing changes.

Ruled out

  • Static initialisation order. RetentionPolicies is built from RawTierCoverage and BaselineAggregates. Both are declared in the same class at lines 891 and 1912, before RetentionPolicies at 1949, so textual init order is correct and the collection cannot be empty. (An empty collection would have produced 0 with no error noise, which fit the symptom well — but it is not possible here.)
  • Within-assembly parallelism. TimescaleSupportTests carries [Collection("live-postgres")], so it is serialised against the other live classes.
  • Cross-job database sharing. Both build.yml and nightly.yml set DARLING_TEST_PG to Host=127.0.0.1;Port=5541, a per-runner instance. Two jobs cannot be stepping on one database.

What got 0 actually means

EnsureRetentionPoliciesAsync increments applied only after each relation's transaction commits. A return of 0 therefore means all 17 relations threw and every one landed in the per-relation catch — not that the loop was skipped.

Why it is undiagnosable, which is the real defect

That per-relation catch is:

catch (Exception ex) when (ex is not OperationCanceledException)
{
    logger?.LogWarning("Retention policy for {Relation} ({DropAfter}) failed - ...: {Message}", relation, dropAfter, ex.Message);
}

and every call site in this test passes null for the logger:

Assert.Equal(RetentionPolicyCount, await TimescaleSupport.EnsureRetentionPoliciesAsync(connection, null, ct));

So all 17 exception messages are discarded by the ?. and the only surviving evidence is the bare count. There is no capturing ILogger anywhere in Darling.Tests to pass instead.

This is the same shape as #2801 and #2816: a failure that reports a plausible-looking value while destroying the one piece of information needed to act on it. A test that can only ever say "expected 17, got 0" cannot be triaged, so it will keep being labelled a flake whether or not it is one.

Suggested fix

Add a minimal capturing ILogger to Darling.Tests and pass it at the EnsureRetentionPoliciesAsync call sites in this test, then include the captured warnings in the assertion message. Then the next occurrence names the actual Postgres error on the first failure instead of costing another round.

Worth doing before concluding anything about flakiness, and worth prioritising because this sits on a code path under active change (#2809 retention work, #2811/#2816 fetch-phase work).

Activity

  1. added a commit that references this issue on Sep 4, 2026
    891ae5b
  2. erikdarlingdata commented on Sep 4, 2026

    @erikdarlingdata
    OwnerAuthor

    Fixed by #2887, merged to dev in 891ae5b.

    What changed. Both live retention tests in Darling/Darling.Tests/TimescaleSupportTests.cs now pass a CapturingTestLogger instead of null to EnsureRetentionPoliciesAsync — and to the EnsureContinuousAggregatesAsync call ahead of it, since a swallowed CAGG failure is a plausible upstream cause of the same all-relations-throw symptom — and every count assertion interpolates that logger's captured text. Each pass gets its own logger (retentionLog, reapplyLog, createLog, convergeLog, settledLog), because CapturingTestLogger has no reset: sharing one would bury whichever pass actually failed under the earlier passes' benign "HELD PAUSED" warnings and success summaries.

    One correction to this issue's premise. It states there is no capturing ILogger anywhere in Darling.Tests. There has been one since #2128 — CapturingTestLogger, added for #1564's purge E2Es — so this was a matter of using it, not writing it.

    What this does NOT do. It does not fix whatever makes the retention pass intermittently return 0. That root cause is still unknown, and this change does not assert one. What it does is make the next occurrence triageable: instead of a bare expected 17, got 0, the assertion message now carries the per-relation PostgreSQL error text for each relation that threw — the evidence previously discarded by the logger?. null-conditional. So this is closed as the diagnosability defect it was filed as, not as a root-cause fix. If the count assertion fires again, the CI log should name the actual PostgreSQL error on the first failure, without needing a re-run.

    CI on the merged head. The Darling PostgreSQL tests job ran the full suite against a live TimescaleDB store: 6888 tests, 0 failed, 6 skipped. All six skips are the DARLING_TEST_SQL / DARLING_TEST_PGRUNTIME_OLD gated tests and neither retention test is among them, so both edited tests genuinely executed rather than skipping for want of a store. The intermittent failure did not reproduce in this run.

  3. added a commit that references this issue on Sep 10, 2026
    6ef3ba9
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions