Skip to content

Retention-purge e2e can deadlock against the background jobs it just created (nightly flake; consider product-side 40P01 retry) #2143

Closed

Description

@erikdarlingdata

Nightly flake: retention-purge e2e deadlocks against background jobs on the shared dev store

TimescaleSupportTests.EndToEnd_DetectConvertAndDropChunksPurge_AgainstDevPostgres failed in nightly run 31334656715:

  • drop_chunks on collection_log hit 40P01: deadlock detected (deadlock partner is almost certainly a TimescaleDB background job — the test converts hypertables and applies compression policies moments before purging, so a policy job waking up on the same chunk is a live race);
  • the DELETE fallback then ALSO deadlocked, so the 70-day row survived and the assertion failed.

Production behavior is already graceful — the purge logged the warning, counted 1 failed table, and moved on; a real store retries next cycle and self-heals. This is a TEST-hardness gap, not a product bug: the e2e asserts single-shot success against a store where its own just-created policy jobs can race it.

Fix options (pick one):

  1. Retry the purge once on 40P01 inside the TEST (deadlocks are transient by definition; one retry makes the race window vanishingly small), or
  2. have the test disable/pause the compression+retention jobs it created before running the purge assertion (deterministic, slightly more test code), or
  3. teach the PRODUCT purge to retry once on 40P01 before falling back to DELETE — defensible on its own merits (a deadlocked drop_chunks wastes a whole cycle for a lock that clears in milliseconds), and the test inherits the fix.

Leaning 3 + 1: a single product-side retry is cheap and honest (log the retry), and the test keeps one retry as belt-and-suspenders.

Caught by the nightly on the #2141 merge commit; unrelated to that PR's content (nothing in it touches retention). The other failure in that run (Settings-box wiring) is fixed in #2142.

Activity

  1. added 3 commits that reference this issue on Aug 9, 2026
  2. erikdarlingdata commented on Aug 9, 2026

    @erikdarlingdata
    OwnerAuthor

    Fixed by #2145: drop_chunks retries exactly once on 40P01 (the partner's locks clear in milliseconds), a second deadlock gives up to the existing DELETE-fallback posture, non-deadlock failures stay single-shot, cancellation propagates. Four unit pins through the delegate seam; the purge e2e inherits the retry, closing the flake at its root. Robot review: no findings — it also confirmed each retry attempt opens a fresh connection, so no aborted-transaction reuse.

  3. added a commit that references this issue on Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions