You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Retention-purge e2e can deadlock against the background jobs it just created (nightly flake; consider product-side 40P01 retry) #2143
Nightly flake: retention-purge e2e deadlocks against background jobs on the shared dev store
TimescaleSupportTests.EndToEnd_DetectConvertAndDropChunksPurge_AgainstDevPostgres failed in nightly run 31334656715:
drop_chunks on collection_log hit 40P01: deadlock detected (deadlock partner is almost certainly a TimescaleDB background job — the test converts hypertables and applies compression policies moments before purging, so a policy job waking up on the same chunk is a live race);
the DELETE fallback then ALSO deadlocked, so the 70-day row survived and the assertion failed.
Production behavior is already graceful — the purge logged the warning, counted 1 failed table, and moved on; a real store retries next cycle and self-heals. This is a TEST-hardness gap, not a product bug: the e2e asserts single-shot success against a store where its own just-created policy jobs can race it.
Fix options (pick one):
Retry the purge once on 40P01 inside the TEST (deadlocks are transient by definition; one retry makes the race window vanishingly small), or
have the test disable/pause the compression+retention jobs it created before running the purge assertion (deterministic, slightly more test code), or
teach the PRODUCT purge to retry once on 40P01 before falling back to DELETE — defensible on its own merits (a deadlocked drop_chunks wastes a whole cycle for a lock that clears in milliseconds), and the test inherits the fix.
Leaning 3 + 1: a single product-side retry is cheap and honest (log the retry), and the test keeps one retry as belt-and-suspenders.
Caught by the nightly on the #2141 merge commit; unrelated to that PR's content (nothing in it touches retention). The other failure in that run (Settings-box wiring) is fixed in #2142.
Fixed by #2145: drop_chunks retries exactly once on 40P01 (the partner's locks clear in milliseconds), a second deadlock gives up to the existing DELETE-fallback posture, non-deadlock failures stay single-shot, cancellation propagates. Four unit pins through the delegate seam; the purge e2e inherits the retry, closing the flake at its root. Robot review: no findings — it also confirmed each retry attempt opens a fresh connection, so no aborted-transaction reuse.
Activity
#2136: synthetic scale test — volume→duration→telemetry→alert, end to…
Changelog entry for the #2143 deadlock retry
Merge pull request #2145 from erikdarlingdata/retention-40p01-retry-2143
erikdarlingdata commentedon Aug 9, 2026
Fixed by #2145: drop_chunks retries exactly once on 40P01 (the partner's locks clear in milliseconds), a second deadlock gives up to the existing DELETE-fallback posture, non-deadlock failures stay single-shot, cancellation propagates. Four unit pins through the delegate seam; the purge e2e inherits the retry, closing the flake at its root. Robot review: no findings — it also confirmed each retry attempt opens a fresh connection, so no aborted-transaction reuse.
erikdarlingdata#2143: retry drop_chunks once on 40P01 before the DELE…