Skip to content

Test flake: StoreToastAndCheckpointerLivePostgresTests reads 96 % live after deleting half the rows under full-suite load #3977

Closed

Description

@erikdarlingdata

Summary

StoreToastAndCheckpointerLivePostgresTests.WithTheExtensionInstalled_TheSweepFillsLiveBytesFromTheFreeSpaceMap_AndTheTupleShareProxyReadsTheLie_AgainstDevPostgres failed once in a full local Darling.Tests run:

after deleting half and vacuuming, the free-space map must read about half live (the rig read 48.4 %), not 96.2 % — 19712000 of 20480000 bytes
StoreToastAndCheckpointerTests.cs(1012)

It passed in the three earlier full runs of the #3896 lane, and when its class ran alone right after the failure. The branch (the #3896 cadence follow-up plus #3928) touches nothing in the store sweep or StoreSelfMetrics, so this is a load flake, not a product regression. Other lanes' suites were running on the same machine at the time, against their own rigs.

What the number says

96.2 % live after deleting half the rows means the VACUUM in the test left the free-space map almost untouched: it either could not remove the deleted tuples yet or skipped their pages. The fix #3846 made (pg_stat_force_next_flush() around the VACUUM) addresses the stats race on the tuple-share proxy, not this.

The class mints its own scratch database (#1776 own-store), so it doesn't serialize against the live-postgres collection and runs alongside the rest of the suite. That concurrency is the likely difference between the full run and the run alone. It wasn't narrowed further.

Possible directions

  • Retry the VACUUM (or use VACUUM (DISABLE_PAGE_SKIPPING)) until the free-space map moves, with a bounded number of attempts, and fail only if it never does.
  • Or put the class in a collection that doesn't run beside the other store-writing classes.

Activity

  1. added
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Sep 23, 2026
  2. added 2 commits that reference this issue on Sep 23, 2026
  3. erikdarlingdata commented on Sep 23, 2026

    @erikdarlingdata
    OwnerAuthor

    Closed by the watcher: delivered in PR #3994, merged to dev.

  4. removed
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Sep 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions