Skip to content

Darling test flake: viewer-role grant test fails with 'tuple concurrently updated' #5560

Closed

Description

@erikdarlingdata

A Darling live PostgreSQL test failed in CI with XX000: tuple concurrently updated. The test, QueryStatsHourLedgerTests.TheViewerRolesProvisionedGrants_ReadBothTables_AndWriteNeither, does not touch the code the PR changed.

Seen once, on 2026-10-08, in a pull request's Darling PG shard 1 (1 of 4336 failed). It has not been seen on dev yet.

The test creates a login role and runs the viewer-role statements on it (QueryStatsHourLedgerTests.cs:390-394). PostgreSQL raises this error when two sessions change the same catalog row at once, for example two GRANTs on one role, or role membership. Roles belong to the whole cluster, not one database. So another test class that runs at the same time on the same cluster can touch the same row.

To fix: find which statement raised it and which other test can run it at the same time. Then make the two not overlap (a shared collection, or a role name of each test's own). Keep every existing check. If the product's own provisioning can do the same thing when two services start together on one cluster, say so in the issue. Show the failure on purpose first, then show the fix holds.

Claude posting for Erik Darling.

Activity

  1. added
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Oct 8, 2026
  2. erikdarlingdata commented on Oct 8, 2026

    @erikdarlingdata
    OwnerAuthor

    Fixed by #5566, merged into dev. The tests that replay the viewer's grants now aim at their own scratch database. At service start, role provisioning, the reload's statement_timeout step and the password-rules step take a bounded lock. When the lock is not free in time, they run anyway, and a write that collides with another session is run again up to twice. It ships in 3.10. Claude posting for Erik Darling.

  3. removed
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions