Skip to content

AggregateJobQuiesceLiveTests: the at-the-cap test fails when the refresh worker ends during the sweep #5603

Closed

Description

@erikdarlingdata

Claude posting for Erik Darling.

AggregateJobQuiesceLiveTests.AtTheCap_TheJobsAreScheduledAgain_TheAggregateStays_AndTheNextPassDropsIt_AgainstDevPostgres failed once in CI. It failed on PR #5599 at 70466ca, which does not touch this code. The run is 37856872462, job "Darling PG tests (1)". The test came in with #5552.

The failing check is at Darling/Darling.Tests/AggregateJobQuiesceTests.cs:445. After the sweep at the cap, it expects the refresh worker's backend to still be running. It found none (expected 1, actual 0). Every check before it passed.

The test uses a zero cap, so the sweep stops the aggregate's jobs and schedules them again within milliseconds. Its own comment says TimescaleDB's scheduler ends the worker of a stopped job within 3 to 4 seconds. A likely cause is that a slow runner left enough time for the scheduler to end the worker. The log cannot show which way it went.

Two ways to fix it. The test can accept that the scheduler can end the worker. Or the sweep can leave alone a job whose worker it does not end. A re-run of the failed job passed.

Activity

  1. added
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Oct 8, 2026
  2. erikdarlingdata commented on Oct 9, 2026

    @erikdarlingdata
    OwnerAuthor

    Fixed on dev by #5605. The live aggregate quiesce tests no longer need to see a TimescaleDB job worker alive inside a time window, so the at-the-cap test no longer races the scheduler. Claude posting for Erik Darling.

  3. removed
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions