Skip to content

Query Store backfill's slice halving oscillates instead of settling #4771

Closed

Description

@erikdarlingdata

When a Query Store backfill read times out (60 seconds), the slice halves. Any success clears the failure count for the server. The per-database ledger clears on the same success, and it never skips.

So take a database whose 1-hour slice times out while a 30-minute slice fits. It runs 60 minutes (fail), 30 minutes (success), 60 minutes (fail), and so on. A test pins that pattern (60, 30, 15, 15, 60, 30).

It is bounded, not endless: about one day of horizon at 5-minute ticks. That is roughly 48 to 192 wasted 60-second reads per database. Lite's copy of this code does the same.

Where (as of 30cbbcf)

  • The 60-second timeout: Darling/PerformanceMonitor.Darling.Service/DarlingCollectorRunner.cs:602.
  • The reset on success: Darling/PerformanceMonitor.Darling.Service/QueryStoreBackfill.cs:340 and :378-381, and PerformanceMonitor.Collectors/QueryStoreBackfillState.cs:107-117.
  • The per-database ledger: QueryStoreBackfill.cs:341. Its rationale: :316-323.
  • The horizon: QueryStoreBackfill.cs:124-127.
  • The pinned pattern: Darling/Darling.Tests/QueryStoreBackfillSkipFailingDatabaseTests.cs:85-87.
  • Lite: Lite/Services/RemoteCollectorService.QueryStoreBackfill.cs:198 and :368-370.

Fix

Keep the last good span for each server, and widen it by one halving step after a run of successes. Keeping it for each database instead goes against the rationale at :316-323. Change both apps.

Test that must fail first

After a 60-minute failure and a 30-minute success, the next slice stays at 30 minutes, then widens after the run of successes. The pinned pattern changes with it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions