Skip to content

The Availability Group read finds each server's newest snapshot by aggregating every retained AG row: 2.8 s and 441 k buffers for 398 rows; a two-step read returns the same rows in ~3 ms #4228

Closed

Description

@erikdarlingdata

Problem

Both AG reads, replica states and database-replica states, find "the newest snapshot per server" the same way:

JOIN (SELECT server_id, MAX(collection_time) AS max_collection_time
      FROM ag_database_replica_states
      GROUP BY server_id) AS latest
  ON d.server_id = latest.server_id AND d.collection_time = latest.max_collection_time

The inner aggregate has no time bound, so it reads every row in retention for every server. The outer join then probes every chunk for the found instants. The cost tracks retained AG history × fleet size, while the answer is one snapshot per server.

The same statement text runs in both SKUs:

#4189 covers not calling the read for the nav probe and memoizing it. This issue is the read's own cost, which every remaining caller still pays.

Measured

Production SQL Server store A (43 servers; ag_database_replica_states is 478 MB across 31 daily chunks), 2026-09-25, EXPLAIN (ANALYZE, BUFFERS), read-only, one at a time:

form exec buffers rows
shipped AgDatabaseReplicaStatesSql 2,803 ms (+31 ms planning) 441 k (49.5 k read) 398
per-server LATERAL (… ORDER BY collection_time DESC LIMIT 1), then join back on the instant 2,090 ms 426 k 398
two-step: (1) per-server newest via LATERAL … LIMIT 1; (2) the shipped statement with collection_time >= <min of (1)> as a literal on both arms 2.1 ms + 1.0 ms 224 + 911 398

Where (origin/dev)

  • Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.AvailabilityGroups.cs: AgReplicaStatesSql ~:31, AgDatabaseReplicaStatesSql ~:64, GetAvailabilityGroupsAsync ~:101.
  • Darling/PerformanceMonitor.Darling.Viewer/MainWindow.xaml.cs:
    • the hidden-tab probe on every fleet tick ~:606;
    • the visible-tab refresh via LoadVisibleTabAsync ~:1170, into RefreshAvailabilityGroupsAsync ~:1102.
  • Darling/PerformanceMonitor.Darling.Service/Mcp/DarlingAgReader.cs: ReplicaStatesSql ~:76–108, DatabaseReplicaStatesSql ~:112–148 (the same shape, plus an optional $1 server filter). Web /api/ag: DarlingWebEndpoints.cs ~:283.

Fix shape

  • Two steps in each reader:

    1. SELECT s.server_id, l.collection_time FROM servers s CROSS JOIN LATERAL (SELECT collection_time FROM ag_database_replica_states WHERE server_id = s.server_id ORDER BY collection_time DESC LIMIT 1) l WHERE s.is_enabled. Ordered descent, 1 chunk per server.
    2. The existing statement with $floor bound on both the inner aggregate and the outer scan, where $floor = the minimum instant from step 1. The same for ag_replica_states.
  • Stale servers: a server whose last AG snapshot is days old would drag $floor back. Don't let it:

    • apply a staleness horizon: servers whose newest snapshot is older than it are read on their own, or listed as stale without a snapshot;
    • or run step 2 per server with that server's own instant as the bound.

    Either way, cost follows servers, not retention.

  • Share one implementation. The Viewer can't reference the Service assembly. Move the reader into PerformanceMonitor.Darling.Storage (like DarlingQueryStoreClutterReader) so the WPF tab, /api/ag and get_ag_health run the same two statements.

  • Pins:

    • result equality against today's statement on a seeded store with several days of AG snapshots, including a server whose AG rows stopped a day ago;
    • a live EXPLAIN assertion that step 2 touches ≤ 2 chunks when every server is current;
    • a source pin that no AG read uses MAX(collection_time) … GROUP BY server_id without a collection_time bound.

Activity

  1. added
    enhancementNew feature or request
    client-siteOwned by the client-site agents (other laptop). Local sessions never pick these up.
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Sep 25, 2026
  2. added 3 commits that reference this issue on Sep 25, 2026
  3. erikdarlingdata commented on Sep 25, 2026

    @erikdarlingdata
    OwnerAuthor

    Closed by the watcher: delivered in PR #4255, merged to dev.

  4. removed
    in-progressActively being worked by a local session or its agents (PR open or in flight)
    on Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    client-siteOwned by the client-site agents (other laptop). Local sessions never pick these up.enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions