Skip to content

Query Store: attribute runtime stats to their replica role; tighten the actual_state gate - #1546

Merged
erikdarlingdata merged 3 commits into
devfrom
feature/qs-replica-attribution
Jul 16, 2026
Merged

erikdarlingdata merged 3 commits into
devfrom
feature/qs-replica-attribution

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

What

Two Query Store collection changes.

1. Replica attribution (the feature). SQL Server 2022+ "Query Store for secondary replicas" gives an AG one shared Query Store that lives on the primary: secondary-replica workload is streamed to the primary and persisted in the primary's QS tables, distinguishable only by replica_group_id. We collect from primaries only (is_primary_replica = 1) — correct, and unchanged — but with no attribution the primary's "Top Queries by CPU" silently blends secondary workload into its own numbers, with no way for a reader to tell. (Microsoft's own Query Performance Insight has this exact bug.)

The shared QueryStoreCollector now LEFT JOINs sys.query_store_replicas on replica_group_id and selects replica_role = qsr.replica_name behind a productVersion >= 16 gate, mirroring the collector's existing version-gated column-fragment pattern. Surfaced as a narrow "Replica" column in Lite's and the Darling viewer's Query Store grids.

replica_role is a GROUP BY key, not a MAX(), in the top-queries readers — showing the role while still averaging across replicas would attach a label that lies about the numbers beside it. On a standalone/non-AG server every row shares one value, so the grouping is a no-op and the grids are unchanged.

2. actual_state gate (the bug fix). actual_state > 0 also admits 3 = ERROR, letting an errored Query Store pass the "is QS usable" gate. Tightened to actual_state IN (1, 2, 4) (READ_ONLY / READ_WRITE / READ_CAPTURE_SECONDARY) at both collector sites and in install/09_collect_query_store.sql.

Empirical findings — the docs are wrong; do not "correct" this back toward them

Verified live against SQL 2022 (16.0.4255.1) and SQL 2025 (17.0.4045.5):

  1. sys.query_store_replicas and runtime_stats.replica_group_id both exist from SQL 2022, not "2025+" as documented. Hence the gate is >= 16.

  2. replica_name IS populated on box SQL Server — observed Primary, Secondary, Geo Secondary, Geo HA Secondary. The docs claim NULL outside Azure Hyperscale named replicas. So it is read directly; the docs' sample additionally CASEs replica_group_id as though it were role_type — it is not, it is a replica set number that accumulates per role across failovers.

  3. The join MUST be LEFT. On a 2022 standalone the view has zero rows while real runtime-stats rows still carry replica_group_id = 1. Measured on SQL2022 StackOverflow2013:

    server LEFT JOIN INNER JOIN
    2022 standalone 31,027 rows 0 rows
    2025 standalone 35,573 rows 35,573 rows

    An INNER JOIN (or a WHERE replica_name = 'Primary' filter) silently deletes ALL Query Store collection on every 2022 standalone server — and is completely invisible if only tested on 2025. A test pins the LEFT JOIN.

replica_role is NULL on a 2022 standalone, 'Primary' on 2025, the real role on an AG. NULL honestly means "the server did not attribute this row" and is deliberately not coalesced, so blank is expected.

Storage

replica_role is appended last in PayloadColumns (Lite DuckDB v46→v47, Darling Postgres V27→V28). Both hosts' bulk writers are positional (DuckDB appender / Npgsql binary COPY) and an upgraded store gets the column from ALTER TABLE ADD COLUMN, which can only append — a mid-list position would put it mid-table on a fresh catalog-generated store and last on an upgraded one, and the COPY would then write shifted values on one of them. V28 also re-expands the pinned v_query_store_stats SELECT *. Same shape as deadlocks.database_name (#1535).

Scope

  • The deprecated Full Dashboard (install/*.sql) gets only the actual_state fix — no attribution, no schema column, no UI. Deliberate.
  • The Darling MCP get_query_store_top reader had the identical conflation and is fixed alongside the two grids (same logical surface, third host).
  • Not changed: the QS regression / comparison / trend / analysis-fact readers still aggregate across replicas. They group at a deliberately coarser grain (query_hash, or a time bucket); re-grouping them would redefine what a "regression" is and change alert semantics — a design decision, not a mechanical mirror. They are unaffected by this change.

Test plan

  • Builds green, 0 errors: Lite, Darling viewer, Darling service.
  • Lite.Tests pins green (33): payload 53→54 columns, the new >= 16 LEFT JOIN gate, the pre-2022 NULL placeholder, the DuckDB golden-schema equivalence pin, capability pins.
  • Darling.Tests pins green (63): PgMigrations-vs-StorageVersion pin (V28), QS SQL grouping, capability pins.
  • Rendered collector SQL executed live on SQL 2022 and SQL 2025 — valid, and returns the expected replica_role per the table above.
  • No DB-touching/integration tests run.

Fixed a latent test-fragility found en route: BuildPerItemQuery_PlanCapture_... asserted on the bare string CONVERT(nvarchar(1), NULL), which replica_role now legitimately shares; scoped to query_plan_text = ....

🤖 Generated with Claude Code

erikdarlingdata and others added 3 commits July 16, 2026 15:51
…he actual_state gate

SQL Server 2022+ "Query Store for secondary replicas" gives an AG ONE shared
Query Store living on the PRIMARY: secondary-replica workload is streamed to
the primary and persisted in the primary's QS tables, distinguishable only by
replica_group_id. We collect from primaries only (is_primary_replica = 1),
which is correct and unchanged -- but with no attribution the primary's Top
Queries silently BLENDS secondary workload into its own CPU/duration/reads.
(Microsoft's own Query Performance Insight has this exact bug.)

The shared QueryStoreCollector now LEFT JOINs sys.query_store_replicas on
replica_group_id and selects replica_role = qsr.replica_name behind a
productVersion >= 16 gate, mirroring the existing version-gated column-fragment
pattern. replica_role is a GROUP BY key (not a MAX()) in the top-queries
readers -- Lite, the Darling viewer, and the Darling MCP get_query_store_top --
because showing the role while still averaging across replicas would attach a
label that lies about the numbers next to it. On a standalone/non-AG server
every row shares one value, so the grouping is a no-op.

Empirical, verified live against SQL 2022 (16.0.4255.1) and 2025 (17.0.4045.5);
the docs contradict all three -- do not "correct" this back toward them:
  * sys.query_store_replicas and runtime_stats.replica_group_id BOTH exist from
    2022, not 2025+ as documented. Hence >= 16.
  * replica_name IS populated on box SQL Server (Primary, Secondary, Geo
    Secondary, Geo HA Secondary), so it is read directly. The docs' sample
    CASEs replica_group_id as though it were role_type; it is not -- it is a
    replica SET number accumulating per role across failovers.
  * The join MUST be LEFT. On a 2022 standalone the view has ZERO rows while
    real rows still carry replica_group_id = 1. Measured on SQL2022
    StackOverflow2013: LEFT JOIN 31027 rows vs INNER JOIN 0 -- an INNER JOIN
    silently deletes ALL Query Store collection on every 2022 standalone. The
    same counterfactual on 2025 is 35573 vs 35573, so this is invisible if only
    tested there. A test pins the LEFT JOIN.

NULL means "the server did not attribute this row" and is deliberately not
coalesced: NULL on a 2022 standalone, 'Primary' on 2025, the real role on an AG.

replica_role is APPENDED last in PayloadColumns (Lite DuckDB v47, Darling
Postgres V28): both hosts' bulk writers are positional (DuckDB appender /
Npgsql binary COPY) and an upgraded store gets the column via ALTER TABLE ADD
COLUMN, which can only append -- a mid-list position would desync a fresh
catalog-generated store from an upgraded one. V28 also re-expands the pinned
v_query_store_stats SELECT *. Same shape as deadlocks.database_name (#1535).

Also tightens the "is QS usable" gate from actual_state > 0 to
actual_state IN (1, 2, 4) (READ_ONLY/READ_WRITE/READ_CAPTURE_SECONDARY) at both
collector sites and in install/09_collect_query_store.sql: > 0 also admits
3 = ERROR. The deprecated Full Dashboard gets ONLY that fix -- no attribution,
no schema column, no UI -- deliberately.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CI caught two things the targeted test runs missed:

1) The V28 migration bumped StorageVersion 27->28 but never extended the viewer's
   schema PROBE. MapProbedSchemaVersion maps observed store sentinels to a
   version; with every sentinel present it still returned 27 while
   RequiredStoreSchemaVersion had moved to 28 -- so the viewer would have judged
   every fully-migrated v28 store "skewed" and refused to connect. Add the
   query_store_stats.replica_role sentinel to StoreSchemaProbeSql, a newest-first
   V28 arm to MapProbedSchemaVersion, the reader ordinal, and the Theory cases.
   The pinning test's own comment predicts exactly this failure mode.

2) ResolveConfigPath_PackagedLayout_FallsBackToParentDirectory fails on any
   machine with DARLING_CONFIG set: that variable is resolution step 2, ahead of
   both probed locations, so the parent-dir fallback under test never runs. It
   passes in CI (no such variable) but fails for any developer pointing a viewer
   at a remote store. Save/clear/restore it around the test; the sibling
   explicit-path test needs no guard (step 1 outranks the variable).

Full suites green: Darling 2085 passed / 0 failed, Lite 1301 passed / 0 failed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@erikdarlingdata
erikdarlingdata merged commit dfe29c9 into dev Jul 16, 2026
6 checks passed
@erikdarlingdata
erikdarlingdata deleted the feature/qs-replica-attribution branch July 16, 2026 21:49
pull Bot pushed a commit to ehtick/PerformanceMonitor that referenced this pull request Jul 22, 2026
Field evidence: 24 RDS read replicas all collecting query_store, each
showing actual_state=READ_ONLY, desired=READ_WRITE, readonly_reason=8
(read-only BECAUSE readable secondary). A secondary's persisted QS
tables are the PRIMARY's content via replication - collecting there is
duplicate, lagged primary data x24, and exactly the pathological
volume shape that fed erikdarlingdata#1557.

Why both guards missed: the enumeration's HADR join only catches
Always On AG secondaries (RDS replicas pass the "not in any AG" arm),
and the erikdarlingdata#1546 actual_state IN (1,2,4) gate deliberately admits
READ_ONLY without asking WHY.

Fix: readonly_reason & 8 = 0 (bitmask) at all three eligibility
sites - shared collector on-prem + Azure enumerations (Lite+Darling)
and the deprecated Dashboard's install/09 proc. The engine's own
mechanism-agnostic replica flag covers AG, RDS, and geo-secondaries
uniformly. Still collected: operator-set read-only QS on a primary
(reason<>8) and 2022+/2025 READ_CAPTURE_SECONDARY (state 4, real
local secondary workload).

Pins updated: both enumeration-text tests assert the new clause.
Lite.Tests 1369/0, Darling.Tests 2161/0 (Release, full).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant