Repository navigation
Query Store: attribute runtime stats to their replica role; tighten the actual_state gate - #1546
Merged
Merged
Conversation
…he actual_state gate
SQL Server 2022+ "Query Store for secondary replicas" gives an AG ONE shared
Query Store living on the PRIMARY: secondary-replica workload is streamed to
the primary and persisted in the primary's QS tables, distinguishable only by
replica_group_id. We collect from primaries only (is_primary_replica = 1),
which is correct and unchanged -- but with no attribution the primary's Top
Queries silently BLENDS secondary workload into its own CPU/duration/reads.
(Microsoft's own Query Performance Insight has this exact bug.)
The shared QueryStoreCollector now LEFT JOINs sys.query_store_replicas on
replica_group_id and selects replica_role = qsr.replica_name behind a
productVersion >= 16 gate, mirroring the existing version-gated column-fragment
pattern. replica_role is a GROUP BY key (not a MAX()) in the top-queries
readers -- Lite, the Darling viewer, and the Darling MCP get_query_store_top --
because showing the role while still averaging across replicas would attach a
label that lies about the numbers next to it. On a standalone/non-AG server
every row shares one value, so the grouping is a no-op.
Empirical, verified live against SQL 2022 (16.0.4255.1) and 2025 (17.0.4045.5);
the docs contradict all three -- do not "correct" this back toward them:
* sys.query_store_replicas and runtime_stats.replica_group_id BOTH exist from
2022, not 2025+ as documented. Hence >= 16.
* replica_name IS populated on box SQL Server (Primary, Secondary, Geo
Secondary, Geo HA Secondary), so it is read directly. The docs' sample
CASEs replica_group_id as though it were role_type; it is not -- it is a
replica SET number accumulating per role across failovers.
* The join MUST be LEFT. On a 2022 standalone the view has ZERO rows while
real rows still carry replica_group_id = 1. Measured on SQL2022
StackOverflow2013: LEFT JOIN 31027 rows vs INNER JOIN 0 -- an INNER JOIN
silently deletes ALL Query Store collection on every 2022 standalone. The
same counterfactual on 2025 is 35573 vs 35573, so this is invisible if only
tested there. A test pins the LEFT JOIN.
NULL means "the server did not attribute this row" and is deliberately not
coalesced: NULL on a 2022 standalone, 'Primary' on 2025, the real role on an AG.
replica_role is APPENDED last in PayloadColumns (Lite DuckDB v47, Darling
Postgres V28): both hosts' bulk writers are positional (DuckDB appender /
Npgsql binary COPY) and an upgraded store gets the column via ALTER TABLE ADD
COLUMN, which can only append -- a mid-list position would desync a fresh
catalog-generated store from an upgraded one. V28 also re-expands the pinned
v_query_store_stats SELECT *. Same shape as deadlocks.database_name (#1535).
Also tightens the "is QS usable" gate from actual_state > 0 to
actual_state IN (1, 2, 4) (READ_ONLY/READ_WRITE/READ_CAPTURE_SECONDARY) at both
collector sites and in install/09_collect_query_store.sql: > 0 also admits
3 = ERROR. The deprecated Full Dashboard gets ONLY that fix -- no attribution,
no schema column, no UI -- deliberately.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ribution # Conflicts: # CHANGELOG.md
CI caught two things the targeted test runs missed: 1) The V28 migration bumped StorageVersion 27->28 but never extended the viewer's schema PROBE. MapProbedSchemaVersion maps observed store sentinels to a version; with every sentinel present it still returned 27 while RequiredStoreSchemaVersion had moved to 28 -- so the viewer would have judged every fully-migrated v28 store "skewed" and refused to connect. Add the query_store_stats.replica_role sentinel to StoreSchemaProbeSql, a newest-first V28 arm to MapProbedSchemaVersion, the reader ordinal, and the Theory cases. The pinning test's own comment predicts exactly this failure mode. 2) ResolveConfigPath_PackagedLayout_FallsBackToParentDirectory fails on any machine with DARLING_CONFIG set: that variable is resolution step 2, ahead of both probed locations, so the parent-dir fallback under test never runs. It passes in CI (no such variable) but fails for any developer pointing a viewer at a remote store. Save/clear/restore it around the test; the sibling explicit-path test needs no guard (step 1 outranks the variable). Full suites green: Darling 2085 passed / 0 failed, Lite 1301 passed / 0 failed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pull Bot
pushed a commit
to ehtick/PerformanceMonitor
that referenced
this pull request
Jul 22, 2026
Field evidence: 24 RDS read replicas all collecting query_store, each showing actual_state=READ_ONLY, desired=READ_WRITE, readonly_reason=8 (read-only BECAUSE readable secondary). A secondary's persisted QS tables are the PRIMARY's content via replication - collecting there is duplicate, lagged primary data x24, and exactly the pathological volume shape that fed erikdarlingdata#1557. Why both guards missed: the enumeration's HADR join only catches Always On AG secondaries (RDS replicas pass the "not in any AG" arm), and the erikdarlingdata#1546 actual_state IN (1,2,4) gate deliberately admits READ_ONLY without asking WHY. Fix: readonly_reason & 8 = 0 (bitmask) at all three eligibility sites - shared collector on-prem + Azure enumerations (Lite+Darling) and the deprecated Dashboard's install/09 proc. The engine's own mechanism-agnostic replica flag covers AG, RDS, and geo-secondaries uniformly. Still collected: operator-set read-only QS on a primary (reason<>8) and 2022+/2025 READ_CAPTURE_SECONDARY (state 4, real local secondary workload). Pins updated: both enumeration-text tests assert the new clause. Lite.Tests 1369/0, Darling.Tests 2161/0 (Release, full). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two Query Store collection changes.
1. Replica attribution (the feature). SQL Server 2022+ "Query Store for secondary replicas" gives an AG one shared Query Store that lives on the primary: secondary-replica workload is streamed to the primary and persisted in the primary's QS tables, distinguishable only by
replica_group_id. We collect from primaries only (is_primary_replica = 1) — correct, and unchanged — but with no attribution the primary's "Top Queries by CPU" silently blends secondary workload into its own numbers, with no way for a reader to tell. (Microsoft's own Query Performance Insight has this exact bug.)The shared
QueryStoreCollectornowLEFT JOINssys.query_store_replicasonreplica_group_idand selectsreplica_role = qsr.replica_namebehind aproductVersion >= 16gate, mirroring the collector's existing version-gated column-fragment pattern. Surfaced as a narrow "Replica" column in Lite's and the Darling viewer's Query Store grids.replica_roleis a GROUP BY key, not aMAX(), in the top-queries readers — showing the role while still averaging across replicas would attach a label that lies about the numbers beside it. On a standalone/non-AG server every row shares one value, so the grouping is a no-op and the grids are unchanged.2.
actual_stategate (the bug fix).actual_state > 0also admits3= ERROR, letting an errored Query Store pass the "is QS usable" gate. Tightened toactual_state IN (1, 2, 4)(READ_ONLY / READ_WRITE / READ_CAPTURE_SECONDARY) at both collector sites and ininstall/09_collect_query_store.sql.Empirical findings — the docs are wrong; do not "correct" this back toward them
Verified live against SQL 2022 (16.0.4255.1) and SQL 2025 (17.0.4045.5):
sys.query_store_replicasandruntime_stats.replica_group_idboth exist from SQL 2022, not "2025+" as documented. Hence the gate is>= 16.replica_nameIS populated on box SQL Server — observedPrimary,Secondary,Geo Secondary,Geo HA Secondary. The docs claim NULL outside Azure Hyperscale named replicas. So it is read directly; the docs' sample additionallyCASEsreplica_group_idas though it wererole_type— it is not, it is a replica set number that accumulates per role across failovers.The join MUST be
LEFT. On a 2022 standalone the view has zero rows while real runtime-stats rows still carryreplica_group_id = 1. Measured on SQL2022StackOverflow2013:An INNER JOIN (or a
WHERE replica_name = 'Primary'filter) silently deletes ALL Query Store collection on every 2022 standalone server — and is completely invisible if only tested on 2025. A test pins the LEFT JOIN.replica_roleis NULL on a 2022 standalone,'Primary'on 2025, the real role on an AG. NULL honestly means "the server did not attribute this row" and is deliberately not coalesced, so blank is expected.Storage
replica_roleis appended last inPayloadColumns(Lite DuckDB v46→v47, Darling Postgres V27→V28). Both hosts' bulk writers are positional (DuckDB appender / Npgsql binary COPY) and an upgraded store gets the column fromALTER TABLE ADD COLUMN, which can only append — a mid-list position would put it mid-table on a fresh catalog-generated store and last on an upgraded one, and the COPY would then write shifted values on one of them. V28 also re-expands the pinnedv_query_store_statsSELECT *. Same shape asdeadlocks.database_name(#1535).Scope
install/*.sql) gets only theactual_statefix — no attribution, no schema column, no UI. Deliberate.get_query_store_topreader had the identical conflation and is fixed alongside the two grids (same logical surface, third host).query_hash, or a time bucket); re-grouping them would redefine what a "regression" is and change alert semantics — a design decision, not a mechanical mirror. They are unaffected by this change.Test plan
Lite.Testspins green (33): payload 53→54 columns, the new>= 16LEFT JOIN gate, the pre-2022 NULL placeholder, the DuckDB golden-schema equivalence pin, capability pins.Darling.Testspins green (63): PgMigrations-vs-StorageVersionpin (V28), QS SQL grouping, capability pins.replica_roleper the table above.Fixed a latent test-fragility found en route:
BuildPerItemQuery_PlanCapture_...asserted on the bare stringCONVERT(nvarchar(1), NULL), whichreplica_rolenow legitimately shares; scoped toquery_plan_text = ....🤖 Generated with Claude Code