Query Store clutter had no surface: get_query_store_clutter composes per-database read cost, plan churn and the options row from rows already collected, one per-server QDS overhead block, replicas excluded by architecture (partial #3797)

[erikdarlingdata]

Partial #3797 — the reader and the MCP tool (arms (a), (b), (d), and (c) with capture mode null). The two viewer surfaces are the second lane the issue's own sequencing names. Does not close it.

The lie

Nothing in the product could say which databases were cluttering Query Store, why, or what it cost the servers hosting them — while every input for that answer was already in the store.

The case that motivated it (2026-09-20): on the largest production store, one database out of dozens carried 92–96% of the query_store collector's per-database read time; on the second store class the plan dimension was taking ~755k new distinct plans a day. Both were read off hand-run SQL against the store, because the four facts that compose the answer were each on their own surface and none of those surfaces put them beside each other: the V80 fan-out rollup on collection_log (fanout_item_count / slowest_item / slowest_item_ms, #2472) says which database the collector spends its time in; plan_id on the raw query_store_stats rows says how many plans a query carries and how fast new ones arrive; query_store_health says what the store is configured to keep; the QDS_* deltas in wait_stats and MEMORYCLERK_QUERYDISKSTORE in memory_clerks say what Query Store itself costs the instance. The maintainer's intent, verbatim spirit: not a new query — analyse what we've already collected. This is that.

The mechanism

DarlingQueryStoreClutterReader — four statements, one per arm, each with its own honest grain and its own retention, and one small clerk read.

• (a) read cost over v_collection_log for collector_name = 'query_store' runs with a fan-out: per (server, database) how often it was slowest_item (runs_slowest / runs_observed), the discrete median and p95 of slowest_item_ms when it was, the run's duration_ms median for scale, slowest_share_pct = the discrete median of slowest_item_ms / duration_ms — get_collection_health's fanout.dominance is slowest-vs-mean, not a concentration verdict — publish the slowest item's share of the pass directly #3502 verdict figure, the slowest item's share of the whole pass — and dominance_ratio: this database's median slowest cost over the POOLED median of every other database's slowest cost on the same server (an others CTE joining runs where a different database was slowest). Named and described as a different quantity from get_collection_health's dominance (slowest × items / run), so the two cannot be read as one. Runs with no fan-out (fanout_item_count IS NULL — a plain run, or an enumeration that yielded no items, which is every run on a replica) and zero-duration runs are not in the denominator.
• (b) plan churn over the raw query_store_stats table by plan_id, not query_plan_hash: plan_id is the identity MAX_PLANS_PER_QUERY caps, the key query_store_plan_map holds one row per (so distinct_plans is what the plan dimension grows by), and it is in idx_query_store_stats_server_db_query_plan_time, so the whole arm is an index-only walk of one server's rows where a hash-keyed count would fetch every heap row in the window; a shape hash also collapses parameter-sniffed recompiles and would under-count exactly this churn (PayloadDimensions' first reason). Per (server, database): distinct_queries, distinct_plans, plans_per_query_p95 / _max (discrete percentile over the per-query plan counts), plans_seen_once and never_seen_twice_fraction (plans under exactly one collection_time over all plans — meaningful because the collector re-fetches an OPEN interval every cycle while its last_execution_time advances, so a plan seen once ran inside one cycle's span and never again; null under two collections), plans_first_seen_in_window and new_plans_per_day (arrivals after the database's first collection in the window, over the days its collections span; the store keeps no first-seen stamp, so a plan idle before the window and run again inside it counts as new — the tool says so).
• (c) config — the newest query_store_health capture per database INSIDE the window (DISTINCT ON, not a MAX(capture_time) anchor: the newest capture is per database), with its own captured_at, every option the health row carries, readonly_reason decoded through the shared QueryStoreReadonlyReason, and query_capture_mode: null, capture_mode_known: false, capture_mode_note naming query_store_health does not collect QUERY_CAPTURE_MODE or WAIT_STATS_CAPTURE_MODE — the one option that names a plan-churn factory is the one the health row omits #3796's rung. Not wired to a column that does not exist yet; Lane 2 / the rung flips it.
• (d) the per-server block — every QDS_* wait type in the window from v_wait_stats, the COLLECTOR's stored deltas summed (no differencing here, so a restart cannot manufacture a negative), the rate formed only from rows with sample_interval_seconds > 0 (rated_wait_ms over measured_seconds, both sums over the same rows) and published as wait_ms_per_hour — a C# quotient over a measured span, never a cadence — with rows_unknowable (the (0, 0) restart rows) and rows_without_interval (pre-V127) counted beside it. excluded_wait_types NAMES the four QDS_* sleep waits IgnoredWaitDefaults drops at collection, derived from that list rather than restated, and the tool filters them again by name so a row that ever carried one lands in excluded_wait_types_present_in_store rather than in the proxy. memory_clerk is MEMORYCLERK_QUERYDISKSTORE latest and window max beside the newest capture of ANY clerk, because the collector stores the top 25 clerks over 1 MB and the clerk's absence is a rank, not a zero — clerk_in_latest_capture says which. The block ends with the window's baseline discontinuities through DarlingTrendReader.GetBaselineDiscontinuitiesAsync, the trend tools' own markers, reused rather than reinvented.

Every statement takes server_id = ANY($1::int[]), deliberately: the one-server read binds a one-element array and the fleet reference binds every enabled SQL Server target, so the reference is computed by the SAME arithmetic as the number it is a reference for. percentile_disc throughout — a discrete percentile is a value some run really had, and it side-steps PostgreSQL's missing round(double precision, integer) (DarlingOversizedPlanBacklogReader's reasoning). The replica census reads the bit off the health row and never off server_properties.ag_replica_role, which only knows Always On and reads Standalone on an RDS read replica.

QueryStoreClutter — the pure half, no store access, executed against planted rows in QueryStoreClutterTests: Compose (union of every arm's databases, ordered worst-first), Judge (the band and its reasons), the derivations (RunsSlowestPct, DominanceRatio, NewPlansPerDay, NeverSeenTwiceFraction, PctOfCap, WaitMsPerHour), DiscreteMedian (PostgreSQL's percentile_disc(0.5) exactly — the live test cross-checks the two on the same rows), IsReplicaServer, ComputeFleetMedian, Recommendations.

DarlingMcpQueryStoreClutterTools.get_query_store_clutter — server_name, hours_back (24, capped by the shared validator), limit (50), include_fleet_median (false), as_of. Rows are cut through McpHelpers.BoundPage with truncated / databases_returned / database_count. The window block carries the requested edges beside the raw tier's reach — window_truncated / effective_start / effective_hours_back off DarlingDataReader.GetQueryStoreWindowFloorAsync with get_query_store_top's ninety-minute slack — nested rather than flat because this payload ALSO carries a page cut and the two facts must not share a block (McpPayloadContractCensusTests holds them apart; the tool joins the window-floor rosters and its description ends with McpHelpers.WindowTruncatedDescription). next_tools per row are the reads that carry the detail, in the {tool, reason, suggested_params} shape the analysis findings use, only tools this server hosts. The miss ladder is get_query_store_top's: engine capability → the recorded Query Store state → the collector's own last outcome → the plain miss.

Why decomposed arms, not a score

The issue asked for rows "ranked by a composite the payload decomposes into its three arms with each arm's raw numbers". A composite that is a NUMBER — a weighted sum of a share, a percentile and a fraction — would have to invent the weights, and the first operator to ask "why is this database a 7.3" could not be answered from the row. So the composite is the BAND (HealthSeverity, the one spelling every band on this wire uses — #3703's canon, written through the enum's own ToString() so it cannot drift by hand) beside verdict_reasons, each a token naming the arm and the bar it crossed (read_cost_dominant, plan_churn_high, one_shot_plans, plans_per_query_at_cap, storage_near_cap, qs_read_only, qs_off, qs_read_only_replica, not_measured), with the bars published under thresholds so a reader can disagree with them on the evidence. Rows are ORDERED by band, then the read-cost share, then plans per query — the ranking the issue wanted without a number nobody can audit.

The bars are stated, not measured: production is uniform at 200 plans / 21 days / 8 GB, so configuration will not explain clutter differences today and churn will; the fleet had one database at 92–96% of the pass. read_cost_dominant needs the database to be the slowest on at least half the runs (one bad run is a run, not a database) AND a fan-out of at least two databases (one database has a 100% share by construction); then 50% of the pass is Warning and 80% Critical. plan_churn_high is plans_per_query_p95 at 4 / 10. one_shot_plans is half the plans seen once, over at least 20 plans. storage_near_cap is 90% of max_storage_size_mb. qs_read_only (not the replica bit) is Critical — the cap case IS clutter at its terminal state. Unknown means unmeasured and never escalates: a database with no runtime rows, no fan-out run naming it and no configuration (or Query Store OFF) says so rather than reading as Healthy — the "zero is a measurement" rule applied to a band.

Replicas. A database whose readonly_reason carries bit 8 — the engine saying "read-only BECAUSE this is a readable secondary", the mechanism-agnostic flag QueryStoreCollector's enumeration already gates on (readonly_reason & 8 = 0, #1558, which is why a replica has no query_store_stats rows and no fan-out runs in the first place) — reads excluded: true, excluded_reason: qs_read_only_replica, verdict Unknown with that one reason and no other: its read cost and churn are its primary's, its configuration is replicated, and judging any of them would file a defect against the wrong server (the topology ruling, 2026-09-20). server_is_replica says when every recording database on a server is one, the qs_overhead block is still reported (waits and clerks are real on a replica), and the fleet reference excludes such servers. Tested as a bit, not an equality: 65544 (cap + replica) is still a replica.

What it does NOT do

• No viewer. Web and WPF are the second lane; this payload is the shape they render. No server-tabs.js panel, no Viewer tab.
• Capture mode is null with capture_mode_known: false on every config row and at the top level. query_store_health does not collect QUERY_CAPTURE_MODE until query_store_health does not collect QUERY_CAPTURE_MODE or WAIT_STATS_CAPTURE_MODE — the one option that names a plan-churn factory is the one the health row omits #3796's rung lands (Lanes 1 and 2 this wave); the churn recommendations say the mode is uncollected and name both readings (ALL beside churn → AUTO; AUTO beside churn → the workload) rather than guess.
• The fleet reference is opt-in. include_fleet_median defaults false because it walks query_store_stats and wait_stats fleet-wide over the window — on the largest store that is the read deadline's whole budget (20 s client-side, 15 s statement_timeout on the managed store's mcp role), and a default call that times out is a tool nobody uses. With it on, fleet_median is the discrete median of slowest_share_pct and plans_per_query_p95 over every DATABASE on every enabled SQL Server target that is not a replica (only servers that fanned out over ≥ 2 databases enter the share population), and of the QDS rate over those SERVERS (each server's sum of its included per-type rates — the per-type rates add, the per-type measured seconds do not), each with the population it was drawn from and replica_servers_excluded.
• It does not touch QueryStoreHealthCollector, the query_store_health columns, get_query_store_health, PgMigrations, CHANGELOG.md, or any viewer file. No table, no rung.
• Two spellings deliberately follow the house vocabulary rather than the brief's list. slowest_share_pct keeps get_collection_health's meaning (the slowest item's share of the pass) and the run-frequency figure is runs_slowest_pct; dominance_ratio is the cross-database ratio the brief defined and is described as NOT get_collection_health's dominance.

Lite

Verified against the brief's hypothesis, which was wrong in the direction that matters: Lite HAS the collection_log fan-out columns (Lite/Database/Schema.cs:60-62, DuckDbInitializer.cs:1471-1475, written by RemoteCollectorService.cs:854), and its query_store_stats / query_store_health / wait_stats / memory_clerks tables are generated from the same shared collector definitions. It does not have query_store_plan_map / query_plan_dim — and this reader does not read them either, because plan_id on the fact rows is all the churn arm needs. So a Lite twin is a DuckDB port of the four statements (percentile_disc → quantile_disc) over the same pure judgment: a "not ported yet", not a SKU boundary, and the CrossAppMcpToolInventoryPinTests ratchet entry says so in those words. Out of scope here by the issue's own sequencing (reader + MCP first) and the brief's Darling-only deliverable.

Registration

The seven sites a new Darling tool lands on: DarlingMcpHostService (WithGeminiCompatibleTools<DarlingMcpQueryStoreClutterTools>()), DarlingWebEndpoints catalog descriptor + dispatch (/api/read/get_query_store_clutter with server, hours, limit, include_fleet_median, as_of), the DarlingMcpInstructions census sentence (157 tools / 70 unique) and Tool Reference row, the root README.md (both Darling counts), llms.txt (88-157), Darling/README.md (a Query Store clutter bullet under the core reads), and the Lite inventory ratchet. McpPayloadContractCensusTests' window-floor rosters (WindowFloorBlocks, WindowFloorTools, SecondBoundCutKeys[window_truncated], CutNoteKeys[truncation_note]) gain the file — roster additions, no rule change.

Tests

Executed on this Mac through the built Darling.Tests.dll (-c Release -p:EnableWindowsTargeting=true, WindowsDesktop framework entry stripped from the runtimeconfig, xunit v3 in-process):

• QueryStoreClutterTests — 49 facts: every judgment boundary on both sides by the thresholds' NAMES (the replica bit beside a would-be-Critical share and churn → Unknown with one reason; 65544 is a replica, 65536 is not; nothing measured → Unknown not_measured; OFF → Unknown qs_off; measured and quiet → Healthy with no reasons; read cost at 90 / 79.9 / 50 / 49.9 % and under the runs gate and on a one-database fan-out; churn at 10 / 9 / 4 / 3; one-shots over 40 vs 19 plans and under one collection; the cap with and without a config row; storage at 90.0 vs 89.99 %; the worst arm wins and every arm keeps its reason), the derivations, DiscreteMedian on six populations, the excluded set equals the shared ignore list's four QDS_* entries, IsReplicaServer, Compose ordering, ComputeFleetMedian with a replica server and a one-database fan-out excluded, the recommendations' knobs, the tool surface and parameters, the description's claims, every statement's text, and the six registration sites.
• QueryStoreClutterLivePostgresTests — against timescale/timescaledb:2.28.1-pg18 on a ScratchPostgres database (#1776 own-store), through the REAL tool: three databases with alpha slowest_item on 8 of 10 runs at 9000 of 10000 ms (asserted: runs_observed 10, runs_slowest 8, runs_slowest_pct 80, slowest_item_ms_p50/p95 9000, run_duration_ms_p50 10000, slowest_share_pct 90, others_slowest_item_ms_p50 3000 — percentile_disc over beta's 4000 and gamma's 3000 — dominance_ratio 3.0, verdict Critical read_cost_dominant; beta's dominance_ratio 0.44 against the others' 9000); a run with no fan-out, a zero-duration run and another collector's run that must not count; plan churn with alpha at one plan per query (plans_per_query_p95 1, never_seen_twice_fraction 0.0, new_plans_per_day 0.0) and beta's one query carrying six (distinct_plans 9, plans_per_query_p95 6, plans_seen_once 3, never_seen_twice_fraction 0.333, plans_first_seen_in_window 3, new_plans_per_day 36.0 over the two hours its collections span, verdict Warning plan_churn_high); gamma measured and Healthy with no reasons; db_off Unknown qs_off; the NEWEST of two health captures (4096 MB, not 2048) with its stamp; the window floor (window_truncated true, effective_hours_back 6.0 against a 24-hour request); QDS_LOADDB from three rated rows, one (0, 0) row and one pre-V127 row (wait_ms_total 1900, measured_seconds 180, wait_ms_per_hour 36000, rows_unknowable 1, rows_without_interval 1), QDS_BLOCKING_TASK 7200/h, totals 2020 ms and 43200/h, the planted QDS_ASYNC_QUEUE in excluded_wait_types_present_in_store and not in the proxy, a PAGEIOLATCH_SH row ignored; the clerk at 640.25 MB in the newest of two captures; the fleet reference with 3 enabled targets, the reason-8 replica server excluded (shares 90/40/30 → 40; plans 1/6 → 1; one server's QDS rate 43200); the page cut at limit 2 (worst two, truncated true, database_count 4, fleet_median null with the opt-in note); the replica server (server_is_replica true, its one row excluded with the reason and the decoded bit, next_tools stopping at the composition sources, the overhead block real); an empty server through the miss ladder to unavailable; a bad limit and a bad hours_back refused through the shared shapes. Positive control: with the others CTE's <> flipped to =, the first live test fails at others_slowest_item_ms_p50 (expected 3000, actual 9000). A second live fact runs the reader alone and asserts QueryStoreClutter.DiscreteMedian over its three shares is the 40 the fleet block published.
• Census and pin families, all green after the rebase onto 0bcbd50b3: RepoFileAdoptionTests, StragglerCommandTimeoutTests, LiveCleanupConversionRatchetTests, ViewerCommandTimeoutTests, MeasurementContractCensusTests, McpPageContractTests, McpPayloadContractCensusTests, FactSourceRegistryTests, DocCommentHygieneTests, TsqlConventionGuardTests, McpReadCommandTimeoutTests, McpLatestSnapshotStampTests, AsOfWindowAnchorTests, DarlingWebEndpointsTests, DarlingCustomViewsTests, ServerPageTabsTests, BaselineDiscontinuityRenderTests, McpToolTypeRegistrationTests, ReadmeDerivedCountPinTests, McpZeroIsAMeasurementTests, CommentFilterAdoptionTests, CollectorStateContractTests, DarlingMcpConfigHistoryToolsTests, DarlingMcpDataToolsTests — 687 facts. LivePostgresCollectionHygieneTests reds on this Mac on clean dev too (PresentationFramework load during assembly-wide reflection — the WPF harness limit), so it is CI's; this PR's live class carries the #1776 own-store marker and mints its own database.
• Lite: CrossAppMcpToolInventoryPinTests 6/6 through a throwaway net10.0 harness compiling that class and ParitySource (Lite.Tests.dll itself hangs at discovery on the Mac).
• Two expression-bodied members were spelled with HasValue / is int instead of is { } so TsqlConventionGuardTests' member walk reads them whole, rather than growing KnownTruncatedRanges.

PerformanceMonitor.Darling.Service, Darling.Tests and Lite.Tests build -c Release -p:EnableWindowsTargeting=true with 0 warnings.

Which component(s) does this affect?

• Lite
• Darling (one new MCP tool, its reader and judgment; the host, web catalog, instructions and READMEs)
• Lite Tests (the inventory ratchet entry)
• Darling Tests
• SQL collection scripts
• Documentation (Darling/README.md, root README.md, llms.txt — tool counts and one bullet)
• Full Dashboard (deprecated)
• CLI Installer (deprecated)

Checklist

• I have read the contributing guide
• My code builds with zero warnings (-c Release -p:EnableWindowsTargeting=true, every touched project)
• I have tested my changes (the pure judgment and every pin in-process on this Mac; the live class against a PG18 / TimescaleDB 2.28.1 scratch store with a positive control; CI's Darling PostgreSQL tests job is its second execution)
• I have not introduced any hardcoded credentials or server names

Partial #3797

[erikdarlingdata]

Head rebuilt as 4f8499a (13 files, +2567/−9) on dev f2e1816. The first head (276923a) was cut with reset --soft origin/dev after another seat's fetch had moved origin/dev past my rebase base, so it carried reverse-diffs of #3810 (34 files) — CI was green on it because the tree was internally consistent, but merging it would have reverted #3810. Do not merge 276923a; the current head is the 13-file change and nothing else. Dev's #3813 V137 row in Darling/README.md is preserved.