Skip to content

Darling collects each database's forced plans every 5 minutes (#5632) - #6069

Merged
erikdarlingdata merged 5 commits into
devfrom
feat/5632-forced-plan-state
Oct 18, 2026
Merged

erikdarlingdata merged 5 commits into
devfrom
feat/5632-forced-plan-state

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Oct 18, 2026 •

Copy link
Copy Markdown
Owner

Part of #5632.

Why

The Query Store collector reads only plans that ran since its last pass, so a forced plan the engine ignores, or that never runs, never reaches the store. Later steps of #5632 need the state of every force. This PR is step 1: the store gets the forced plans themselves. The state read (WORKING, NOT_EXERCISED, IGNORED, FAILING, REMOVED) is not built yet. See Deferred. The seat ships this collector half on its own, so that the V195 rung lands before the next rung (#5632 step 5) and the table gathers history before its read exists. A new lane builds steps 3 and 4.

What changes

  • New collector query_store_forced_plans, every 5 minutes, 30 days retention (QueryStoreForcedPlansCollector.cs, modelled on plan_correction). Per database it reads sys.query_store_plan where is_forced_plan = 1: plan_id, query_id, query_plan_hash, plan_forcing_type (gated to SQL Server 2017+ and Azure), force_failure_count, last_force_failure_reason (NULL when NONE), last_execution_time, last_compile_start_time (both UTC). It never reads plan XML. It is capped at 500 rows per database and sets CommandTimeout 120 s and a 120 s per-server wall-clock budget.
  • One all-NULL marker row per captured database per pass, always, then the forced plans. A later read can tell a force that was REMOVED (gone from the newest capture of a captured database) from a database that was not captured. A database whose Query Store is off, in error or a readable secondary returns no result set, so it writes no marker.
  • Same databases as query_store: the same vendor-name screen, AG-primary rule and scope filter. The Query Store state check runs inside each database's own statement. Azure SQL DB has the per-database branch.
  • Migration V195: collect.query_store_forced_plans plus index (server_id, collection_time), identical to the generator's output. SchemaVersion 195, viewer probe arm and RungRewindArtifacts entry added. No grant needed (the collect schema carries blanket SELECT). provision-roles.sql and DarlingManagedRoles.cs are untouched.
  • A plain table, not a hypertable (StaysPlainTable, like attention_events): a few rows per database per pass, so the grid stays at 72 hypertables and no worker-sizing numbers move. Retention is the existing time-sliced DELETE.
  • New ICollectorSchemaInfo.IsDarlingOnly (default false). Lite's StoredCollectors skips it, so Lite creates no table and lists no schedule row. The Lite twin is under Deferred.
  • Registration: catalog, schedule default, DarlingWorker dispatch, ServerHealthBands user-database enumerator list. No CollectorEngineCapability capture-path entry: the version gate is inside the collector's own SQL, and CollectorEngineCapabilityTests accepts only entries gated in the capability map. Counts and docs: PgSchemaGeneratorTests 73, llms.txt and Darling/README.md 44 SQL Server / 73 total, ConsumedTimestampFrameDisciplineTests (two UTC columns), the two empty-enumeration inventory tests, AttentionPlainTableTests opt-out census.
  • New pins in QueryStoreForcedPlansCollectorTests:
    • The SQL reads only is_forced_plan = 1, carries the marker row and sets a command timeout.
    • The version gate.
    • Registration as a plain Darling-only table at 5 min and 30 days.
    • V195 equals the generated schema.

Where the writer is: DarlingWorker dispatch query_store_forced_plans runs QueryStoreForcedPlansCollector through the normal runner, so the rows are written by the running service. The reader (get_forced_plans) is not written yet.

Test plan

  • Mac build, Darling.Tests and Lite.Tests: 0 warnings, 0 errors.
  • Seat, full Darling.Tests with the store (VM_PG=1) at 19f43c922: Darling.Tests Total: 34090, Errors: 0, Failed: 10, Skipped: 182. Full Lite.Tests at the same head: Lite.Tests Total: 10703, Errors: 0, Failed: 0, Skipped: 5.
  • Fix round (bed1a5ab3, comment below): 9 of the 10 were census pins the new collector joined, and one was a defect in the version-gate pin (its test context set no engine version). The database scope was already applied by BuildEnumerationQuery, and a pin now holds it. RollupFloorCacheLiveTests passed twice in the fix round and is not changed. Fix round VM run with VM_PG=1: Darling.Tests Total: 325, Errors: 0, Failed: 0, Skipped: 0.
  • origin/dev (da7cf8832) merged.
  • CI at bed1a5ab3 found one more: CollectionCaveatLedgerParityTests. The two SKUs' get_collection_health descriptions must be byte-identical, and the fix round had changed only Darling's fan-out sentence. Seat fix 5e835504e: both now say "Up to six fan out ... query_store_forced_plans in Darling only", because Lite does not run this collector. VM at 5e835504e: Darling.Tests Total: 42, Errors: 0, Failed: 0 (CollectionCaveatLedger*, CollectionLogFanoutRollupStoreTests, McpToolGuideTests, CollectionHealthPayloadBudgetLiveTests, with VM_PG=1) and Lite.Tests Total: 185, Errors: 0, Failed: 0.
  • Full suite after the fix round: CI on this PR is the gate. The seat's VM is shared by several lanes tonight.
  • Not run: the census classes for the read, because the read is not built (step 1b).

CHANGELOG entry

SECTION: Added
ENTRY: - **Darling collects each database's forced plans** ([#5632]) - A new collector reads the forced plans in `sys.query_store_plan` every 5 minutes into the new store table `collect.query_store_forced_plans` (store schema 195). It reads only rows with `is_forced_plan = 1`, at most 500 per database, plus one marker row per database. The read that reports whether each force works comes in a later change. Lite does not run this collector.
REF: [#5632]: https://github.com/erikdarlingdata/PerformanceMonitor/issues/5632

(Seat: the entry is rewritten to describe only the collector, which is what this PR ships.)

Deferred and follow-ups

Deferred (the brief's steps 3 and 4, not started because the investigation of the registration points used the lane's budget):

  1. State classification (WORKING, NOT_EXERCISED, IGNORED, FAILING, REMOVED, with the "first seen can be up to one collector pass early" caveat and a seam for the journal's force time).
  2. get_forced_plans MCP read: server_name, optional database_name, hours_back, as_of (DarlingReadWindow), registered like get_plan_force_precheck (An MCP read checks a Query Store plan before anyone forces it (#5632) #6065): tool class and WithGeminiCompatibleTools, DarlingWebEndpoints descriptor and dispatch (PDatabases/DatabaseNames or IDENTITY), WebReadReach, DarlingMcpToolClassification, plain-text.js MCP_TOOL_NAMES, database-filter-reads.js, CatalogCensus count (206 to 207), docs tool counts (docs/mcp-server.md, docs/editions.md), WebReadCancellationPinTests fake request, WebDatabaseFilterBehaviourTests scope count, McpToolGuideTests.
  3. get_query_store_query_history always lists forced plans, with state (DarlingMcpQueryStoreHistoryTools.cs about line 253, DarlingWebEndpoints.cs). HistorySql and HistoryTableSql untouched so far.
  4. Behaviour pins for each state and for REMOVED after a marker-only capture, each proved to fail with its rule removed.
  5. Lite twin of the collector and read (IsDarlingOnly currently hides it from Lite).
  6. A first complete VM run of the V195 rung, AgentLevel2StoreTests rewind and the other live classes.

Follow-ups proposed (not filed): none beyond the items above.

@erikdarlingdata

Copy link
Copy Markdown
Owner Author

Claude posting for Erik Darling.

Fix round on the 10 failures from the seat's full run at 19f43c922. Fix commit bed1a5ab3, with origin/dev (da7cf8832) merged first. No migration rung added, no packages.lock.json committed.

Result: after the fixes, 25 pattern groups (every class that names a file I changed, plus the migration, rung-rewind, schema generator, provision-roles ACL and alert-read-surface classes) ran in the VM with VM_PG=1: Darling.Tests Total: 325, Errors: 0, Failed: 0, Skipped: 0. Build 0 warnings, 0 errors.

# Failure Cause Fix
3 CollectorDatabaseScopeRungTests census Not a product gap. BuildEnumerationQuery already goes through DatabaseScopeFilter.BuildEnumerationPredicate, so the include and exclude scope is applied. The census list simply had five names. Census now lists QueryStoreForcedPlansCollector (six enumerators). Added a behaviour pin in QueryStoreForcedPlansCollectorTests: scope and exclusion arrive as @scope_db_* / @excl_db_* parameters, never as text, an unscoped context adds none, and Azure SQL DB does not enumerate. That pin passes on the old code too, because the code was already right. It guards the scope seam for this collector against a later edit.
7 TheToolDescription_NamesEveryCollectorThatFansOut The collector enumerates, so it fans out. The get_collection_health description said "Five fan out". Description now says "Six" and names query_store_forced_plans. The count pin moved from 5 to 6.
9 ThePlanForcingTypeColumn_IsAVersionGate_NullBelow2017 A defect in the pin, not the code. The body gates on the engine version from the enumeration probe (EnumerationProbeResult), the way QueryStoreCollector does. The test only set Target.SqlMajorVersion, so every context read as the 2016 default. The test context now sets EnumerationProbeResult. The pin fails on the old context (that is the failure above) and passes now.
8 EveryCapturePathEntry_... The collector runs on every SQL Server target and no read asks the capability question about it, so its CapturePathByCollector prose is unreachable. Removed the entry. Without one, NotCollectedMessage falls back to a correct generic message.
4, 5, 6 StatementColumnCensusTests query_store_forced_plans.query_plan_hash matches the payload pattern. Listed as Exempt: a hash, and the collector never reads plan XML or statement text. Same reason as the query_store and query_stats hash columns.
1, 2 ConsumedTimestampFrameDisciplineTests The two new UTC columns are classified from the collector's own AT TIME ZONE 'UTC' text, so the T-SQL-readable arm grew. TsqlUtcCount was still 15. TsqlUtcCount 15 to 17 with a comment. The frame is read from the query text, no declaration needed.
10 RollupFloorCacheLiveTests.ADatabaseDroppedAndRecreated... Did not fail for me. It passed in the first VM run on 19f43c922 (with VM_PG=1) and in the last run. I did not run it on origin/dev separately. Likely a load or timing flake in the seat's full run. None.

The first VM run on 19f43c922 showed 9 of the 10 failing (item 10 passed) and every cause above was readable from its messages.

Deferred and follow-ups

None.

@erikdarlingdata erikdarlingdata changed the title Darling reads each database's forced plans and reports whether each force works (#5632) Darling collects each database's forced plans every 5 minutes (#5632) Oct 18, 2026
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review October 18, 2026 19:34
… to six, the forced-plans collector in Darling only (#5632)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant