Skip to content

plan_correction has the same bimodal-cost BODY_OVERRUN risk #2701 fixed for query_store, and was never included #2717

Closed

Description

@erikdarlingdata

Background

Erik flagged plan_correction's Collector Cost Regression alert on one fleet server (217,072 ms/day, 3.4x its 14-day baseline of 63,758 ms/day) as unacceptable and asked for further performance tuning beyond #2687 (materialize sys.dm_db_tuning_recommendations to #temp before joining Query Store catalog views) and #2688 (TRY_CONVERT→TRY_CAST).

Investigation

Confirmed via get_collector_cost: fleet-wide plan_correction total daily cost is trending DOWN since #2687/#2688 shipped (4.7M → 9.4M → 7.6M → 5.7M-partial-day ms across the ~40-server fleet) — those fixes are real and working.

The one affected server is the outlier. get_collection_health shows plan_correction there: avg_duration_ms 1057, p95 1554, max_duration_ms 21351, with fanout showing 97% of the worst cycle's cost concentrated in one database. That server's other databases are trivial by comparison.

Checked collect.query_store_plan_map for that database directly: 36,232 distinct plan_ids, 28,260 distinct digests — the identical application-class decimal-parameter-instability signature already root-caused for query_store on other servers earlier tonight. Several of the highest-volume plan_hash groups do collapse to a single digest (meaning many plan_ids there genuinely do share identical content, unlike the earlier-found servers' top offenders which showed zero collapsing), so this server's shape isn't byte-identical to the others, but the scale and mechanism are the same: an abnormally large, still-churning distinct-plan population feeding an Automatic Plan Correction recommendation set (200 live recommendations, all against the one large database, confirmed static in count across three checks tonight) that plan_correction must correlate every cycle.

plan_correction's own SQL has no further tuning headroom. It's already correctly seek-based post-#2687. The occasional 20+ second spike is proportional to real catalog size on an already-affected database — a data-volume problem downstream of the same app-side bug already being fixed on Erik's end, not a query-shape defect.

The actual actionable finding

plan_correction's cost shape — average ~1 second, occasional 20-230+ second spike depending on the affected server — is exactly the bimodal shape #2701 fixed for query_store (average 5-35s, occasional 100-230s spike on the same class of affected server), which caused query_store to block every other due collector sharing its per-server sequential body and risk BODY_OVERRUN. #2701's detach fix (RunDetachedQueryStoreAsync, IsQueryStoreCollector gate) applies only to query_store. plan_correction is still awaited inline in RunDueCollectorsAsync's sequential foreach, so a plan_correction spike on an affected server can still block that server's other due collectors for the duration, and — combined with query_store's own occasional spike on the SAME server — can still push a cycle toward BODY_OVERRUN.

plan_correction's own PerItemWallClockBudget of 120 seconds (set in #2673, "this collector cannot run minutes on a monitored server even if a future shape regresses") already anticipates runs heavy enough to risk overlapping its own next scheduled tick — the same self-overlap concern #2165's QueryStoreServerGate exists to prevent for query_store.

Proposed fix

Generalize #2701's detach pattern rather than special-casing a second collector: a small per-(server, collector) single-flight gate, structurally like QueryStoreServerGate but collector-agnostic, so plan_correction (and any future collector shown to have this same bimodal shape) can be detached from the sequential body and single-flighted against itself without inline-blocking every other due collector on an affected server. QueryStoreServerGate itself stays untouched (it has an orthogonal, second job: coordinating against the separate first-contact backfill loop, which a generic detach gate doesn't need to solve).

Activity

  1. erikdarlingdata commented on Sep 1, 2026

    @erikdarlingdata
    OwnerAuthor

    Fixed by #2720 (plan_correction detached from the sequential body, generalizing #2701). Merged to dev; ships in the next release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions