Skip to content

Resolve gzip plan content in the analysis engine's two plan-XML reads - #3395

Merged
erikdarlingdata merged 1 commit into
devfrom
fix/plan-advisory-gz-reads
Sep 13, 2026
Merged

erikdarlingdata merged 1 commit into
devfrom
fix/plan-advisory-gz-reads

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

The analysis engine's two plan-XML reads never followed the #2069 / V54 gzip change, so on a store using the default plan_xml_compression = 'gzip' they return zero rows and analyze_server silently stops producing MISSING_INDEX and PLAN_WARNING facts and drill-downs.

The mechanism

PgFactCollector.PlanAdvisorySql and PgDrillDownCollector.PlanAdvisoryXmlSql selected query_plan_xml from v_query_stats and guarded on that one column. That view resolves the #1767 plan dimension as COALESCE(fact.query_plan_xml, dim.query_plan_xml) (PgSchemaGenerator.GenerateQueryStatsResolvingView), and since #2069 / V54 new dimension rows carry gzip bytes in query_plan_gz with the dimension's text column NULL. So the coalesced expression is NULL for every row the dimension holds, the guard discards all of them, planXmls.Count == 0, and both collectors take their early return.

No exception and no log line, so the outcome is indistinguishable from a server that has nothing to advise. A store set to plan_xml_compression = 'none' (#2171) still works, which is what kept this hidden.

Confirmed against a live production store, not just read off the code: query_plan_dim.query_plan_xml is NULL on all 5,705,706 rows there, with query_plan_gz carrying the content. Every dimension-resolved row on that store fails the old guard.

The fix

Both reads now project query_plan_xml, query_plan_gz, guard with (query_plan_xml IS NOT NULL OR query_plan_gz IS NOT NULL), and resolve text-else-gzip through PayloadDimensions.ResolveContent. That is the pattern DarlingStoredPlanReader.QueryStatsPlanXmlByHashSql and ViewerDataService.Plans.QueryStatsPlanXmlSql already use — copied rather than invented. Darling.Analysis already had the project reference to Darling.Storage, so nothing moved to reach PayloadDimensions.

The reference helper ReadPlanTextOrGzipAsync reads one row under LIMIT 1; these two reads take up to ten, so the resolution is inline in each loop rather than a shared single-row helper.

The pins, and which one is load-bearing

Four, in AnalysisPlanAdvisoryGzReadTests. Red-first verified by mutation, not asserted:

mutation result
fact collector's SQL guard reverted to the bare column its SQL pin fails, identity pin fails
drill-down's projection reverted to one column its SQL pin fails, identity pin fails
both read loops reverted to GetString(0), SQL left fixed only the source pin fails — all three SQL pins pass
restored 4/4 pass

That third row is why the source pin exists. A SQL-shape pin cannot see the read loop, and the half-fix is the likely failure mode: with the projection widened but the loop unchanged, every gzip row resolves to a NULL text column and is skipped exactly as before, while the SQL pins go green.

The identity pin exists for the same reason in the other direction. The two constants select the same top-ten-by-cost set on purpose — the fact collector counts what it finds, the drill-down renders the specifics of the same plans — so fixing one and not the other yields facts whose drill-down is empty, or a drill-down describing plans no fact counted. They live in different files and only one has to be found.

Run on macOS by compiling the real test file plus RepoFile.cs into a net10.0 console harness with an xunit shim, since Darling.Tests targets net10.0-windows. FleetIdentifierScrubTests run in the same harness against the real tracked tree: both pass. PerformanceMonitor.Darling.Service, which depends on this project, builds.

Scope

I swept every other query_plan_xml IS NOT NULL in the tree. One other read against this view turned up — the FinOps workload query in ViewerDataService.FinOps.Workload.cs — and it is already correct: it projects query_plan_gz in a parallel correlated subquery beside the text one. No change. PlanDimRecompression's text-only guards are correct by design, since that is the code migrating text into gzip.

Lite's bare query_plan_xml IS NOT NULL reads (Lite/Analysis/DrillDownCollector.Plans.cs, Lite/Analysis/DuckDbFactCollector.QueryPerf.cs, Lite/Services/LocalDataService.QueryStats.cs) are correct for DuckDB, which has no plan dimension and no gzip column. Untouched.

`PgFactCollector.PlanAdvisorySql` and `PgDrillDownCollector.PlanAdvisoryXmlSql` selected
`query_plan_xml` from `v_query_stats` and guarded on that column alone. That view resolves the #1767
plan dimension as `COALESCE(fact.query_plan_xml, dim.query_plan_xml)`, and since #2069 / V54 new
dimension rows carry gzip bytes in `query_plan_gz` with the dimension's text column NULL — so the
expression is NULL for every row the dimension holds and the guard matched none of them.

Both reads therefore returned zero rows on any store running the default
`plan_xml_compression = 'gzip'`, both collectors took their `planXmls.Count == 0` early return, and
`analyze_server` produced no MISSING_INDEX or PLAN_WARNING fact and no drill-down detail. No error,
no log line — indistinguishable from a server with nothing to advise. A store set to
`plan_xml_compression = 'none'` still worked, which is what kept it hidden.

Both now project and guard both forms and resolve text-else-gzip through
`PayloadDimensions.ResolveContent`, which is the pattern `DarlingStoredPlanReader` and
`ViewerDataService.Plans` already use. The `Darling.Analysis` project already referenced
`Darling.Storage`, so nothing moved to reach it.

Four pins. The two SQL-shape pins would pass on a half-fix — a widened projection with the loop
still taking `GetString(0)` leaves every gzip row resolving to a NULL text column and skipped — so
the source pin on the read loops is the load-bearing one, and the identity pin keeps the two
constants from drifting apart into facts whose drill-down is empty.

Swept every other `query_plan_xml IS NOT NULL` against this view: the FinOps workload read already
projects `query_plan_gz` in a parallel correlated subquery and needed no change. Lite's bare reads
are correct for DuckDB, which has no plan dimension and no gzip column, and are untouched.
@erikdarlingdata
erikdarlingdata merged commit 16a4e10 into dev Sep 13, 2026
8 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/plan-advisory-gz-reads branch September 13, 2026 04:03
@erikdarlingdata

Copy link
Copy Markdown
Owner Author

Reviewed and tested from the fleet-monitoring session (Erik's delegated review per the 02:39Z dispatch note).

Verified independently, not just read:

  • PayloadDimensions.ResolveContent(string?, byte[]?) exists (Storage:272) and the pattern claim is real — DarlingStoredPlanReader:180, ViewerDataService.Plans:171, DarlingWorker:6503 all already resolve text-else-gz this way. The Analysis→Storage project reference pre-existed (csproj line 39), so the new using adds no dependency edge.
  • Both call-site diffs are byte-identical in shape, matching the identity pin's contract.
  • Ran all four new tests for real at the PR head (net10.0 shim harness, Darling.Tests assembly name for parity): 4/4 pass.
  • Re-ran the load-bearing mutation myself: reverting the fact collector's read loop to GetString(0) with the SQL left fixed makes BothPlanAdvisoryReadLoops_ResolveTextElseGzip fail while both SQL pins stay green — the half-fix is caught exactly as the table claims. (My first mutation run also tripped the identity pin, which turned out to be my own editing tool silently converting the file's CRLF to LF — verbatim strings embed source line endings, so the two constants diverged at \r\n vs \n. My artifact, not yours — and a nice accidental proof the identity pin catches line-ending drift between the two constants too.)

CI green on all 8 checks, no bot findings on either review surface. Correct fix, correctly pinned — ready to merge on Erik's word.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant