Plan analysis: a deeply nested plan no longer crashes the process that parses it (#4512)

[erikdarlingdata]

Refs #4512: the depth limit, the size limit and the stack-overflow crash. The cancellation plumbing follows in a second PR, which closes the issue.
Part of #4511

Why

ShowPlanParser walks plan XML with recursive methods that had no depth limit. A plan nested deep enough overflows the stack, .NET cannot catch a StackOverflowException, and the process that parses the plan ends.

This isn't limited to crafted input. Measured on PerformanceMonitor's parser, each nested RelOp costs about 12.8 KB of stack. So a legitimately deep plan overflows at about 80 levels on a 1 MB thread (the viewer's UI thread) and about 120 levels on a 1.5 MB thread (the service's thread-pool and web-host threads).

Where plans are parsed, and on which thread:

Caller	File	Thread
Darling analysis pass (plan advisories)	PerformanceMonitor.PlanAnalysis/PlanAdvisoryAggregator.cs	service thread pool
Darling drill-down plans	Darling/PerformanceMonitor.Darling.Analysis/PgDrillDownCollector.Plans.cs	service thread pool
MCP plan tools (analyze_query_plan, analyze_procedure_plan, analyze_query_store_plan, analyze_plan_xml), Darling and Lite	PerformanceMonitor.PlanAnalysis/McpPlanAnalysisFormatter.cs	web host / thread pool
Lite drill-down	Lite/Analysis/DrillDownCollector.Plans.cs	Lite
Plan viewer	PerformanceMonitor.Ui/PlanViewerControl.xaml.cs	thread pool via Task.Run (parse and analyze); only Layout runs on the UI thread (1 MB)

For the service it's a crash loop, not a one-off: a deep stored plan that an analysis pass meets after every restart stops collection for every monitored server each time.

What changes

• Ported from PerformanceStudio (erikdarlingdata/PerformanceStudio@e8e5a21, erikdarlingdata/PerformanceStudio@f2153e2):
  • MaxParseDepth (1,000) through ParseStatementAndChildren and ParseRelOp, carried across procedure and UDF sub-plans (before, the count restarted at each sub-plan);
  • MaxParseCharacters (16 MB) on the synchronous Parse;
  • a catch around the whole tree walk that sets a new ParsedPlan.ParseError instead of throwing.
• PerformanceMonitor-specific: Parse runs its recursive walk on a dedicated thread with a fixed 32 MB stack (ParseThreadStackBytes) and joins it. Its public signature is unchanged. Every failure inside the walk is caught on that thread and returned as ParsedPlan.ParseError (nothing may run outside those catches, since an unhandled exception on a dedicated thread would end the process).
  • Why it's needed: without it, the 1,000-level guard is unreachable on a real caller: 1,000 levels need about 13 MB, and 32 MB gives 2× margin.
  • PerformanceStudio has no equivalent.
• The walks after parsing (PlanAnalyzer.Analyze, BenefitScorer.Score, PlanLayoutEngine.Layout) were measured on a 999-level tree. Each stays on the caller's thread with a wide margin, and each has a doc comment recording its measured floor:

Walk	Overflows below	Margin on a 1 MB / 1.5 MB caller
PlanAnalyzer.Analyze	~120 KB	~8× / ~13×
BenefitScorer.Score	~68 KB	~15× / ~23×
PlanLayoutEngine.Layout	~96 KB	~10× / ~16×

• ParseError is read everywhere now, so a refused or broken plan never fails silently:
  • the MCP plan tools report parse_error;
  • the plan-advisory aggregator (both apps' fact collectors) skips the plan;
  • PlanAnalysisPipeline.Run returns the plan untouched, so a partially parsed plan is neither analyzed nor scored;
  • both drill-down collectors stop right after parsing;
  • the plan viewer shows "This plan couldn't be parsed: " instead of an empty plan.
• Malformed XML (an unclosed tag, for example) sets a readable ParseError instead of quietly parsing to nothing.
• ScopedDescendants (the parser's walk over non-operator elements inside one operator, such as seek predicates) is iterative with an explicit stack. The recursive version could overflow on a deep run of such elements, which MaxParseDepth does not count, and it was also quadratic: 500,000 nested elements ran for over a minute at full CPU.
• Not in this PR: the async-path size limit and CancellationToken plumbing (PerformanceStudio@4ab01f1, @dc03dc1, @91f8617), which come in a follow-up.

Test plan

• PlanSync4512Tests, parsing from a thread created with a 1 MB stack:
  • 1,500 nested RelOps → ParseError ("depth limit"), with no crash;
  • a 1,500-level procedure chain → the same, which shows the depth carries across sub-plans;
  • 50 levels of legitimate procedure nesting still parse;
  • 16 MB + 1 characters → ParseError;
  • a normal plan parses unchanged.
• PlanSync4512PostParseTests: depth 999 and depth 150 each complete parse → analyze → score → layout on a real 1 MB thread and on a real 1.5 MB thread. 1,001 levels are refused by the parser's guard before any later walk sees the tree.
• PlanSync4512ParseErrorSurfacingTests: a refused plan reaches MCP with parse_error set and is skipped by the aggregator; a partially parsed plan (one good batch, one over the depth limit) is excluded by the aggregator and left untouched by the pipeline; malformed XML sets ParseError; both drill-down collectors keep their guard. Each has a mutation.
• A deep run of non-operator elements: 100,000 nested levels below one operator parse to completion on a real 1 MB thread.
• The REDs (crash-safe, in a child process, never the test host; local only, not in CI): with the dedicated thread reverted (the stack set to 1 MB), parsing 1,000 levels on a 1 MB caller crashes with a stack overflow through ParseRelOp. With the fix it returns ParseError.
• Mutation: MaxParseCharacters = int.MaxValue fails the size test.
• Run (after merging current dev): every PlanSync* class plus the plan-analysis classes: Total: 287, Failed: 0, Skipped: 2 (the live-Postgres plan-tool class runs in CI). Lite.Tests builds.

CHANGELOG

SECTION: Fixed
ENTRY: - A deeply nested execution plan no longer crashes the Darling service, the MCP plan tools or the plan viewer ([#4551]) - Plans nested deeper than about 80–120 operators overflowed the stack and ended the process that parsed them. Parsing now runs on its own thread with a large enough stack, stops cleanly at 1,000 levels, and rejects plan XML over 16 MB.
REF: [#4551]: #4551