The plan-regression analysis read cannot complete on a large store, and its timeout is swallowed silently #2810

What

PgFactCollector.CollectPlanRegressionFactsAsync (PerformanceMonitor.Darling.Analysis/PgFactCollector.QueryPerf.cs:409) runs PlanRegressionSql with no CommandTimeout, so it inherits Npgsql's undocumented 30 s default. On the use1 monitoring host the query cannot finish in 300 s, so it is cancelled on every single analysis pass and has almost certainly never produced a fact on that store.

Running the shipped query string (extracted from the assembly source, not retyped), against the busiest server_id, with production-shaped parameters:

PREPARE pr(int, timestamp, timestamp) AS <PlanRegressionSql>;
EXPLAIN (ANALYZE, BUFFERS, TIMING, SUMMARY)
EXECUTE pr(1671144557, now() - interval '15 days', now() - interval '16 days');

SET statement_timeout = '300s';
Time: 300303.379 ms (05:00.303)     <- hit the ceiling, did not complete

This is the dominant remaining store cancellation

Grouping every canceling statement due to user request in the store's own PostgreSQL log since the 2026-09-02 22:54:56Z restart:

81 x  WITH deduped AS ( -- Collapse incremental re-collections of the same open runtime-stats interval ...
12 x  SELECT DISTINCT database_name FROM query_store_stats WHERE server_id = $1 AND collection_time > $2 ORDER BY database_name
---
93 total in ~1 hour

The first is PlanRegressionSql. The second is QueryStoreBackfill.cs:404.

Note for anyone re-running this analysis: an earlier pass reported ~1,673 of these as an unattributed "blank STATEMENT" bucket. That was a parser artifact, not a mystery — PostgreSQL writes STATEMENT: and then the SQL on following lines for a multi-line statement, so a parser reading only the remainder of the STATEMENT: line sees an empty string. There is no unattributed bucket.

The failure is invisible

catch (Exception ex) when (!AnalysisShutdown.IsExpectedAbandon(ex, context.CancellationToken))
{
    /* Table may not exist or have no data. An abandonment is NOT swallowed here (#2443). */
}

A timeout is neither "table missing" nor an abandonment, but it lands here and is discarded. Nothing reaches collection_log, nothing reaches the service log. The only trace anywhere is a cancellation line in the store's own PostgreSQL log — the same blind spot as #2795.

The consequence is a false negative in product output: "no PLAN_REGRESSION findings" and "the plan-regression read failed" are indistinguishable to anyone reading an analysis.

Why raising the timeout is NOT the fix

At >300 s against a 30 s default, no plausible CommandTimeout makes this succeed, and a larger one would only burn more store CPU before failing. The query is slow because query_store_stats holds 19 days / 65 GB at a 4-day policy — see the retention-paused issue filed alongside this. On a correctly retained store the query's own 15-day chunk-exclusion bound (#2387) would prune to ~5 chunks; here it prunes 3 of 19.

So the ordering is: fix retention first, then re-measure this query before deciding whether it needs any change at all.

What still wants doing regardless of retention

1. The silent swallow. A timed-out fact read should say so, naming the consequence. Doing it properly needs an ILogger on PgFactCollector, which today takes only an NpgsqlDataSource — a ctor change across 7 call sites, worth its own PR.
2. Explicit CommandTimeout. Not to stop the cancellations, but so the value is a deliberate, documented choice rather than an Npgsql default nobody wrote down. This is the Server-scoped watermark read is unbounded, times out, and silently degrades every collector to its fallback window #2795 lesson applied.

Scope of the missing-timeout pattern is larger than #2795 recorded

#2795's "Not covered here" cites three files (18/13/9 commands). Counted across all non-test, non-deprecated Darling production sources:

35 files, 160 `new NpgsqlCommand` sites with no CommandTimeout anywhere in the file

Including PgAnomalyDetector (12), PgFindingStore (8), all six PgFactCollector.* partials (31), DarlingSelfAlertEvaluator (6), PgMuteRuleStore (6), DarlingObservability (6).

For contrast, the paths that do set it: DarlingCollectorRunner 9/9, PgDrillDownCollector.Queries 8 commands / 9 timeouts (DrillDownCommandTimeoutSeconds = 30), TimescaleSupport 32/26. The convention exists; the analysis, alerting and config paths never adopted it.

Evidence gathered read-only (default_transaction_read_only = on) via psql on the box.