Skip to content

PlanRegressionSql timeouts are silently swallowed — a cancelled query is indistinguishable from 'no plan regressions' #2826

Description

@erikdarlingdata

Summary

When PlanRegressionSql is cancelled by its (unchosen, inherited) 30 s Npgsql deadline — 325 times on 2026-09-03 alone, see #2827 — the failure is discarded with no log line, no retry, and no fact. "The plan-regression query timed out" and "this server has no plan regressions" produce byte-identical output.

Darling/PerformanceMonitor.Darling.Analysis/PgFactCollector.QueryPerf.cs:479:

catch (Exception ex) when (!AnalysisShutdown.IsExpectedAbandon(ex, context.CancellationToken))
{
    /* Table may not exist or have no data. An abandonment is NOT swallowed here (#2443). */
}

The body is empty. A PostgresException for 57014 query_canceled is neither "table may not exist" nor "no data", and nothing distinguishes it from the early return four lines above:

if (offenderCount == 0) return;

Both paths emit no PLAN_REGRESSION fact. So on the servers where this query reliably exceeds 30 s, plan-regression detection is silently disabled, and has been for as long as the query has been over the line.

This is the same family as #2801 (abandoned cycle recorded SUCCESS), #2796 (cancelled watermark read returned null, indistinguishable from a first run, and silently downgraded the collection window), #2816 (probe:0ms misattributed to other:), #2818 (all 17 relations threw but every message was discarded by a null logger), and #2820 (timed-out baseline cached a null for an hour). In every case the code was correct about the happy path and blind about its own failure.

It is a category, not an instance

grep -c "Table may not exist or have no data" across Darling/ returns 25. Fixing only the plan-regression site would be scenario-shaped — the same repair that had to be made twice in #2822's recovery path and twice again in the PgStatementText / #2344 enumerated-pin gap.

The fix is cheaper than previously assessed

#2795 records that logging here "needs an ILogger on PgFactCollector, which today takes only an NpgsqlDataSource — a ctor change across 7 call sites."

That is not required. Three siblings in the same project already take the logger as an optional parameter, so no call site changes at all:

  • PgBaselineProvider(NpgsqlDataSource postgres, ILogger? logger = null)
  • PgAnomalyDetector(NpgsqlDataSource postgres, PgBaselineProvider baselineProvider, ILogger? logger = null)
  • PgPlanFetcher(…, ILogger? logger = null)

PgFactCollector(NpgsqlDataSource postgres) can take the same optional parameter and log in the catch, matching the convention already established beside it.

Suggested shape

Distinguish the three outcomes rather than collapsing them:

  1. Expected abandon — already handled by the when filter, unchanged.
  2. Query cancelled / timed out (PostgresException 57014, NpgsqlException wrapping a timeout) — log at WARN naming the collector and the server, so it is greppable the way [PgBaselineProvider] Baseline query for io_latency did not finish within its command timeout is. That message is what made io_latency baseline times out: the robust scaffold's MAD tiers join on an OR'd non-equi predicate (23.7s -> 4.2s) #2820 findable at all.
  3. Anything else — log, rather than assuming "table may not exist".

Worth considering alongside: a fact or collection_log-visible marker so the gap is detectable from the store and not only from the app log, since the analysis pass leaves no collection_log row of its own.

A pin should assert over the family of catch sites by reflection rather than the one method touched — an enumerated pin is exactly how a broken sibling survived a green suite in #2344 and again in PgStatementText.

Activity

  1. added a commit that references this issue on Sep 3, 2026
    c25379a
  2. erikdarlingdata commented on Sep 3, 2026

    @erikdarlingdata
    OwnerAuthor

    Resolved by #2832, merged to dev. Closing manually: Fixes #N only fires on the dev->main release merge, so completed work otherwise sits open until release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions