Skip to content

A bare unlogged catch defaults the compose statement_timeout onto the roles, desyncing it from #2931's client-side deadline #2952

Closed

Description

@erikdarlingdata

A bare unlogged catch writes the default compose timeout onto the roles, and can desync it from the client-side deadline that reads the same column

Found while working #2874 group C and verified independently here.

The shape

Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs:424, ReadComposeStatementTimeoutAsync:

:434    catch (Exception)
:435    {
:436        return 15;
:437    }

A bare, unlogged catch (Exception) returning the shipped default. Three consequences, in increasing order of importance:

  1. It swallows cancellation. An OperationCanceledException during shutdown is indistinguishable from a genuine read failure, and both silently produce 15.
  2. It logs nothing at all, so a store that is slow or briefly unreachable at start produces a wrong value with no evidence anywhere that a read failed.
  3. The value is then written onto the roles — its caller at :223 uses it to set the role-level statement_timeout. So the operator's configured horizon is replaced by the default on the server side, persistently, until the next successful provisioning pass.

The desync, which is the part that makes this worth fixing

#2874 group A's McpCommandDeadlines.ComposedQuerySeconds now reads the same config.config_service.compose_statement_timeout_seconds column — live per run, clamped through StoreConfigProvider.ClampComposeStatementTimeoutSeconds (landed in #2931).

So after a failed read here, the two halves of the same bound disagree: the role carries 15 because this catch defaulted it, while the client-side deadline carries whatever the operator actually configured. An operator who set 120 gets a server-side statement_timeout of 15 and a client deadline of 120 — the GUC fires first and every composed query dies at 15 seconds, with the configured value visible in the UI the whole time.

That is a genuinely confusing failure: the setting reads correct, the behaviour doesn't match it, and nothing logged anything.

Shape of a fix

  • Log the failure. Whatever else changes, a silent default here is indefensible; the read has a known column and a known fallback, so the log line can be specific.
  • Don't swallow cancellation. catch (Exception ex) when (ex is not OperationCanceledException) is the pattern already used in this file's neighbourhood (DarlingWorker.cs:1015 uses exactly that filter).
  • Decide whether defaulting is right at all. Provisioning a role with a value you failed to read is a choice; leaving the role's existing setting alone may be safer, since it at least preserves the last known-good horizon rather than overwriting it with a default.
  • Consider whether the two readers should share one accessor, so the role side and the client side cannot diverge by construction. That is the durable fix; the logging is the cheap one.

Note this is adjacent to #2936 (migrate failures are all terminal, no retry, no triage) — the same theme of a startup read whose failure mode was never chosen. They are separate code paths and separate fixes, but whoever takes one should read the other.

Not verified

Read from source. The desync is derived from both readers targeting the same column, not observed in a running system, and I have not measured how likely the read is to fail in practice. The catch's scope — exactly which exceptions can reach it — was not enumerated beyond the observation that it catches everything.

Activity

  1. erikdarlingdata commented on Sep 5, 2026

    @erikdarlingdata
    OwnerAuthor

    Closed by PR #2946. Verified on dev: zero bare catch (Exception) remain in DarlingManagedRoles.cs — all four are filtered now (:343, :402, :482 on ex is not OperationCanceledException, :427 on specific IO types).

    The fix went further than this issue asked, and the extra part is the valuable half. Filtering the catch would have stopped it swallowing cancellation, but it would still have defaulted a value it failed to read onto the roles. Instead the two outcomes are now separated: "the store has no opinion" (an unseeded row, or 42703/42P01) still defaults and logs at Debug, while "I could not hear the store" returns null, warns, and omits the two ALTER ROLE statements entirely so the roles keep their last known-good horizon. Provisioning reports a negative sentinel so #2918's gate re-asserts on a later pass.

    That closes the desync this issue was actually about: the role can no longer be overwritten with 15 while #2931's client-side ComposedQuerySeconds reads the operator's configured value from the same column.

    A second unfiltered catch was found and fixed — TryHardenRoleCredential — surfaced by the pin's own file-wide sweep rather than by this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions