Repository navigation
A bare unlogged catch defaults the compose statement_timeout onto the roles, desyncing it from #2931's client-side deadline #2952
Description
A bare unlogged catch writes the default compose timeout onto the roles, and can desync it from the client-side deadline that reads the same column
Found while working #2874 group C and verified independently here.
The shape
Darling/PerformanceMonitor.Darling.Service/DarlingManagedRoles.cs:424, ReadComposeStatementTimeoutAsync:
:434 catch (Exception)
:435 {
:436 return 15;
:437 }
A bare, unlogged catch (Exception) returning the shipped default. Three consequences, in increasing order of importance:
- It swallows cancellation. An
OperationCanceledExceptionduring shutdown is indistinguishable from a genuine read failure, and both silently produce15. - It logs nothing at all, so a store that is slow or briefly unreachable at start produces a wrong value with no evidence anywhere that a read failed.
- The value is then written onto the roles — its caller at
:223uses it to set the role-levelstatement_timeout. So the operator's configured horizon is replaced by the default on the server side, persistently, until the next successful provisioning pass.
The desync, which is the part that makes this worth fixing
#2874 group A's McpCommandDeadlines.ComposedQuerySeconds now reads the same config.config_service.compose_statement_timeout_seconds column — live per run, clamped through StoreConfigProvider.ClampComposeStatementTimeoutSeconds (landed in #2931).
So after a failed read here, the two halves of the same bound disagree: the role carries 15 because this catch defaulted it, while the client-side deadline carries whatever the operator actually configured. An operator who set 120 gets a server-side statement_timeout of 15 and a client deadline of 120 — the GUC fires first and every composed query dies at 15 seconds, with the configured value visible in the UI the whole time.
That is a genuinely confusing failure: the setting reads correct, the behaviour doesn't match it, and nothing logged anything.
Shape of a fix
- Log the failure. Whatever else changes, a silent default here is indefensible; the read has a known column and a known fallback, so the log line can be specific.
- Don't swallow cancellation.
catch (Exception ex) when (ex is not OperationCanceledException)is the pattern already used in this file's neighbourhood (DarlingWorker.cs:1015uses exactly that filter). - Decide whether defaulting is right at all. Provisioning a role with a value you failed to read is a choice; leaving the role's existing setting alone may be safer, since it at least preserves the last known-good horizon rather than overwriting it with a default.
- Consider whether the two readers should share one accessor, so the role side and the client side cannot diverge by construction. That is the durable fix; the logging is the cheap one.
Note this is adjacent to #2936 (migrate failures are all terminal, no retry, no triage) — the same theme of a startup read whose failure mode was never chosen. They are separate code paths and separate fixes, but whoever takes one should read the other.
Not verified
Read from source. The desync is derived from both readers targeting the same column, not observed in a running system, and I have not measured how likely the read is to fail in practice. The catch's scope — exactly which exceptions can reach it — was not enumerated beyond the observation that it catches everything.
Activity
Stop recording failed control-plane store reads as successful ones (#…
erikdarlingdata commentedon Sep 5, 2026
Closed by PR #2946. Verified on
dev: zero barecatch (Exception)remain inDarlingManagedRoles.cs— all four are filtered now (:343,:402,:482onex is not OperationCanceledException,:427on specific IO types).The fix went further than this issue asked, and the extra part is the valuable half. Filtering the catch would have stopped it swallowing cancellation, but it would still have defaulted a value it failed to read onto the roles. Instead the two outcomes are now separated: "the store has no opinion" (an unseeded row, or
42703/42P01) still defaults and logs at Debug, while "I could not hear the store" returns null, warns, and omits the twoALTER ROLEstatements entirely so the roles keep their last known-good horizon. Provisioning reports a negative sentinel so #2918's gate re-asserts on a later pass.That closes the desync this issue was actually about: the role can no longer be overwritten with 15 while #2931's client-side
ComposedQuerySecondsreads the operator's configured value from the same column.A second unfiltered catch was found and fixed —
TryHardenRoleCredential— surfaced by the pin's own file-wide sweep rather than by this issue.