Skip to content

Raise the viewer and mcp roles' 15 s statement_timeout to 60–90 s, and measure read durations for p9x #4442

Closed

Description

@erikdarlingdata

Claude posting for Erik Darling.

The viewer and mcp roles run with a 15-second statement_timeout. That ceiling now cuts off real reads, so we can't see how long they would actually take. This issue raises it to a normal 60–90 s and adds per-read durations we can take p50/p95/p99 from.

Where the 15 s comes from (dev b111f363c)

  • The server side:
    • ComposeSpec.StatementTimeout = "15s" (Darling/PerformanceMonitor.Darling.Service/Compose/ComposeSpec.cs:63);
    • DarlingConfig.ComposeStatementTimeoutSeconds seeds config.config_service.compose_statement_timeout_seconds (clamped 5–600 by StoreConfigProvider.ClampComposeStatementTimeoutSeconds);
    • startup provisioning writes it as ALTER ROLE viewer/mcp SET statement_timeout (DarlingManagedRoles), and a reload re-asserts it (The compose statement_timeout only reaches the live roles on a restart, unlike every other config_service knob #2918);
    • the column itself was added with DEFAULT 15 (PgMigrations, V78). After seeding, the store's value is authoritative, so changing the code default alone does not move an existing store.
    • Darling/tools/provision-roles.sql:122-123 hard-codes '15s' for bring-your-own stores.
  • The client-side deadlines that pair with it, which must stay ABOVE the server ceiling so the server's 57014 is the error a user sees:
    • McpCommandDeadlines.ReadSeconds = 20 and ComposedQueryFallbackSeconds = 15;
    • StorageCommandDeadlines.McpReadSeconds = 30;
    • ViewerCommandDeadlines.InteractiveReadSeconds = 15;
    • plus any web request timeout on the same paths.

What the field shows

  • One production store logged 50 statement-timeout cancellations from these roles in 7 days, rising (5, 2, 8, 4, 13, 15, 3 by day). Two other stores logged none.
  • Where the statement was captured, it's a composed-panel read over query_stats (a SUM(... ) FILTER (...) grouped by server/database/object or query hash).
  • The same store logged 39 slow statements (≥ 5 s) in 7 days. The roles already set log_min_duration_statement = 5000ms.
  • Attribution gap: the managed log_line_prefix is '%m [%p] ', with no %u or %a, so a timeout or slow line can't be tied to viewer, MCP or the service's own sessions.

Scope

  1. Raise the default ceiling to 60–90 s. The PR picks the value and says why.
    • It covers the role GUC AND every client-side deadline that pairs with it, keeping each client deadline above the server ceiling. The 5–600 clamp stays.
    • Existing stores: a migration rung moves a store still at the shipped 15 to the new default, and changes the column default. An operator-set value other than 15 is left alone.
    • Bring-your-own stores: provision-roles.sql gets the new value, and the release notes say to re-run it.
  2. Durations we can take p9x from. The design is proposed before any build:
    • (a) make the existing ≥ 5 s slow log attributable (%u, or %a with a set application_name, in the managed log_line_prefix), which gives the tail only;
    • (b) an app-side per-read duration histogram for web and MCP reads (p50/p95/p99), which gives the full distribution.
    • p9x needs (b); (a) helps attribute the outliers.
  3. A timeout stays a visible signal: a read that hits the ceiling says so to the user, and is counted.

Activity

  1. erikdarlingdata commented on Sep 26, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    Scope item 1 is merged in #4447 (e254521). The viewer and mcp roles' default statement_timeout is now 60 s. Store migration V147 moves stores still at the shipped 15 s to 60 s, and leaves an operator-set value alone. The web dashboard's and MCP server's composed reads get a 75 s client deadline, so the store's own timeout is the error those users see. The desktop viewer's (15 s) and the storage-layer readers' (30 s) client limits stay at their measured values for now, and will be revisited with the latency data from scope item 2. This issue stays open for items 2 and 3.

  2. erikdarlingdata commented on Sep 27, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    All three scope items are merged, so this issue is closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions