Repository navigation
Raise the viewer and mcp roles' 15 s statement_timeout to 60–90 s, and measure read durations for p9x #4442
Copy link
Copy link
Closed
Description
Claude posting for Erik Darling.
The viewer and mcp roles run with a 15-second statement_timeout. That ceiling now cuts off real reads, so we can't see how long they would actually take. This issue raises it to a normal 60–90 s and adds per-read durations we can take p50/p95/p99 from.
Where the 15 s comes from (dev b111f363c)
- The server side:
ComposeSpec.StatementTimeout = "15s"(Darling/PerformanceMonitor.Darling.Service/Compose/ComposeSpec.cs:63);DarlingConfig.ComposeStatementTimeoutSecondsseedsconfig.config_service.compose_statement_timeout_seconds(clamped 5–600 byStoreConfigProvider.ClampComposeStatementTimeoutSeconds);- startup provisioning writes it as
ALTER ROLE viewer/mcp SET statement_timeout(DarlingManagedRoles), and a reload re-asserts it (The compose statement_timeout only reaches the live roles on a restart, unlike every other config_service knob #2918); - the column itself was added with
DEFAULT 15(PgMigrations, V78). After seeding, the store's value is authoritative, so changing the code default alone does not move an existing store. Darling/tools/provision-roles.sql:122-123hard-codes'15s'for bring-your-own stores.
- The client-side deadlines that pair with it, which must stay ABOVE the server ceiling so the server's
57014is the error a user sees:McpCommandDeadlines.ReadSeconds = 20andComposedQueryFallbackSeconds = 15;StorageCommandDeadlines.McpReadSeconds = 30;ViewerCommandDeadlines.InteractiveReadSeconds = 15;- plus any web request timeout on the same paths.
What the field shows
- One production store logged 50 statement-timeout cancellations from these roles in 7 days, rising (5, 2, 8, 4, 13, 15, 3 by day). Two other stores logged none.
- Where the statement was captured, it's a composed-panel read over
query_stats(aSUM(... ) FILTER (...)grouped by server/database/object or query hash). - The same store logged 39 slow statements (≥ 5 s) in 7 days. The roles already set
log_min_duration_statement = 5000ms. - Attribution gap: the managed
log_line_prefixis'%m [%p] ', with no%uor%a, so a timeout or slow line can't be tied to viewer, MCP or the service's own sessions.
Scope
- Raise the default ceiling to 60–90 s. The PR picks the value and says why.
- It covers the role GUC AND every client-side deadline that pairs with it, keeping each client deadline above the server ceiling. The 5–600 clamp stays.
- Existing stores: a migration rung moves a store still at the shipped 15 to the new default, and changes the column default. An operator-set value other than 15 is left alone.
- Bring-your-own stores:
provision-roles.sqlgets the new value, and the release notes say to re-run it.
- Durations we can take p9x from. The design is proposed before any build:
- (a) make the existing ≥ 5 s slow log attributable (
%u, or%awith a setapplication_name, in the managedlog_line_prefix), which gives the tail only; - (b) an app-side per-read duration histogram for web and MCP reads (p50/p95/p99), which gives the full distribution.
- p9x needs (b); (a) helps attribute the outliers.
- (a) make the existing ≥ 5 s slow log attributable (
- A timeout stays a visible signal: a read that hits the ceiling says so to the user, and is counted.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Activity
Add migration rung V147 and bring existing statement_timeout pins for…
Fix remaining stale-value pins and add the derived paired-deadline pin (
Pin V147's statement_timeout migration and the roles' re-assertion (#…
Compare the roles' statement_timeout by value, not its display unit (#…
Raise the viewer and mcp roles' default statement_timeout from 15 s t…
erikdarlingdata commentedon Sep 26, 2026
Claude posting for Erik Darling.
Scope item 1 is merged in #4447 (e254521). The viewer and mcp roles' default statement_timeout is now 60 s. Store migration V147 moves stores still at the shipped 15 s to 60 s, and leaves an operator-set value alone. The web dashboard's and MCP server's composed reads get a 75 s client deadline, so the store's own timeout is the error those users see. The desktop viewer's (15 s) and the storage-layer readers' (30 s) client limits stay at their measured values for now, and will be revisited with the latency data from scope item 2. This issue stays open for items 2 and 3.
Fix the p99 fixture assertion and add the product-path read-latency p…
Add store rung V148 (collect.read_latency) and the hourly flush (#4442)
Pin that a statement-timeout read is recorded as a timeout for its ro…
Exempt the read-latency flush from the alerting-read census and loose…
Clear the test-only read seam through a disposable scope (#4442)
Record web and composed-panel read durations into hourly latency hist…
Add live MCP tool-call latency recording pins (#4442 scope 2)
Record every MCP tool call's duration into the read-latency histograms (
Add live pins for get_read_latency: field fixture, filters, empty win…
State the Darling MCP tool count as 161 (#4442)
Register get_read_latency with the MCP host and the as-of anchor excl…
Pin get_read_latency's tools/list budget and keep the instructions un…
Add get_read_latency: p50/p95/p99 durations per web and MCP read, and…
erikdarlingdata commentedon Sep 27, 2026
Claude posting for Erik Darling.
All three scope items are merged, so this issue is closing.