Repository navigation
Viewer: InteractiveReadSeconds (15s) is derived from a single-read measurement but bounds ten-wide concurrent fan-outs #3004
Description
ViewerCommandDeadlines.InteractiveReadSeconds was set to 15 s by #2901 and now bounds every interactive store read in PerformanceMonitor.Darling.Viewer. The value was derived from a single-read worst case. Fifteen of this project's read sites issue their reads concurrently, and #2901's own rig measured that population at 24.4–64.1 s. The constant is therefore derived from the wrong population, and the widest fan-out cannot complete under it.
The derivation error
#2901 justified 15 s as "5x the 3.01 s worst measured shipped read" — TopQueriesSql over a 30-day custom range at production per-server density, measured alone.
The same PR body reports the concurrent figure two paragraphs later:
Ten concurrent 30-day reads on the rig above measured 24.4-64.1 s, six of them past the silent default they used to inherit; cutting each at 15 s returns permits sooner, which is the outcome to want in that state.
Both numbers were in the PR. The ceiling was taken from the first and applied to reads that run as the second.
Mechanism
CorrelatedTimelineLanesControl.RefreshAsync (CorrelatedTimelineLanesControl.xaml.cs:183) awaits one Task.WhenAll over exactly ten reads:
await Task.WhenAll(cpuTask, waitTask, blockingTask, deadlockTask, memoryTask, fileIoTask,
cpuBaselineTask, waitBaselineTask, ioBaselineTask, blockingBaselineTask);MaxPoolSize = 10 (ViewerSettings.cs:354), so all ten get a permit and all ten execute against the store at once. Npgsql's CommandTimeout bounds execution, so each of the ten is cut at 15 s.
Against the 24.4–64.1 s concurrent measurement:
- at the 30 s these reads inherited before Part of #2874 #2901, the reads at the fast end of the band completed
- at 15 s, none of them can
- the panel's auto-refresh timer re-issues the same batch, so it retries into the same wall
The window is reachable: ViewerServerTab.xaml.cs:476 passes GetOverviewCustomRange() straight through as fromDate/toDate, so the toolbar's custom range reaches the lanes. (RefreshAsync's own doc comment still says the Overview "always passes a fixed 24-hour window" and that the custom-range branch is "kept for a future custom-range picker" — that comment is stale.)
Note that the concurrent measurement makes concurrency here net negative: ten reads at 3.01 s each is 30.1 s of serial work, and running them together took 64.1 s of wall clock. Concurrency costs about 2x in total throughput on this store and inflates each read roughly 8–21x.
The shape is general, not one panel
Fifteen Task.WhenAll fan-outs over store reads in this project, all inheriting the same constant:
| site | width |
|---|---|
CorrelatedTimelineLanesControl.xaml.cs:183 |
10 |
ViewerServerTab.Filters.cs:121 |
6 |
ViewerServerTab.Memory.cs:104 |
6 |
ViewerServerTab.Postgres.cs:297 |
5 |
ViewerServerTab.Postgres.cs:586 |
5 |
FinOpsTab.Loaders.cs:470 |
5 |
ViewerServerTab.LatchSpinlock.cs:68 |
4 |
ViewerServerTab.QueryTrends.cs:74 |
4 |
ViewerServerTab.ConfigChanges.cs:36 |
3 |
ViewerServerTab.PlanCache.cs:55 |
2 |
ViewerServerTab.Postgres.cs:843 |
2 |
ViewerServerTab.CpuScheduler.cs:53 |
2 |
ViewerServerTab.CollectionHealth.cs:66 |
2 |
FinOpsTab.Loaders.cs:565 |
one per server in the inventory |
MainWindow.xaml.cs:1410 |
one per server in the fleet |
The last two are unbounded — Task.WhenAll(servers.Select(...)) — so their width is the fleet size, well past the ten permits the pool has.
What the fix has to do
The deadline's job is to bound one read. Ten reads competing for ten permits is a different question, and a per-read ceiling derived from a solo measurement cannot answer it. Whatever replaces it has to come from the concurrent measurement.
Three shapes, in decreasing order of how principled they are:
- Make the deadline a function of the fan-out width, so a read issued as part of a W-wide batch gets a ceiling that covers what a W-wide batch was measured to cost. Cannot make anything slower than it is today.
- Cap the fan-out behind the pool size. This addresses the cause rather than the ceiling, and because concurrency is net negative on this store it would likely make the panel faster as well as making a 15 s per-read deadline honest. Choosing the cap needs a concurrency sweep that does not exist yet — per-read duration at width 1, 2, 4, 6, 8, 10 — and picking it from the two data points available would mean a cap of about 2, which would serialize every panel in the viewer on the strength of a two-point extrapolation.
- A per-panel override. Cheapest, least principled, and leaves the same trap for the next fan-out.
Also worth stating for the record: for the two unbounded fan-outs the binding constraint is not the command deadline but ConnectionTimeoutSeconds (default 5 s), since reads past the tenth wait for a permit and then throw a connect error — the misattribution #2901's own doc comment describes. That is a distinct defect from this one and predates #2901.
Activity
erikdarlingdata commentedon Sep 5, 2026
Two things #3007 does not close, recorded here so the follow-up work is not re-derived.
1. The concurrency cap still needs a measurement. #3007 takes shape (1) — the deadline scales with the declared fan-out width — and explicitly declines shape (2). The reason is that the cap cannot be chosen from what is known: the two data points available (3.01 s at width 1, 24.4–64.1 s at width 10) extrapolate to a per-read worst of roughly
3.01 + 6.79(C−1), which stays under 15 s only to aboutC = 2. Capping every fan-out in the viewer at two on a two-point extrapolation would serialize panels that are probably fine. The measurement that decides it is per-read duration at width 1, 2, 4, 6, 8, 10 on a store at production density — and because concurrency measured net negative here (30.1 s of serial work took 64.1 s of wall clock), a cap is likely to make the widest panel faster, not just safer.2. The pool clamp covers the managed pool only.
FanOutReadSecondsclamps the width atViewerSettings.ManagedMaxPoolSize, which is correct for a managed store becauseMaxPoolSize = 10is set on the managed-derived connection string.ViewerSettings.Parsedoes not impose it on a bring-your-own store: that path returns the operator'spostgres.connectionStringverbatim, so the pool is whatever they configured, and Npgsql's own default is 100.So on a BYO store with a large pool, a fleet-wide per-server fan-out (
FinOpsTab.Loaders.cs's inventory overlay,MainWindow's overview cards) faces fleet-many-way contention while the deadline covers only ten-way. It is strictly better than the solo constant it replaces in every deployment, and it is not complete in that one.Fixing it properly means the deadline has to read the live pool size, which means threading runtime settings into a static constant family that was deliberately built as compile-time constants. The cheap version — parsing
MaxPoolSizeout of the connection string once atViewerDataServiceconstruction and passing it toFanOutReadSeconds— is probably the right shape, but it changes the signature every command site would eventually want, so it belongs with whatever resolves item 1 rather than ahead of it.erikdarlingdata commentedon Sep 5, 2026
Closed by #3007, merged to
devas406d644603.What landed
The deadline is now derived from fan-out width rather than from a solo read.
ViewerReadFanOut.Of(W)declares a batch's width;CurrentInteractiveReadSecondsreturnsmax(InteractiveReadSeconds, W * FanOutLaneSeconds)with both factors clamped into1..MaxPoolSizebefore the multiply.FanOutLaneSeconds = 7is 64.1s/10 rounded up, so a ten-wide batch gets 70s where it had 15s. 17 production declaration sites, 194 stamp lines converted.This is shape 1 of the three this issue named. The asymmetry decided it: scaling the deadline cannot make anything slower — solo and unscoped reads are bit-identical to pre-PR behavior — whereas capping concurrency (shape 2) changes how every panel behaves with no signal here that would say the model was wrong.
Scoped out by this issue itself, and still true
This issue's own closing paragraph separated the unbounded fan-outs from the defect it was filing:
That remains unfixed, and it is the reason the two widest sites in the table above — the per-server inventory and fleet fan-outs — are not repaired by a wider command deadline. Filed separately rather than left in a closed issue's comment.
Two other bounds are recorded there as well: the concurrency sweep that would validate 70s (or decide a cap), and the BYO-store pool that
ViewerSettings.Parsenever clamps.Not verified
70s has not been measured as sufficient. It is 9% over the worst read in a ten-wide batch on #2901's rig — a local rig at production density, not a production store. No runtime observation exists. If the real ceiling is worse than 64.1s this is still short, in the safe direction.
Restore viewer/storage-MCP client deadlines to their #3004 bands
Raise the viewer and mcp roles' default statement_timeout from 15 s t…