query_store probe: ~99% of probe: phase is client-side, not the database (measurement) #2831

Activity

1. erikdarlingdata commented on Sep 3, 2026

   

   erikdarlingdata

   on Sep 3, 2026

   Last edited by erikdarlingdata

   OwnerAuthor

   More actions

   Client-side measurement (recommendation 1) — the cost is inside the service process, not the database, wire, or Npgsql

   Npgsql harness built against Npgsql 10.0.3 (the box's exact version), run on the use1 monitoring host
   against the live store, issuing the shipped QueryStorePlanMap.TouchAndProbeSql extracted from source.
   Framework-dependent net10.0, run under the box's own dotnet 10.0.11 runtime. All DML inside
   BEGIN … ROLLBACK. Target is the busiest pair, server_id=2067296744 / OMEGA (115,973 map rows) — the same
   server the production numbers below come from.

   The decisive comparison

   Same batch size, same server, same store, same box, same Npgsql:

   	N=1,356	N=2,935
   harness, references an active cycle would actually pass	36.2 ms (0.0267 ms/row)	111.6 ms (0.0380 ms/row)
   production probe: (n=3,338 paired lines, today)	~1,451 ms (p50, at rows p50=1,356)	~2,512 ms (rows 1200+ band)

   ~40×. Production fleet-wide probe: is p50 1.083 ms/row, p90 4.06, and it is flat across batch-size
   bands (0.93 / 0.86 / 0.90 / 1.23 for bands 0-150 / 150-400 / 400-1200 / 1200+), so the phase is genuinely
   linear in N. The harness is 0.027–0.038 ms/row on the same work.

   Phase split at the shipped call site (median of 7)

   N	build arrays	marshal (AddWithValue)	ExecuteReaderAsync	read loop	total
   78	0.013	0.014	5.04	0.27	5.30
   272	0.043	0.013	8.48	0.58	9.08
   847	0.087	0.024	26.4	2.39	29.2
   2000	0.155	0.012	57.8	2.55	62.0

   Array construction and parameter marshalling are noise — 0.01–0.16 ms, under 0.3% of the phase at every
   size. The read loop is 0.2–2.6 ms. ExecuteReaderAsync is ~95%+ of the client-side time. Controls: SELECT 1
   round trip 0.30–0.49 ms; receiving 847 rows with no parameters 2.2–3.2 ms; sending 4×847 arrays and receiving
   one row 4.1–6.6 ms.

   Two corrections to this issue

   Q2 is not closed — the "no headroom" conclusion was measured on the read-only path. The touched CTE
   guard means a batch of freshly-touched rows does zero UPDATE work. Selecting stale references instead
   (last_seen < now() - 2h, which is what the UPDATE path actually costs):

   N=847	fresh refs	stale refs
   total	29.2 ms	102.0 ms

   3.5×. The 3.35 ms V1 figure in this issue is the read-only path. I made the identical mistake first —
   I loaded references ORDER BY last_seen DESC and measured the cheap path without noticing.

   Q3's "no lock contention" does not survive controlled concurrency. Sampling pg_stat_activity found
   waiting_lock = 0, but running k concurrent probes directly shows contention specific to the UPDATE path:

   k concurrent	fresh refs (read-only)	stale refs (UPDATEs fire)
   1	23.5 ms	68.1 ms
   4	24.6 ms	145.7 ms
   16	23.9 ms	166.6 ms

   Read-only is flat; the UPDATE path inflates 2.4×. That is consistent with row-lock serialisation on the
   shared content-addressed query_plan_dim digests — the mechanism this issue speculated about and then
   dismissed. It is invisible to sampling because each statement is short.

   But none of that explains production

   Stacking every amplifier I could reproduce — stale refs (3.5×) × concurrency (2.4×) × in-process CPU pressure
   (1.4×, 2 spinners on the 4-core box) — reaches ~0.2 ms/row against production's 1.08. A 5–40× residual
   remains, and I did not isolate it.

   What it is not, by elimination: the query (3 ms server), the wire (0.3 ms RTT), parameter marshalling
   (0.02 ms), the read loop (2 ms), result-set size, or store-side concurrency at the read-only path.

   What is left is inside the service process. The box is 4 logical cores at 83% CPU; the service holds
   21 threads, 1,091 MB working set, 25,921 s CPU. I could not distinguish thread-pool queueing, GC pause, or
   the probe: timer enclosing more in practice than it does in isolation, without instrumenting the running
   service.

   The touch is working, so the touched-set is small: for OMEGA, last_seen is <1h on 2,420 rows and 1–2h on
   1,580, against 99,271 older than a day. Production batches (p50 1,356, p90 2,935) fit inside that fresh set —
   which is why the realistic-batch harness number above is the read-only path, and why it is the right
   comparison.

   Recommendations, revised

   Recommendation 2 (drop the constant arrays) — do not do it. Measured both ways, it is not safely better:

   N	shipped	scalar variant
   847 (fresh)	29.2 ms	18.0 ms
   847 (stale)	102.0 ms	71.6 ms
   2000 (fresh)	62.0 ms	309.1 ms
   2000 (stale)	235.7 ms	462.0 ms

   ~30% better in the operating range, but scalar parameters flip the plan at N=2000 and it becomes 2–5×
   worse. That cliff sits just above the production p90 batch size of 2,935. And the win it buys is ~30% of
   36 ms against a 1,451 ms phase — under 1%. Not worth the risk for the return.

   Recommendation 3 (present-cache) is the only lever, and it survives not knowing the mechanism. Production
   probe: is provably linear in N — flat ms/row across four batch-size bands, n=3,338. So cutting references
   probed by ~10× cuts the phase ~10× whatever the in-process cost turns out to be. That is the strongest
   argument for the cache and it does not depend on resolving the residual first.

   New recommendation: find the in-process cost. A 40× gap between the same statement in a quiet process and
   the service process is the whole remaining story, and no SQL-side change can touch it. It wants
   ThreadPool queue-delay and GC-pause instrumentation inside the service, or a probe: sub-split separating
   the awaited round trip from everything else in the timed region.

   Harness at C:\Windows\Temp\probeharness on the box; no production code changed, nothing deployed.
2. erikdarlingdata commented on Sep 4, 2026

   

   erikdarlingdata

   on Sep 4, 2026

   Last edited by erikdarlingdata

   OwnerAuthor

   More actions

   The residual, with the true denominator: connection acquisition is not in it, and the gap is ~8–10x, not 40x

   Follow-up to the client-side measurement above. Two things changed under it, and both had to be established
   before the residual could be sized.

   First, the instrument improved: probed is deployed now

   The comment above had to use the parent line's batch row count as a proxy denominator because probed was
   merged but not shipped. It is shipped now — the use1 monitoring host runs 3.6.0.0, installed
   2026-09-04 04:02Z, service restarted 04:10:25Z, and the split lines carry (N chunk(s), N ids, N probed).
   So every number below uses the true reference count as the denominator, not a proxy.

   Window: 04:15Z–16:21Z today, steady state, no restart or deploy inside it. 7,320 split lines,
   12,805,558 references probed, 114 distinct (server, database) pairs. Read-only throughout — I parsed
   the service log and changed nothing.

   Two premise corrections while I am here, both from the box rather than from inference:

   • The box is 8 logical processors, not 4, and total CPU sampled 36%. The service has averaged
     ~1.2 cores busy since its restart (53,092 s CPU over 43,415 s wall). The "4 logical cores at 83%" the
     comment above reasoned from is not the current state, so the "in-process CPU pressure 1.4x" term does not
     have the headroom that argument gave it.
   • probe: is now 59.9% of fetch-phase time (5,771 s of 9,627 s), not 87.1%.

   The acquisition hypothesis fails three independent ways

   The idea was that #2819's 673 ms store-connection acquisition floor is a large component of production
   probe:, and that a dedicated-connection harness omits exactly that. It is not, for three separate reasons.

   1. Structurally, acquisition is not inside the timer. Post-#2822 the fetch borrows the collector body's
   connection. probeWatch.Restart() is immediately followed by RestoreBorrowedStoreConnectionAsync, which
   returns on a State == Open check and does nothing on the ordinary pass. There is no OpenConnectionAsync
   in the measured region at all. The comment above already said this; it is worth restating because it means
   the hypothesis was dead before any measurement.

   2. #2819's 673 ms figure is itself the denominator bug that #2823 was filed to fix. #2819 read the
   0 ids bucket as "issues no SQL, therefore measures acquisition alone". But {Ids} is
   PerItemPlanIdsAttempted, incremented only in the target-fetch chunk loop — it counts what came back
   missing, not what was probed. 0 ids means nothing was owed, not nothing was probed. Measured today:

   bucket	n	probe: p50	probed p50 in the same rows	probed mean
   ids == 0	764	174 ms	780 references	654
   probed == 0	0	—	—	—

   The "acquisition floor" bucket was doing real probe work on ~650–780 references, and 174 ms / 780 refs is
   ordinary for this fleet. A genuinely zero-input bucket does not exist in the log, because the probe is
   only reached when there is input. The source comment at the probed stamp already names #2819 as one of the
   two places this produced a phantom gap; this is that same artifact.

   3. Empirically there is no fixed per-call cost at all — the intercept is negative. Least squares of
   probe: against probed, on band medians:

   set	slope	intercept	R²
   plan_fetch, 7 bands	0.2885 ms/ref	−11.8 ms	0.956
   both phases, 5 bands	0.3663 ms/ref	−41.5 ms	0.998

   A per-call acquisition cost is a positive intercept. There isn't one. And the cheapest real band is
   probed p50 = 80 → probe: p50 = 4 ms total — fifteen times less than the claimed 673 ms floor, so
   that floor cannot be hiding anywhere in this distribution.

   Verdict: the acquisition hypothesis is falsified. Not "unsupported" — contradicted.

   The gap is entirely marginal, and it is ~8–10x

   Same fit against the harness numbers from the comment above (N=272…2935): slope 0.0376 ms/ref,
   intercept −6.1 ms. So:

   	fixed per call	marginal per reference
   harness (quiet process, same box, same store, same driver)	~+5 ms	0.0376 ms/ref
   production, plan_fetch	−11.8 ms	0.2885 ms/ref
   production, both phases	−41.5 ms	0.3663 ms/ref

   Marginal gap 7.7x (plan) to 9.7x (both phases). At matched N, against the harness's measured points:

   N	production probe: p50	harness (measured / interpolated)	ratio
   ~80	4 ms	5.30 ms (N=78, measured)	0.75x — production is FASTER
   ~274	37 ms	9.08 ms (measured)	4.1x
   ~912	281 ms	31.6 ms	8.9x
   ~1,789	642 ms	53.5 ms	12.0x
   ~3,241	1,135 ms	123 ms	9.2x

   At small N production and the harness agree, and production is if anything cheaper. The gap opens with N
   and then flattens. That is the signature of a marginal-cost difference with no fixed-cost difference.

   And the 40x headline: recomputed on today's build, probe: p50 is 459 ms at probed p50 = 1,507, against
   1,451 ms at ~1,356 in the comment above. That is a 3.2x improvement in production probe: itself. Today's
   ratio at that N is 8.9x. I cannot attribute the 3.2x — the build changed and the load changed between
   the two measurements, and I have no way to separate them from the log. Stated so it is not read as a fix.

   Concurrency does not move the median

   Controlled for N (probed 1201–2500), bucketed by a weak concurrency proxy — distinct servers emitting
   split lines in the same 2-second window:

   concurrent servers	n	probed p50	ms/ref p50	ms/ref mean
   1	2,083	1,789	0.374	0.469
   2	563	1,801	0.393	0.516
   3	98	1,906	0.370	0.648
   4	52	1,503	0.528	0.721
   5	13	1,692	0.294	0.504

   The median is flat; only the mean drifts, so what concurrency touches is the tail, not the centre. Label
   this weak: sweeps emit lines sparsely, so the proxy under-counts and 73% of rows land in bucket 1. But it
   points the same way as the harness's own read-only arm in the comment above, which was flat at k=1/4/16
   (23.5 / 24.6 / 23.9 ms). Two weak-to-moderate instruments, same answer: no median concurrency effect.

   The cost is invariant in every direction I can vary it

   This is the part I did not expect. The marginal per-reference cost barely moves:

   Across 12 hours, while probed p50 swung 2.7x (1,019 → 2,782):

   hour (Z)	05	06	07	08	09	10	11	12	13	14	15	16
   ms/ref p50	.278	.412	.309	.316	.416	.357	.436	.366	.317	.321	.312	.248

   Across all 114 (server, database) pairs (per-pair median ms/ref, probed 700–3000):

   pairs	min	p10	p50	p90	max
   114	0.1609	0.2229	0.2825	0.3762	0.4431

   p90/p10 = 1.69x; the whole fleet fits inside 2.75x. 114 pairs whose map sizes span three orders of
   magnitude, all paying within a 1.7x band per reference. If the marginal cost were dominated by data-dependent
   store work — index page residency, per-pair row counts — I would expect a far wider spread than that.

   So the residual is not load-dependent, not batch-size-dependent above ~900 references, not
   concurrency-dependent at the median, and not pair-dependent. It behaves like a near-deterministic
   per-reference cost that is ~8–10x what the same code path costs in a quiet process on the same box.

   One crack in the "purely client-side" reading, and it argues the other way: plan_fetch and text_fetch run
   the identical client method — same array construction, same AddWithValue calls, same five result
   ordinals, differing only in the SQL string and the tables it touches. Their per-reference costs differ by
   37% (p50 0.280 vs 0.383 ms/ref). Client-side work cannot explain a table-dependent difference, so some
   of the marginal cost is store-side. I cannot size that share from the log.

   The decomposition, and the residual

   term	contribution to the median gap	how established
   store-connection acquisition	0x	measured + structural (three ways above)
   concurrency	~1.0x at the median	measured (weak proxy) + harness read-only arm
   stale/UPDATE path	not separable — see below	not measurable from the log
   residual	~8–10x, i.e. essentially the whole gap

   The residual is the finding. Stacking the amplifiers does not get there, and two of the three are now
   measured at ~1.0x rather than assumed.

   Where I have to stop, and the one instrument that would move this

   The stale-path term is not measurable with what the phase logs today, and I want to correct the comment
   above rather than build on it. That comment reasoned the production median is the fresh path because the
   touched set is small. The delivered cadence in this window is ~22.6 min/cycle (32.1 cycles in 12.09 h at
   114 pairs x 2 phases), not ~5 min, so against a 1-hour touch guard a pair is probed ~2.6 times per hour and a
   large minority of probes should be firing some UPDATEs — with a partial, staggered stale set, not the
   clean all-or-nothing the fresh/stale harness arms model. What I can see is the tail: median within-pair
   p92/p50 of ms/ref is 2.73x (plan) and 1.98x (text), and 9.7% / 6.7% of probes exceed 3x their own
   pair's median. That is consistent with the UPDATE path costing ~3x, matching the harness. It does not
   tell me what share of the median pays it.

   Recommendation: log the touched cardinality. The statement's map_touch CTE already knows exactly how
   many rows it updated. Surfacing that count into the split line — a touched: field beside probed: — would
   split the fresh and stale paths in production, on real batches, and settle Q2 without a harness. It is
   the same move probed was, and probed is what made this comment possible: every conclusion above that the
   previous one could not reach came from having the right denominator. Cheap, additive, no behaviour change.

   That, plus the one read I could not get: pg_stat_statements on the store, for the two probe statements —
   calls, total_exec_time, rows, shared_blks_hit, shared_blks_read. total_exec_time / rows is the
   production server-side cost per reference, directly. If it comes back near 0.25 ms/ref the residual is
   store-side and "the cost is inside the service process" is wrong; if it comes back near 0.005 the residual is
   in-process and the ThreadPool/GC instrumentation recommended above is the right next step. One query
   splits the last remaining fork, and it is read-only.

   What I measured vs. what I reasoned

   • Measured: everything in the tables above, from the service log on the box over 04:15Z–16:21Z (7,320
     split lines, 12,805,558 references, 114 pairs), plus the deployed build, thread/CPU/core facts, and the
     store's retention and background-job state (296 GB, 18 retention policies, 0 held, worst background job
     at 1.7% of its own cadence — so the store is not visibly saturated by its own maintenance).
   • Reasoned, not measured: that the 37% plan-vs-text difference implies a store-side component (it follows
     from the client code being identical, but I did not measure the split); that the tight across-pair spread
     argues against a data-dependent cost; and the ~2.6-probes-per-hour stale-share argument, which is arithmetic
     from the observed cadence, not an observation of UPDATEs firing.
   • Not measured at all: the harness side is the previous comment's numbers, from yesterday, on the
     previous build. I could not re-run it — the box has no .NET SDK, and the store-credential path a harness
     needs was unavailable to me this session. So every ratio above is cross-day and cross-build on one side.
     The k=1/2/4/8 and pooled-vs-dedicated arms are not done. Given acquisition is falsified three ways
     and concurrency measures ~1.0x at the median, I do not think the pooled arm would change the conclusion,
     but I have not run it and am not claiming it.

   Leaving this open: the residual is unresolved and it is now the entire question.
3. erikdarlingdata commented on Sep 4, 2026

   

   erikdarlingdata

   on Sep 4, 2026

   OwnerAuthor

   More actions

   The connection-acquisition hypothesis is falsified, and the 673 ms floor it rested on is a phantom this repo already documents

   I proposed that the harness-vs-production gap was store-connection acquisition under production concurrency. It is not, and I should have caught that before proposing it. Two of the three refutations are verifiable from source alone.

   1. There is no acquisition in the timed region. Post-#2822 the probe borrows the collector body's connection. probeWatch.Restart() (DarlingCollectorRunner.cs:2440) is immediately followed by RestoreBorrowedStoreConnectionAsync (:2441), whose first statement is if (storeConnection.State == ConnectionState.Open) return true;. On the happy path nothing is opened inside the measurement. The deployed box (build 3.6.0.0, installed 04:02Z) is well past #2822.

   2. The 673 ms floor from #2819 is a documented conflation artifact. {Ids} is PerItemPlanIdsAttempted, incremented only inside the target-fetch chunk loop — it counts references that came back missing, not references probed. So #2819's "0 ids" bucket was never a zero-work bucket. Measured over 12 h: the ids == 0 bucket has n=764 with probe: p50 174 ms and probed p50 = 780 references. The probed == 0 bucket has n = 0 — it does not exist.

   This is already pinned in the repo. Darling/Darling.Tests/ProbeDenominatorTests.cs says it outright: "Dividing one by the other produced a phantom finding twice. #2819 read a 673ms cost on…" and "the exact conflation that produced a phantom 140x gap in #2819 and made #2822 read as a…". A git grep for #2819 in source would have shown me the number was retracted before I built a hypothesis on it. That is the lesson worth keeping from this pass.

   3. The regression has a negative intercept. Least-squares of probe: against probed: plan_fetch slope 0.2885 ms/ref, intercept −11.8 ms (R² 0.956); both phases 0.3663, −41.5 (R² 0.998). Acquisition — or any fixed per-call cost — requires a positive intercept. The cheapest band is 80 references at 4 ms total, fifteen times less than the claimed floor.

   The gap is ~8–10×, not 40×, and it is entirely marginal

   probed is deployed now, so the true denominator exists and the earlier proxy is unnecessary. Window 04:15Z–16:21Z, 7,320 split lines, 12.8M references, 114 (server, database) pairs.

   	fixed per call	marginal per reference
   harness	~+5 ms	0.0376 ms
   production (plan)	−11.8 ms	0.2885 ms

   Matched against the harness's own measured points: 0.75× at N≈80 — production is faster — then 4.1× at 274, 8.9× at 912, 12.0× at 1,789, 9.2× at 3,241. The gap opens with N and then flattens. That is a marginal-cost difference with no fixed-cost difference, which is a different question from the one this issue opened with.

   Production has also improved 3.2× on its own: probe: p50 is now 459 ms at probed 1,507, against 1,451 ms at ~1,356 yesterday. Build and load both changed between those readings and they cannot be separated — not claimed as an attribution.

   Two stale figures in this issue's own framing

   • probe: is 59.9% of fetch-phase time now, not 87.1%.
   • The linearity claim was too weak, and the stronger version helps: ms/ref rises across bands (0.057 → 0.118 → 0.32 → 0.377 → 0.331), i.e. sub-linear below ~900 references and linear above. Combined with the negative intercept this rules out every fixed-per-call explanation — acquisition, per-call GC pause, per-call thread-pool dispatch — not just the flat-cost ones.
   • Separately: the commit this issue's body says the box runs cannot be resolved in this repository (the API returns 422, no commit for that SHA). That provenance line is unverifiable as written.

   What the residual actually looks like — invariance is the finding

   After acquisition (0×), concurrency (~1.0× at the median, N-controlled) and the measurable stale-path signal, the residual is essentially the whole ~8–10×, and it is remarkably stable:

   • Across 12 hours, ms/ref p50 stays 0.248–0.436 while N swings 2.7×.
   • Across all 114 pairs, per-pair median ms/ref is p10 0.223 / p50 0.283 / p90 0.376 — p90/p10 = 1.69×, the whole fleet inside 2.75×.

   A near-deterministic per-reference cost, ~8–10× the same code in a quiet process. Contention explanations predict variance with load; this does not vary. That invariance is the substantive finding of this pass.

   And one crack in the "purely client-side" framing: the plan and text probes run the identical client method yet differ 37% per reference (0.280 vs 0.383 ms). Client code cannot explain a table-dependent difference, so some of the marginal cost is store-side. Unsized, and it is the thread worth pulling.

   Correcting the stale-path reasoning, against my own earlier comment

   I reasoned about the touched CTE guard assuming a ~5-minute cadence, which would make a given reference fresh on ~11 of 12 cycles. Delivered cadence is ~22.6 minutes per cycle, so against the 1-hour guard a pair is probed ~2.6×/hour and a large minority of probes should legitimately fire UPDATEs, partially and staggered rather than all-or-nothing. Only the tail is visible today (within-pair p92/p50 = 2.73× plan, 1.98× text; 9.7%/6.7% of probes above 3× their pair's median). The stale share of the median is not measurable from what the phase line currently logs.

   The next step needs no new tooling — and the one that does is unavailable

   Recommended: log the map_touch CTE's cardinality as a touched: field on the existing phase line. That splits fresh-vs-stale in production, on real batches, and closes the open question without a harness — exactly the move probed was, and probed is what made this pass possible at all.

   The measurement that would decide store-side vs in-process is one read: pg_stat_statements on the store for the two probe statements, giving total_exec_time / rows as server-side ms/ref directly. Near 0.25 puts the residual in the store; near 0.005 puts it in the process. I cannot obtain it. It requires store credentials, and neither the read-only monitoring surface nor any sanctioned path here exposes them — this is a limit on what can be measured, not a task waiting to be scheduled. Stating it plainly so nobody plans on it.

   Issue stays open: the residual is now characterised (marginal, invariant, ~8–10×) but not located.
4. erikdarlingdata commented on Sep 4, 2026

   

   erikdarlingdata

   on Sep 4, 2026

   OwnerAuthor

   More actions

   Closing. The residual is characterised but not located, and the one measurement that would locate it is not obtainable.

   What is established:

   • The cost is not in the database — server execution is ~1–3.3 ms for 847 references.
   • It is not connection acquisition, and the 673 ms floor from query_store probe phase is connection acquisition, not SQL: probe query is 0.4-37ms, probe phase is 673-6663ms #2819 that suggested it is a documented phantom: ProbeDenominatorTests says so outright, because {Ids} counts references that came back missing, not references probed. Measured, the ids == 0 bucket has probed p50 = 780 references, and probed == 0 has n = 0 — that bucket does not exist. Post-Borrow the collector body's store connection in the Query Store fetches (#2819) #2822 the probe borrows the body's connection and there is no open inside the timed region.
   • It is not a fixed per-call cost of any kind. Least squares against the true denominator gives slope 0.2885 ms/ref with intercept −11.8 ms (R² 0.956); both phases 0.3663 / −41.5 (R² 0.998). A negative intercept rules out acquisition, per-call GC and per-call dispatch alike. The cheapest band is 80 references at 4 ms total.
   • The gap is ~8–10×, not 40×, and entirely marginal: 0.75× at N≈80 (production is faster), rising to 12.0× at 1,789 then flattening.
   • Its invariance is the real finding. Over 12 hours ms/ref p50 holds 0.248–0.436 while N swings 2.7×, and across all 114 (server, database) pairs per-pair median ms/ref is p10 0.223 / p50 0.283 / p90 0.376 — p90/p10 = 1.69×, the whole fleet inside 2.75×. Contention explanations predict variance with load; this does not vary.
   • One crack in "purely client-side": plan and text probes run the identical client method yet differ 37% per reference (0.280 vs 0.383 ms). Client code cannot explain a table-dependent difference, so some of the marginal cost is store-side. Unsized, and it is the thread worth pulling if this is ever revived.

   Why it cannot be located from here. The deciding read is one query — pg_stat_statements on the store for the two probe statements, where total_exec_time / rows gives server-side ms/ref directly: near 0.25 puts the residual in the store, near 0.005 in the process. It requires store credentials, which are not available through any sanctioned surface here. That is a limit on what can be measured, not a task awaiting scheduling.

   Two figures in this issue's body are stale, recorded rather than edited: probe: is 59.9% of fetch-phase time now, not 87.1%; and the commit its body says the box runs cannot be resolved in this repository (422 for that SHA).

   Also corrected: the stale-path reasoning here assumed a ~5-minute cadence, which would make a reference fresh on ~11 of 12 cycles. Delivered cadence is ~22.6 minutes, so against the 1-hour touched guard a pair is probed ~2.6×/hour and a large minority of probes should legitimately fire UPDATEs. Only the tail is visible today (within-pair p92/p50 = 2.73× plan, 1.98× text).

   The concrete next step, if this is revived — and it needs no credentials: log the map_touch CTE's cardinality as a touched: field on the existing phase line. That splits fresh-versus-stale in production on real batches, which is exactly the move probed was — and probed is the only reason this pass could correct #2819 at all.
5. erikdarlingdata

   closed this as completedon Sep 4, 2026