sweep_pressure reports OK while the body overruns: an infrequent 37s collector is invisible to an amortized model #2446

Found soaking dev on prod-sql-use2-monitor-01. The worker is logging real skipped relaunches while sweep_pressure reports the server healthy, and both are correct about what they measure.

The disagreement

Six overruns in three hours on one server:

12:58  shard-A         collection body has not completed after 66s of execution - skipping relaunch
13:29  shard-A         62s
14:01  shard-A         69s
14:23  single-C  62s
15:02  shard-A         60s
15:34  shard-A         61s

get_collection_health for that server, at the same moment:

busy_ms_per_minute: 12250    busy_percent: 20.4    verdict: OK

Every collector reads HEALTHY. Nothing in the tool says this server is collecting at a multiple of its configured interval, which is the exact condition #2296 was built to surface.

Why the model misses it, and why the model is not wrong

SweepPressureClassifier amortizes each collector's average duration by its own cadence. Reconciled against the reported number:

The total lands on the reported 12,250 ms/min, so the arithmetic is doing what it says. But index_object_stats takes 37 seconds in a body that has 60, and contributes 26 ms/min to the verdict. On the cycle where it coincides with query_store (13.8 s) and plan_correction (11.2 s), the serialized body is over 60 s — which is precisely what the log shows, roughly every half hour.

So the metric answers "can this server's total collection demand fit inside its cadence on average", and the operator's question is "does any single sweep overrun". Those diverge exactly when one collector's single-run cost approaches the interval while its amortized cost is negligible — which is the signature of an infrequent heavy collector, not of a saturated server.

#2296's model choice was right and should not be reverted. Its own note explains that gap-based detection flags every server at fleet scale because delivered cadence stretches benignly from bounded sweep concurrency. Sustained demand is the correct primary signal. What is missing is a second dimension, not a different one.

What would close it

The data is already collected — avg_duration_ms per collector is right there in the same payload, and the interval is known. Two candidates:

A burst term in the verdict. Any collector whose single-run average exceeds some fraction of the fastest cadence is a body-overrun risk regardless of amortized cost. On this server index_object_stats at 37 s against a 60 s minute is 62% of the budget in one collector.

Say it in heaviest_collectors. That list currently ranks by amortized contribution, so the collector actually causing the overruns is not in it — index_object_stats is absent while three collectors that fit comfortably are named. Ranking or annotating by single-run cost would put the cause in front of whoever is looking.

Worth deciding whether the verdict should change at all, or whether this is a note rather than an AT_RISK. An operator told "SATURATED" for a once-daily 37-second collector would reasonably ignore the next one; a note saying "index_object_stats takes 37 s of a 60 s budget when it runs" is actionable and does not cry wolf.

Also worth a look while in here

query_store on this server reports enumeration yielded 0 items - nothing to collect this cycle on 958 of 1155 runs, and still costs 13,834 ms average. That is 83% of its executions doing nothing at a cost that dominates its amortized share. Possibly correct (the collector is on a use2 box whose Query Store went quiet by design after the #2296 split) — but if an enumeration that yields nothing still pays 13.8 s, that is worth understanding separately.

Edited: server and database identifiers replaced with role placeholders. This is a public repository and the originals named tenants and customer databases; the measurements, ratios and reasoning are unchanged, because those are what the issue is about.