You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
An abandoned collection cycle records that it stopped, not what it was doing #2864
A collector cycle that hits the #2673 120s wall-clock budget records ABANDONED and nothing else. It does not record what it was doing, how far it got, or what the rest of its sweep looked like — so a real production abandonment can be investigated for hours and still end in "insufficient evidence."
This issue is not a fix for the stalls. It is the telemetry that would make the next one diagnosable.
What an investigation could and could not establish
Over 42 servers / 42,000 runs / 1.81h, ten runs exceeded 60s and two abandoned. That was enough to separate two populations by hand:
Population A — a large query doing real work.query_store at 60-73s returning 7,371-12,557 rows, while every other collector in the same sweep ran at or below its baseline (0.3-0.4x). One sweep: wait_stats 1ms, latch_stats 1ms, spinlock_stats 2ms, then query_store 71,977ms for 12,557 rows. Not a stall.
Population B — sweep-wide degradation. Light collectors at 34-47x baseline in the same sweep (wait_stats 380ms against a 2ms prior sweep), before two heavy collectors each burned exactly 120s and abandoned with rows_collected = 0.
The distinction matters because the fixes are unrelated, and today it required manual cross-referencing of neighbouring rows to see.
What could not be established, and why:
rows_collected = 0 is rows stored. There is no way to tell whether the run never received row 1 or stalled at row 149 — which is exactly the split between "could not execute" and "could not drain".
The wait series has a 6.5-minute hole across the event. Collectors run strictly sequentially per server, so a stalled collector blocks every other observation of that target — including wait_stats, waiting_tasks and dmv_blocking_snapshot, the three that would explain it. waiting_tasks ran 2 seconds after the stall cleared and returned 0 rows. The instrument stops sampling exactly when the thing being measured happens.
Proposed
1. Abandon forensics (client-side only, no target contact). When the budget fires, record what the client already knows: which phase it was in, rows read so far, bytes read so far, elapsed-in-phase, and the timestamp of the last successful read. No new connection, no new query, no target load. This alone separates "never got row 1" from "stalled mid-drain".
2. Record the run's own session_id.@@SPID on open, essentially free. Without it no retrospective join to any snapshot is possible even where one exists.
3. Record concurrent light-collector latency in the same sweep. The A/B signature above is a ratio against that server's own baseline. Capturing it at abandon time makes the two populations separable automatically rather than by hand.
4. Out-of-band watchdog (design input wanted). When an item exceeds N seconds, capture sys.dm_exec_requests and wait_type for the stalled spid over a separate connection. This is the only proposal that can observe Population B, because the per-target instruments are queued behind the stall. It is also the only one that must break the sequential model, and it opens a connection to a target that is already not responding — so it needs bounding (one shot, hard timeout, never retried) and is worth designing deliberately rather than adding alongside 1-3.
Notes
Items 1-3 are purely additive: no new queries against monitored servers, no schema change to what is collected, no behaviour change to any collector.
ABANDONED as a status only exists since A wall-clock-budget-abandoned cycle is recorded as SUCCESS in collection_log #2801; before it, budget-abandoned cycles recorded SUCCESS with rows_collected = 0. Any historical analysis must count the shape (duration at the budget with zero rows), or it measures that deploy rather than the thing it is looking for.
Separately: get_collection_log returns the most recent N within hours_back with no offset, capping analysis at ~1.8h per server at limit 1000. Backward paging would have allowed a before/after against the monitoring-box resize; that comparison could not be run.
Items 1-3 built and merged in #2868 (b6f571b6), schema rung V109.
drain_rows_read / drain_bytes_read — from a counting decorator around the provider reader, so none of the 66 collectors' own read loops needed editing and none can forget to count.
drain_last_read_ms — the one that carries the diagnosis. Subtracted from sql_drain_ms it gives the time the reader sat with nothing arriving, which is what separates a slow stream from a stalled one when both end at the budget with a positive row count.
target_session_id — read off the open connection as a client property, never SELECT @@SPID.
sweep_peer_max_ms — the slowest non-budgeted collector in the same body, captured at DISPATCH so the fire-and-forget collectors are attributed to their own body rather than an unrelated later tick.
Closing rather than leaving open for item 4: Fixes #N only fires on the dev→main release merge, so completed work otherwise sits open until release. Item 4, the out-of-band watchdog, is refiled as #2880 — it is the only proposal that must break the sequential model and open a connection to an already-unresponsive target, so it wants a design decision rather than being carried as an open checkbox on a finished issue.
Summary
A collector cycle that hits the #2673 120s wall-clock budget records
ABANDONEDand nothing else. It does not record what it was doing, how far it got, or what the rest of its sweep looked like — so a real production abandonment can be investigated for hours and still end in "insufficient evidence."This issue is not a fix for the stalls. It is the telemetry that would make the next one diagnosable.
What an investigation could and could not establish
Over 42 servers / 42,000 runs / 1.81h, ten runs exceeded 60s and two abandoned. That was enough to separate two populations by hand:
Population A — a large query doing real work.
query_storeat 60-73s returning 7,371-12,557 rows, while every other collector in the same sweep ran at or below its baseline (0.3-0.4x). One sweep:wait_stats1ms,latch_stats1ms,spinlock_stats2ms, thenquery_store71,977ms for 12,557 rows. Not a stall.Population B — sweep-wide degradation. Light collectors at 34-47x baseline in the same sweep (
wait_stats380ms against a 2ms prior sweep), before two heavy collectors each burned exactly 120s and abandoned withrows_collected = 0.The distinction matters because the fixes are unrelated, and today it required manual cross-referencing of neighbouring rows to see.
What could not be established, and why:
rows_collected = 0is rows stored. There is no way to tell whether the run never received row 1 or stalled at row 149 — which is exactly the split between "could not execute" and "could not drain".collection_logcarries onlysql_duration_ms. The Phase split (wm/open/drain) is not emitted for server-scoped collectors, so the largest collector on the fleet cannot be attributed #2851open:/drain:split exists solely as app-log text, so it cannot be queried, aggregated or trended (tracked separately).wait_stats,waiting_tasksanddmv_blocking_snapshot, the three that would explain it.waiting_tasksran 2 seconds after the stall cleared and returned 0 rows. The instrument stops sampling exactly when the thing being measured happens.Proposed
1. Abandon forensics (client-side only, no target contact). When the budget fires, record what the client already knows: which phase it was in, rows read so far, bytes read so far, elapsed-in-phase, and the timestamp of the last successful read. No new connection, no new query, no target load. This alone separates "never got row 1" from "stalled mid-drain".
2. Record the run's own
session_id.@@SPIDon open, essentially free. Without it no retrospective join to any snapshot is possible even where one exists.3. Record concurrent light-collector latency in the same sweep. The A/B signature above is a ratio against that server's own baseline. Capturing it at abandon time makes the two populations separable automatically rather than by hand.
4. Out-of-band watchdog (design input wanted). When an item exceeds N seconds, capture
sys.dm_exec_requestsandwait_typefor the stalled spid over a separate connection. This is the only proposal that can observe Population B, because the per-target instruments are queued behind the stall. It is also the only one that must break the sequential model, and it opens a connection to a target that is already not responding — so it needs bounding (one shot, hard timeout, never retried) and is worth designing deliberately rather than adding alongside 1-3.Notes
ABANDONEDas a status only exists since A wall-clock-budget-abandoned cycle is recorded as SUCCESS in collection_log #2801; before it, budget-abandoned cycles recordedSUCCESSwithrows_collected = 0. Any historical analysis must count the shape (duration at the budget with zero rows), or it measures that deploy rather than the thing it is looking for.get_collection_logreturns the most recent N withinhours_backwith no offset, capping analysis at ~1.8h per server atlimit1000. Backward paging would have allowed a before/after against the monitoring-box resize; that comparison could not be run.Server names in this issue are synthetic.