Repository navigation
Page on a same-statement pileup while it is happening, from the collectors that already saw it (#3467) - #3469
Conversation
…current while it ran (#3467) A parameterized statement joining a #temp table flipped to a plan doing ~35x the logical reads. Every caller shares the parameterized plan, so ten tenants inherited the flip together, the copies convoyed on PAGEIOLATCH_SH, physical reads saturated the volume, and sixteen executions died at the client's 30-second command timeout. The episode ran about ninety seconds and self-cleared when the plan churned out. The store was AHEAD of it the whole time. The 20:10:31Z active-query snapshot held three concurrent copies of the one statement - 29,124 ms / 28,343 ms / 27,703 ms elapsed, 287,049 logical reads on the leader, two suspended on page IO - next to that same statement's 54 ms observation from 3.5 minutes earlier. Nothing could page on any of it: the timely findings were correct and capped under the threshold (ANOMALY_QUERY_DURATION 0.72, ANOMALY_BATCH_REQUESTS 1.0, ANOMALY_CPU_SPIKE 0.56, against notify_severity 1.5), the finding that DID clear it keys off query_store data that had not landed yet (~28 minutes), and no alert family covers the shape - long_running_query fires at 30 minutes and the victims died at 30 seconds, poison_wait does not class PAGEIOLATCH, CPU stayed near 50%. The tool watched an outage it could describe and could not report. * The finding. SAME_STATEMENT_PILEUP: N concurrent sessions on one statement at one snapshot instant, EVERY copy past a floor of elapsed time, at least one suspended on an IO-class wait, where that same statement's own snapshots across the prior 45 minutes were sub-second. The conjunction is what makes it specific rather than any single threshold - on four production servers x 24 hours (~4,160 snapshot instants) it fired exactly once, on this incident. * Placement is the point, not an implementation detail. It is evaluated on the COLLECTION cadence - the Darling worker's per-server sweep on the 30-second alert-sweep stamp, Lite's one-minute collection loop - immediately behind the collectors that write its input, so the 20:10:31Z snapshot produces a notify-eligible finding inside one collection cycle instead of 21 minutes later at the scheduled pass. It is NOT on the 5-360 minute analysis interval, and it deliberately does not inherit the scheduled pass's 24-hour data-span gate: that gate exists because fraction-of-period fact scoring distorts on short windows, and this finding's evidence is one snapshot plus one statement's own recent history. * Query Store independence is a requirement, not a convenience. On at least one production store class the Query Store readers are deliberately disabled (#2296) - the deployment this incident happened on, where tonight's plan-count evidence had to come from outside the product. A detector keyed on query_store identity would compile, test green against a QS-enabled fixture, and ship blind on exactly the servers that needed it. So the trigger, the baseline, the statement identity and the incident fingerprint are all computed from the active-query snapshot collector alone, and the statement key is the DMV's own query_hash off sys.dm_exec_requests rather than anything Query Store hands out. SameStatementPileupSourceCensusTests holds that from source for the detector and BOTH editions' readers: no query_store identifier in code or SQL, and the only tables the trigger path may read are the permitted collector tables. PLAN_REGRESSION and the #2138 force-plan bot stay what they were - follow-on enrichment where QS happens to be readable, never the trigger. * Thresholds from fleet distributions, named per #3060, each derivation stated on its constant in the unit the gate compares. Three sessions, because n>=2 same-statement groups number 484 in four server-days and are full of ordinary paired workloads while n>=3 past the elapsed floor numbered four; a two-session pileup therefore produces NO finding at any duration, which is the issue's "must not notify" held one step stronger than asked. Ten seconds, because the victims died at the client's 30-second timeout and a 10 s floor pages while they are still alive (the measured pack cleared it with 2.7x headroom). One IO-waiting session, not two: page-IO waits are millisecond-scale - the incident's suspended copies showed wait_time_ms 5 - so demanding two SIMULTANEOUSLY suspended makes the gate a coin flip on sampling phase, and specificity is carried by the conjunction instead. A 45-minute sub-second baseline window with at least one observation in it, which is what turns "normally fast" into a measurement: the three fleet shapes that reach n>=3 past ten seconds and are NOT incidents fail precisely this gate or the IO one - a sync job at n=5 with no IO wait and no prior observation, a permission rebuild whose priors were already at 11.9 s, a nightly catalog scan with no baseline at all. * Severity prices magnitude, not presence (#3462), in the unit the gate compares (#3463): sessions x the pack's minimum elapsed seconds, over 40 session-seconds per severity point, capped at the band ceiling. The measured incident - 3 x 27.703 s = 83.109 session-seconds - scores 2.0777 before the cap, 0.578 severity points above the 1.5 gate, and would still clear it with two of its three sessions discounted to the floor. The minimum firing shape (3 x 10 s = 30 session-seconds) scores 0.75: a persisted, drill-downable, visible finding that does not page. Notify-eligibility begins at exactly 60 session-seconds - three concurrent copies each past two-thirds of the timeout that killed the measured victims. * Recurrence folds instead of surprising. The measured statement re-drew its regressed plan later the same evening after fully self-clearing, so the finding gets the incident-fingerprint treatment every alert family already has: the IncidentId recipe extended with the statement identity, which keeps episode two on episode one's incident trail as an occurrence (first_seen / last_seen / occurrences, the drill-down an operator wants when deciding whether the plan deserves forcing) and gives #2138 a trigger token that is stable across episodes, plan churn, restarts and QS resets. The story path carries the statement identity too, so muting or cooling down one statement's pileup cannot answer another statement's question (#3459's rule). * Both editions, one detector. Lite's active-query snapshots feed the same seam - one shared collector definition, one table shape - so the predicate, the thresholds, the severity arithmetic, the fingerprint and the noise filters live once in PerformanceMonitor.Analysis as a pure function of snapshot rows, and each edition contributes only its window reader (Npgsql over query_snapshots, DuckDB over v_query_snapshots) and its sweep call site. * It adds no delivery semantics. The finding rides the existing analysis-finding machinery end to end - mute registry, two-phase materialize-enrich-persist, the (story_path_hash, incident_id) occurrence folding, notify_severity, the incident-keyed cooldown and its #2054 worsening rule - which also means a recurrence inside the 6-hour cooldown folds silently rather than re-paging, because a band-saturating severity cannot worsen. It IS a new delivery call site on both SKUs, which is a different thing and is accounted for as one: each sits under ShouldNotifyAnalysisFindings (master AND family), each carries its own #3465 census entry, and AlertMasterSwitchSurfaceTests pins both in both directions, with the predicate's call count raised from two to three. The README sentence that makes alerts_enabled a promise widened from "scheduled-analysis finding notifications" to "analysis finding notifications" - the narrower wording described a promise smaller than the one the gates keep, and an operator could reasonably have read a non-scheduled finding as escaping the switch, which is the exact confusion #3464 was about. * One read-side change, in both stores, for the same reason. The latest-batch read excludes pileup rows from DEFINING MAX(analysis_time) and overlays the ones newer than it. #2448's contract is that the newest batch is one indivisible statement about a server - the complete output of one pass - and a per-sweep row claiming the MAX would replace the last complete analysis with a single-row one, making a server read HEALTHIER mid-incident. That is #2448's own misreading arriving through the write cadence instead of a torn batch. The overlay is self-limiting (visible until the next scheduled pass completes, then in the windowed reads and its incident trail), and the epoch fallback keeps a pileup visible where analysis is disabled and no scheduled batch exists. No migration rung is claimed and none is needed: the finding persists through analysis_findings' existing columns, reads query_snapshots' existing columns, and keeps its one piece of cross-sweep state (the last-evaluated snapshot instant) in memory, where a restart costs at most one repeated evaluation that folds onto the same incident id. Tests: the 20:10:31Z incident reconstructed verbatim from the store's own rows (notify-eligible severity, statement identity, session and elapsed evidence, the drill-down's sessions and baseline, and the frozen value-stated advice quoting the measured wait type); the 20:07:01Z pre-incident snapshot must not fire; a busy-but-healthy eight-session parallel workload on DIFFERENT statements must not fire; a long-running single session must not fire; two sessions must not fire at 12 s, 29 s or 600 s; the fleet's three other deep groups must not fire, each failing a different gate; no-IO, stale-snapshot, noise-shape, one-deep-copy-among-fast-siblings and duplicate-session shapes must not fire; episode two folds onto episode one's incident id and mute key while the fingerprint still separates statements, databases and servers; the severity table is pinned at six points with the notify boundary expressed in session-seconds; and QS independence is pinned both behaviorally (the row type is the contract) and structurally (the source census over three files).
…p from outrunning the persist (#3467) Review of the pileup rung surfaced one behavior worth fixing and one worth stating out loud instead of fixing. The statement first, because it is the bigger admission. The detector's baseline gate reads the statement's prior in-flight observations - and the collector can make the episode itself one of them. If the first sample of an episode lands while the sessions are between the sub-second ceiling and the elapsed floor (1-10 s in), that instant correctly fires nothing, and its rows then sit in the 45-minute lookback as multi-second priors: every later instant of the same episode fails the sub-second ceiling against them, and the episode never fires at all. It fails quiet - a missed page, never a false one - and the exposure is sampling phase: a 9-second landing band in a 60-second collection cycle, roughly 9-in-60 on the measured 90-second shape. This is accepted, deliberately: both repairs are gate changes (onset-exclusion, or pricing on the qualifying subset past the floor), each with edges of its own, and neither ships ahead of a measured miss to calibrate against - the recurrence watch, which folds episodes onto a statement-scoped incident id, is the instrument that would produce one. The detector's remarks now carry the full paragraph, and the poisoned sequence is pinned as accepted behavior: instant one at ~5 s (no finding, the floor), instant two past the floor (still no finding, the ceiling reads instant one's rows), and the counterfactual - the identical second instant without the onset sample - firing. The every-session floor's fresh-arrival dilution gets the same treatment beside its comment: the qualifying-subset alternative is named, the reason it waits (the fleet calibration was measured under this predicate) is stated, and the same reopen shape covers it. The fix: both call sites advanced the per-instant dedup stamp BEFORE InsertFindingsAsync, so a transient persist fault marked a firing instant "done" and lost its finding for good if the episode cleared before the next collector write. The stamp now advances only once the instant's outcome is durable - evaluated clean, fully muted, or persisted - and a fault leaves it behind so the next sweep re-derives the same instant, where occurrence folding absorbs the re-persist and the incident-keyed cooldown absorbs the re-notify. The ordering is pinned from source in both files. The polish, each item its own small honesty: * The census's permitted-tables set pre-cleared waiting_tasks, which made "adding it later is a deliberate census change" false - the read could have arrived without touching a census line. The permission is removed: adding the read now means widening the set in the same change, so the dependency lands as a recorded decision. * The worker's doc quoted the README's pre-#3467 contract sentence ("scheduled-analysis finding notifications"); the quote now matches the widened sentence this lane shipped. * Both finding stores justified the epoch COALESCE with a deployment that cannot exist (analysis disabled writes no pileup rows either - the sweep rides the same gate); the comments now name the real anchors: a young install whose pileup fired before the data-span gate let a first scheduled batch exist, and analysis toggled off after pileup rows landed. * PgPileupSnapshotReader's catch now uses the directory's own AnalysisShutdown.IsExpectedAbandon filter (#2299) instead of a bare OperationCanceledException carve-out, so shutdown residue propagates quietly and a cancellation that nobody requested keeps its log line. Out-of-band verification (the test projects target net10.0-windows): the detector harness replays 102 checks and the census harness 93, both clean, including the onset pin and the stamp-ordering pin.
| var baselineFloor = latest.AddMinutes(-BaselineLookbackMinutes); | ||
|
|
||
| /* The candidate groups: the newest instant's rows, by statement identity. */ | ||
| foreach (var group in usable.Where(r => r.CollectionTime == latest).GroupBy(StatementIdentity)) |
There was a problem hiding this comment.
Correctness: grouping by StatementIdentity (query_hash) alone, without DatabaseName, can conflate unrelated slow queries in different databases into one bogus "shared plan" pileup.
query_hash is computed from the normalized query text and is not scoped to a database — two different user databases running textually-identical parameterized SQL (a very common shape in exactly the multi-tenant-per-database SaaS deployments this finding targets) produce the same hash. But the entire causal story this detector tells ("a shared parameterized plan flipped, every caller of that one plan piles up together") is only true within one database, because a SQL Server plan-cache entry is keyed per-database (dbid is part of the compiled-plan cache key) — a plan going bad in database A cannot be "the same plan" as one in database B, even for byte-identical text.
Concretely, three sessions in three different databases, each independently slow (unrelated causes) for >10s on a query shape that happens to share a query_hash, with one on an IO wait, would be merged here into a single SAME_STATEMENT_PILEUP finding — miscounted as "3 concurrent sessions piled up on one statement in {one arbitrarily-picked db}" and paged at a severity computed from the combined (unrelated) session count.
This also affects:
- The baseline check at line ~317-320 (
StatementIdentity(r) == identity, no database filter): a genuinely-new slow query in DB B could pass the "normally sub-second" baseline gate using DB A's unrelated fast history for the same hash — or a real same-DB pileup could be suppressed because an unrelated slow occurrence of the same hash in another DB pollutes the baseline'ssub-second ceilingcheck. databaseNameat line 390 (pack.Select(r => r.DatabaseName).FirstOrDefault(...)): picks one database out of what could be a multi-database pack, non-deterministically (row order isn't guaranteed stable for ties at the samecollection_time). SinceComputeIncidentIdfolds on(serverName, RootFactKey, databaseName, identity), this can also make the same recurring incident fold onto different incident IDs across evaluations if the "first" database picked changes — undermining the occurrence-folding guarantee this PR is built around.
Worth scoping the grouping (and the baseline lookup) to (DatabaseName, StatementIdentity) rather than StatementIdentity alone, matching how a plan-cache entry is actually scoped.
There was a problem hiding this comment.
Real finding, all three parts — and it's now #3474 rather than a quiet default, because the fix has a fork in it: the measured incident's convoy may itself have spanned tenant databases ("ten tenants inherited the flip together"), in which case scoping the group key per-database would have missed the exact outage this detector was built for. The issue lays out both options plus the deciding evidence (the incident's own snapshots record DatabaseName per session — that lookup happens before any code moves). The FirstOrDefault fold-attribution nondeterminism is real under every option and flagged to ship first.
|
Reviewed the same-statement-pileup finding (#3467): detector, both SKUs' readers, worker/service wiring, finding-store overlay SQL, advice, and the extensive pinning tests (census, master-switch, calibration). Overall: well-scoped and heavily self-verified — the source-census tests keep the trigger path off Query Store as claimed, the master-switch surface tests keep delivery gated identically to the scheduled path, and the calibration/severity math checks out against the fleet numbers cited in the docstrings. Lite and Darling stay in lockstep (same detector, same D0 production/delivery split, same dedup-stamp-after-persist discipline, same latest-batch carve-out SQL shape, verified by Left one inline finding: No SQL injection, secrets, or file/network/process concerns — all reads are parameterized ($1/$2 or DuckDBParameter), and no T-SQL was touched by this PR (the shared collector query itself is pre-existing). |
…ps (#3467) CI's only failure on 601ab8a (1 of 9,449) was FindingStoreSql_CarriesTheDashboardShape: its GetLatestFindingsSql pin still asserted the pre-#3467 single-line anchor, "SELECT MAX(analysis_time) FROM analysis_findings WHERE server_id = $1" — a string that no longer exists, because the carve-out deliberately rewrote it: the MAX now excludes SAME_STATEMENT_PILEUP rows from DEFINING the batch (a per-sweep pileup row claiming the MAX would replace the last complete analysis pass and read the server healthier mid-incident), and the subquery spans two lines. The re-pin asserts MORE than the old pin, not less: both fragments as they actually appear in the const — the MAX anchor line and the WHERE line carrying the exclusion predicate. The full carve-out contract (overlay arm, epoch anchor, Lite twin parity) already belongs to SameStatementPileupSourceCensusTests, which normalizes source whitespace and passed on the same run; this pin's job stays what it was — the dashboard shape of the Darling store's own consts — now stated against the SQL that ships. Why the lane's out-of-band verification missed it: the detector and census harnesses replay the NEW tests (102 + 93 checks, clean); this pin lives in the pre-existing suite that only executes on Windows CI. A Contains pin between two compile-time strings is fully verifiable by grep, which is how this fix was checked: each fragment appears twice in GetLatestFindingsSql (anchor arm and COALESCE arm), and the stale single-line form appears nowhere outside DELETE cleanup statements.
Review: SAME_STATEMENT_PILEUP finding (#3467)Went through the full diff — detector ( Correctness
Lite/Darling parity
Security: no injection surface — both readers use parameterized queries throughout, no string-built SQL. One minor nit (style, not a bug): Otherwise didn't find a correctness, parity, or security issue to flag — the census tests and the fleet-shape counterexamples in |
…nent ordinals (#3466) CI's only failure on 2b45c77 (1 of 9,486, identical in both jobs) was the V122 rung's round-trip pin: TheViewerRow_RoundTripsThroughTheBind_AtTheAppendedOrdinals asserted its pair at Parameters[^2]/[^1] — "the END" — which was true only while V122 was the newest rung on config_alert_settings. This lane's V124 append (fleet_sweep_enabled $65, fleet_sweep_interval_minutes $66) followed the append rule exactly, and the first appended knob is a bool: the V122 pin's IsType<NpgsqlParameter<int>> at [^2] met it and failed — the correct failure, loudly refusing to read a bool seat as an int seat. The append discipline itself held on all four parallel sequences (column list, upsert placeholders, bind order, reader ordinals — verified from the diff: $65/$66 appended last, reader GetBoolean(64)/GetInt32(65) appended last). No bind bug; a positional pin aged out. The fix seats V122's pin at its permanent absolute ordinals: indexes 62/63, matching the upsert's $63/$64 and the reader's GetInt32(62)/(63) — verified mechanically from the bind source (62 Adds precede the pair). Absolute is the point: appended columns never move again, which is what "every earlier ordinal keeps its column" promises, and which a distance-from-the-end pin structurally cannot assert past the next rung. The END pin now lives where it belongs — in this lane's own FleetSweepCadenceKnobRungTests, until V125 repeats this handover, for which the V122 test's comment is now the recipe. Same failure class, third instance today (#3457's cross-merge arity, #3469's stale anchor pin, now this): a pre-existing pin meeting newly appended state on the Windows-CI-only suite that out-of-band harnesses do not replay. The sweep census tests this lane ships are the pattern that ends the class — census from source, not pins at a distance.
…d-have-paged ledger (#3466, lane 2 of 4) (#3471) * Sweep the fleet with memory: the engine that writes lane 1's tables, on the operator's cadence (#3466) Lane 2 of the approved four-lane build order, stacked on the V123 state store. The sweep engine is the scheduled, stateful, Darling-only evaluation the spec decomposes: collect through the daily-summary scoring path, diff against the previous sweep's persisted rows, advance watch items through lane 1's state machine, prove the instruments were alive before believing the quiet, and write the would-have-paged ledger when the fleet is muted. No feed, no delivery here; those are lanes 3 and 4 against the rows this engine now writes. * The cadence knob lives where the analysis cadence lives The spec says user-configured cadence, hourly by default, and the product already has exactly one "run a whole-fleet evaluation every N minutes" knob: analysis_enabled / analysis_interval_minutes on the config_alert_settings control-plane row, wholesale-swapped into the running config, exposed through get/update_alert_settings, editable in the Viewer. V124 adds fleet_sweep_enabled / fleet_sweep_interval_minutes on that exact path - with the one deliberate difference stated everywhere it matters: the sweep's switch is its OWN, never the alert master switch's, because sweeps under master-off are the muted-mode contract's whole point. Bounds are FleetSweepCadence's named constants ([15, 1440], default 60) at every writer - the MCP write bound, the Viewer's save gate, the worker's read-side clamp - so no accepted value is ever silently rewritten, the parity every knob rung since V119 holds. The floor is where "two consecutive sweeps" still confirms a sustained condition rather than one burst read twice; the ceiling is one sweep a day, the grain lane 4's channel rollup is ruled to. Darling-only structurally (Lite has no fleet to sweep), recorded as a group-level omission in McpAlertSettingsKeyTests with the paying test the self_alerts and health_bands omissions established. * Verdicts reuse the daily-summary scorer - the same SQL, the same bands Per-server verdicts come from DailySummarySql.RangeSql bound to the sweep's own span (GetWindowSignalsAsync - the one aggregate the calendar, get_daily_summary and now the sweep all read) folded by the shared DailyHealthBandCalculator, with BuildReasons as every non-Healthy card's stated reason. Nothing is forked: when a banding threshold moves, the sweep and the day surfaces move together, and the engine tests assert the verdicts against the calculator's own answers rather than against literals. The span anchors on the previous sweep's instant so no gap opens between documents, capped at the cadence ceiling so a sweep waking from downtime never claims a window its retention tiers might not hold; sub-day spans always sit inside the 4-day raw window, so the raw tier is correct by construction. * The diff is against persisted rows, and a missing anchor says so Sweep N joins its verdicts to sweep N-1's by server through the store's new engine-grade verdict read (throwing, unlike the presentation read: a fault swallowed into empty would make every server read as NEW - a diff that restates absolutes because the store failed). Band transitions, new servers and departed servers land in the document; a first sweep declares no_previous_sweep rather than implying a quiet history. This is the compare_analysis leg conceptually - two windows, per-key deltas - applied to sweep state. * Instrument liveness is judged before anything else is believed The sweep proves its own instruments and says which: per-server signal reads that faulted render as unreadable-not-quiet (No Data card, red liveness, no fabricated staleness), fleet-wide quiet outside the settle window is dead collection instruments, and the alert path's pass counter is compared against the PREVIOUS run's recorded total - the frozen-counter-beside-a-delivering-path reading that caught #3464, now running on the sweep's own persisted memory. Every unjudgeable comparison carries its stated cause instead of a verdict: master-off freezes the counter by design (the #3464 gate stops the pass before RecordPass), and a restart resets the baseline, detected from the service's own counting_since. * Watch items: narrow qualification, honest omission Three item families in v1, the spec's own contracts: a server whose verdict banded Critical, a server that reported nothing outside the settle window, and the fleet-scope instruments-dead condition on the store's sentinel id. Every active item is evaluated every sweep - a miss is an evaluation - except where the sweep cannot honestly judge: staleness items inside the settle window and any item on a server whose read faulted are OMITTED from the upsert whole, so their counters stand rather than advancing on evidence that does not exist. A miss hands the store null evidence so the COALESCE keeps what opened the item. The episode stamps always describe the current episode; history lives in the documents that reported it. * The would-have-paged ledger is derived honestly Under master-off, each Critical verdict decomposes into the summary scoring's own critical triggers - deadlocks, collection errors, severe memory pressure, sustained CPU, heavy blocking - one row per (server, family) with the measured figure and the threshold it crossed. This is deliberately NOT an alert-engine replay: the families are the sweep's vocabulary, and every evidence payload states the derivation, so nobody reads the ledger as the engine's counterfactual. What it claims is what it can prove, which on the evidence day was the only surface that carried an 18-minute CPU pin through an operator mute. * Restart awareness is a gate Inside PostRestartSettleWindow (10 minutes - double the worst of the three ~4-5 minute measurements) a stale fleet is classified startup transient: no-data cards say so, staleness opens nothing, liveness is not failed on it, and real data still bands and still advances. The detector is the service's own start instant, the same counting_since every health surface reports. * The worker hosts it like the backlog sweep, and a failure costs a slot Fire-and-track off the fleet loop with the IsCompleted stacking guard and the stamp advanced before launch; RunAsync never throws (quiet on cancellation, one error line otherwise), so a sweep failure costs exactly one sweep slot and the collection loop nothing. The store's one-transaction writer means a failed sweep leaves the PREVIOUS sweep as the newest complete document. The engine makes no delivery call anywhere - AlertMasterSwitchSurfaceTests' source census has nothing of this path's to count, and the engine tests pin both the absence and the master-off compose. * Retention lands beside the writer, as lane 1's rung doc promised Runs and their two children prune at the base data horizon (children first, through their run's swept_at, with an orphan arm covering a purge that failed between statements - the price of no FKs, paid as one cheap NOT EXISTS instead of an ordering constraint); watch items prune on last_seen_at, the backlog's rule, so a carried episode never loses its row. * The rung, the probe, and the handoff V124 follows V122's knob-rung shape exactly: ADD COLUMN IF NOT EXISTS with the constants as defaults, no GRANT, no second reload beacon (V17's trigger already bumps config_version). The probe gains the sentinel at all three sites (ordinal 99); FleetSweepStateRungTests hands the top-rung claims to FleetSweepCadenceKnobRungTests and keeps everything true of V123 wherever it sits - the V120->V121->V122->V123 note, carried once more. Tests: FleetSweepCadenceKnobRungTests (top-rung claims, every settings-row surface names both columns, write-bound/clamp/gate parity on the named constants, defaults named not restated, the worker gates on the sweep's own switch with the master switch as data, the viewer bind round-trip at the appended ordinals); FleetSweepEngineTests (pure compose with a fixed clock: scorer reuse asserted against the calculator itself, the two-sweep diff, the full watch lifecycle through the real state machine, miss-on-unsighted, fleet-quiet and read-fault liveness, the frozen-counter judgment in all four judgeability shapes, master-off produces-and-ledgers vs alerts-on empty ledger, the settle window's suppressions and its ten-minute figure, span arithmetic, and the no-delivery-seam source pin); McpAlertSettingsKeyTests gains the fleet_sweep group omission with its paying test; the MCP round-trip census extends through the sample row's two new named arguments. Development on macOS, where the test assemblies build but cannot run, so verification was: the full solution builds clean (0 errors; all warnings pre-existing in untouched files), and an out-of-band harness referencing the built Service assembly ran 78 checks - every pure compose behaviour above, plus the LIVE half against a real TimescaleDB pg17: full-ladder migration to V124 with the knob columns' defaults on the wire, rung replay idempotence, the probe's on-the-wire column count against the map arity (100 == 100, mapping to 124 and to 123 without the new sentinel), two real end-to-end FleetSweepEngine.RunAsync sweeps against the store (first-sweep settle-window document, then a master-off sweep anchored on it with the mute header on the run row), and the real DarlingRetention.PurgeAsync pruning a planted 40-day-old sweep, its children, a planted orphan child and a long-unseen watch item while the fresh sweeps survive. All pass. No changelog entry in this lane, deliberately, continuing lane 1's stated disposition: nothing renders until the web feed (lane 3), and the entry belongs to the lane that makes the feature visible rather than to the engine that fills its tables. Part of #3466 - lanes 3 (web feed at configurable spans) and 4 (daily channel rollup ceiling + get_sweep_reports) follow against these rows. * Last-seen means sighting: carry the watch-item stamps through a miss, and name the second cadence (#3466) Review caught the sweep engine stamping last_seen_sweep_id/last_seen_at on every evaluation, which quietly violated lane 1's last-seen-means-sighting contract and made one class of row immortal: a pending item nothing ever resights stays pending (the state machine resets its entry count and holds), the active read re-evaluates it every sweep, and every evaluation re-stamped last_seen_at fresh — so the retention arm keyed on that column could never mature for exactly the rows nothing will ever close, while the readers' ORDER BY last_seen_at DESC ranked "most recently evaluated", which every active row is, every sweep. BuildItemImage now carries the row's standing sighting pair through a miss unchanged and stamps only on a hit. Safe for live episodes: an open item takes at most one standing miss before the exit bar closes it, so a carried row is never more than one sweep from its last sighting. The engine tests gain the discriminator (a miss does not move the pair, a hit does), and the harness now plants a 40-day-old pending item that a REAL sweep evaluates and misses — the sighting pair stands at its planted values on the wire, and the real purge prunes the row the pre-fix stamping kept immortal (85 checks, up from 78, all passing against TimescaleDB pg17). Also from review: get_alert_settings' description walked self_alerts and health_bands but never mentioned the fleet_sweep group it reports and called the analysis cadence the only cadence — the READ tool now carries the update tool's fleet_sweep paragraph in shorter form (the two knobs, default hourly, and that the master switch deliberately governs delivery, never sweep production). Verified before touching it: no census pins the get-side text, and the cross-SKU byte-identity rule does not apply — Lite's copy is its own deliberately different words, and its fleet_sweep payload omission is already argued and paid for by McpAlertSettingsKeyTests. And the AdvanceWatchItems omission doc now states the settle-window consequence as decided rather than overlooked: a non-staleness pending item missed inside the window has its entry count reset, bounded to one sweep because the window sits under the floor cadence, and a single miss cannot close an open item under the exit bar. * The end belongs to the newest rung: seat V122's bind pin at its permanent ordinals (#3466) CI's only failure on 2b45c77 (1 of 9,486, identical in both jobs) was the V122 rung's round-trip pin: TheViewerRow_RoundTripsThroughTheBind_AtTheAppendedOrdinals asserted its pair at Parameters[^2]/[^1] — "the END" — which was true only while V122 was the newest rung on config_alert_settings. This lane's V124 append (fleet_sweep_enabled $65, fleet_sweep_interval_minutes $66) followed the append rule exactly, and the first appended knob is a bool: the V122 pin's IsType<NpgsqlParameter<int>> at [^2] met it and failed — the correct failure, loudly refusing to read a bool seat as an int seat. The append discipline itself held on all four parallel sequences (column list, upsert placeholders, bind order, reader ordinals — verified from the diff: $65/$66 appended last, reader GetBoolean(64)/GetInt32(65) appended last). No bind bug; a positional pin aged out. The fix seats V122's pin at its permanent absolute ordinals: indexes 62/63, matching the upsert's $63/$64 and the reader's GetInt32(62)/(63) — verified mechanically from the bind source (62 Adds precede the pair). Absolute is the point: appended columns never move again, which is what "every earlier ordinal keeps its column" promises, and which a distance-from-the-end pin structurally cannot assert past the next rung. The END pin now lives where it belongs — in this lane's own FleetSweepCadenceKnobRungTests, until V125 repeats this handover, for which the V122 test's comment is now the recipe. Same failure class, third instance today (#3457's cross-merge arity, #3469's stale anchor pin, now this): a pre-existing pin meeting newly appended state on the Windows-CI-only suite that out-of-band harnesses do not replay. The sweep census tests this lane ships are the pattern that ends the class — census from source, not pins at a distance.
… baseline, and the tempdb rule the evidence demanded (#3475) The review on #3469 asked the question the merge had defaulted: the same-statement pileup detector grouped concurrent sessions by query_hash alone, and query_hash is not database-scoped - on a fleet of identical tenant schemas, textually-identical SQL in different databases shares a hash by construction - while a compiled-plan cache entry IS keyed per database, so the causal story the finding tells (one shared parameterized plan flipped, every caller of that one plan convoyed) is only strictly true within one. #3474 pulled the deciding evidence from the store's own snapshots before any code moved, and the evidence chose option A: - The measured incident's three convoy sessions all ran in ONE tenant database (the "ten tenants" are application-level tenants inside it, sharing exactly one plan), so per-database scoping preserves the motivating incident intact - its group, its 2.08 severity, and its notify margin are unchanged. - The conflation the review predicted was measured, not hypothetical: within the same hour, the same statement text ran 9.7 s / 7.4M reads in a DIFFERENT tenant database on the same server - one snapshot's timing away from joining the incident's pack under the server-wide key. - The DMV sometimes attributes a copy's database context to tempdb (a 16.5 s copy of the incident statement, the day before - the statement joins a #temp table and the request's context followed it), a shape neither option in the issue had named. The change, in the detector only (both SKUs' readers already project database_name and are untouched): - The group key is (DatabaseScope, StatementIdentity). Tempdb-attributed rows form their own (tempdb, statement) group - honest about what the DMV said, with reattribution deliberately not attempted: the snapshot carries no reliable session-to-database mapping at read time, and a guessed attribution would poison exactly the per-database baseline this change exists to keep clean. - The "normally sub-second" baseline reads only the same database's history, closing both pollution directions: another database's fast history cannot vouch for a statement that is new-and-slow where it actually runs, and another database's unrelated slow occurrence cannot suppress a real pileup - the direction that fails quiet. - The evidence pack's database is the group key's database, structurally, retiring the first-non-empty pick over the pack that was nondeterministic for multi-database packs - ComputeIncidentId folds on databaseName, so the pick's wobble destabilized the occurrence folding the detector was built around and the stable token #2138 wants. - The calibration carries over without a re-pull, by argument written into the derivation's doc comment: per-database scoping PARTITIONS the server-wide groups and can never grow one, so every gate is strictly harder to pass and the four-instants-in-four-server-days derivation stays conservative under the new predicate. Tests pin the kill-shot (three same-hash sessions in three databases are three groups, not a pileup), both baseline-isolation directions, the tempdb rules (own group, own history, no dilution of the tenant pack's arithmetic), fold determinism under row-order permutation, and the null-database fold to one scope. Closes #3474.
Page on a plan-flip pileup from the two collectors that were already current while it ran
Closes #3467
A parameterized statement joining a
#temptable flipped to a plan doing ~35x the logical reads. Every caller shares the parameterized plan, so ten tenants inherited the flip at once: concurrent executions convoyed onPAGEIOLATCH_SH, physical reads saturated the volume, and sixteen executions hit the client's 30-second command timeout together. The whole episode ran about ninety seconds and self-cleared when the plan churned out.The collection side was never behind. The 20:10:31Z active-query snapshot held three concurrent copies of the one statement — 29,124 ms / 28,343 ms / 27,703 ms elapsed, 1,351 ms CPU and 287,049 logical reads on the leader, two suspended on page IO — sitting next to that same statement's 20:07:01Z observation at 54 ms. Evidence sufficient to describe the incident existed in the store while the incident was still running, and nothing could page on it:
ANOMALY_QUERY_DURATION0.72,ANOMALY_BATCH_REQUESTS1.0,ANOMALY_CPU_SPIKE0.56, againstnotify_severity1.5;PLAN_REGRESSION, 20:38:49Z) keys offquery_storedata that had not landed yet;long_running_querytriggers at 30 minutes and the victims died at 30 seconds,poison_waitdoes not class PAGEIOLATCH,blocking/deadlocksdo not apply, CPU stayed near 50%.So the tool watched an outage it could describe and could not report. This adds the finding the issue proposed, computed from the two collectors that were already current.
The finding
SAME_STATEMENT_PILEUP: N concurrent sessions on one statement at one snapshot instant, every copy past a floor of elapsed time, at least one suspended on an IO-class wait, where that same statement's own snapshots across the prior 45 minutes were sub-second. The conjunction is what makes it specific — on the fleet data it fired exactly once in four server-days, and that firing was this incident.Severity prices magnitude rather than presence (#3462):
sessions × min-elapsed-seconds ÷ 40, capped at the band ceiling.Placement, and the achieved latency
The finding is evaluated on the collection cadence, not the analysis interval — the Darling worker's per-server sweep on the 30-second alert-sweep stamp, Lite's one-minute collection loop, in both cases immediately behind the collectors that write its input.
Achieved latency for the measured incident: inside one collection cycle of the snapshot that proved it. The 20:10:31Z snapshot is read and evaluated on the next sweep stamp (≤30 s on Darling, ≤60 s on Lite), so the finding exists at severity 2.0 against the 1.5 gate — and notifies — by roughly 20:11Z, while the episode that self-cleared around 20:11 was still alive. Against the measured baseline: the scheduled anomalies landed 20:31:48Z (21 minutes, all capped ≤1.0), and the notify-eligible
PLAN_REGRESSIONcould not exist before ~20:38Z (~28 minutes). Detection moves from ~28 minutes after the damage to inside the damage.It also deliberately does not inherit the scheduled pass's 24-hour data-span gate. That gate exists because fraction-of-period fact scoring distorts on short windows; this finding's evidence is one snapshot plus one statement's own recent history, so it works on a young install.
Alternatives rejected, and why the issue was right to reject them
ANOMALY_BATCH_REQUESTSwas already at 1.0, so a mass-timeout window would need roughly a 50% scale-up to clear 1.5 — recalibrating a whole-server anomaly family's ceiling on the strength of one incident class, which is what Anomaly findings fire at 0σ against healthy baselines - memory pressure reports "spiked to 100% - 0σ above its 100% baseline" #1996 documents the cost of. The cadence half is the fatal one: a sub-pass fast enough to matter for a 90-second episode is per-collection evaluation, so "anomalies plus a faster sub-pass" converges on this change while carrying the anomaly family's whole-server framing — and a whole-server anomaly cannot name the statement, so it cannot carry the statement-scoped fingerprint the recurrence needs or the force-plan drill-down [ON HOLD] [FEATURE] Auto force plan bot — detect bad Query Store plans (CPU-weighted), force with self-review, evict-first option #2138 wants. The cost argument also does not hold: a per-sweep evaluation here is one index-pruned window read per server (idx_query_snapshots_timeon(server_id, collection_time)prunes the range; the projection fetches the ~50–150 rows) and a pure in-memory predicate, on the same cadence the long-running-query alert already reads this table.notify_severity. Rejected in the issue and not revisited: it trades this gap for noise fleet-wide, and Anomaly findings fire at 0σ against healthy baselines - memory pressure reports "spiked to 100% - 0σ above its 100% baseline" #1996 records what under-qualified anomaly findings do to trust.Query Store independence is a requirement, not a convenience
On at least one production store class the Query Store readers are deliberately disabled (#2296) — which is the deployment this incident happened on, where the plan-count evidence had to come from outside the product. A pileup detector that reached for
query_storeidentity would compile, test green against a QS-enabled fixture, and ship blind on exactly the servers that needed it.So the trigger, the baseline, the statement identity and the incident fingerprint are computed from the active-query snapshot collector alone, and the statement key is the DMV's own
query_hashoffsys.dm_exec_requestsrather than anything Query Store hands out.SameStatementPileupSourceCensusTestspins this from source, for the detector and both editions' readers (enumerated rather than intersected — a census that checks one SKU certifies only that SKU):query_storeidentifier in code or in SQL literals on the trigger path (comments are exempt on purpose — the detector's own remarks have to be able to explain why the dependency is forbidden; this is the inverse of the delivery census's strip-the-strings rule, because here a table named inside a literal is the dependency under test);FROM/JOINtarget in the readers' literals is one of the permitted collector tables, and the detector's literals name none at all (it is a pure function of rows);(server, database, statement identity)— no store read, nothing a disabled Query Store could withhold.PLAN_REGRESSIONand the #2138 force-plan bot stay exactly what they were: follow-on enrichment where QS happens to be readable, never the trigger.Thresholds, derived from fleet distributions
Pulled live from the Darling store's active-query snapshots: four production servers × 24 hours, ~4,160 snapshot instants. Same-statement concurrent groups: 484 at n ≥ 2, 103 at n ≥ 3 — but only four n ≥ 3 groups had every session past ten seconds, and three of those four are routine batch shapes. Each constant is named per #3060 with its derivation stated on it, in the unit the gate compares (#3463).
MinConcurrentSessionsMinElapsedMsMinIoWaitSessionswait_time_ms5 — so sessions bounce between suspended and runnable and demanding two simultaneously suspended makes the gate a coin flip on sampling phase. The incident showed two; requiring one keeps the evidence without betting the page on sampling luck. Specificity comes from the conjunction.IoClassWaitPrefixesBaselineLookbackMinutesSubSecondBaselineCeilingMsMinBaselineObservationsStaleSnapshotCutoffMinutesThe measured incident clears every gate; ordinary workload does not. The three non-incident fleet shapes that reach n ≥ 3 past ten seconds each fail a different gate, which is the argument that no single threshold is carrying the finding:
All four are tests.
Severity calibration, in the unit the gate compares
severity = min(2.0, sessions × min-elapsed-seconds ÷ 40). Stated to the figures the arithmetic actually produces, because a calibration claim stated in a rounded unit is not the claim the gate evaluates (#3463):>=)The measured incident clears the gate even with two of its three sessions discounted to the 10-second floor. The notify boundary is three concurrent copies each past two-thirds of the timeout that killed the measured victims. And the issue's "a two-session brief pileup must not notify" holds one step stronger than asked: the session floor is a predicate, not a severity input, so two sessions produce no finding at 12 s, 29 s or 600 s.
Recurrence folds instead of surprising
The measured statement re-drew its regressed plan later the same evening after fully self-clearing, so the finding gets the incident-fingerprint treatment every alert family already has: the
IncidentIdrecipe extended with the statement identity. Episode two lands on episode one's incident trail as an occurrence (occurrences/first_seen/last_seenvia the existing(story_path_hash, incident_id)folding) rather than reading as a fresh surprise, and #2138 gets a trigger token stable across episodes, plan churn, restarts and QS resets. The story path carries the statement identity too, so muting or cooling down one statement's pileup cannot answer another statement's question (#3459's rule).Stated plainly because it is a real behavior and not an accident: a recurrence inside the 6-hour notify cooldown folds silently rather than re-paging, because a band-saturating severity cannot satisfy the #2054 worsening step. The occurrence trail and the viewer show it; the channel does not repeat it. That is the house's standing-condition policy, and it is what "folds onto episode one rather than reading as fresh" means.
Both editions
Lite's active-query snapshots feed the same seam — one shared collector definition, one table shape — so the predicate, thresholds, severity arithmetic, fingerprint and noise filters live once in
PerformanceMonitor.Analysisas a pure function of snapshot rows. Each edition contributes only its window reader (Npgsql overquery_snapshots, DuckDB overv_query_snapshots) and its sweep call site. Both ship it.It adds no delivery semantics — and is censused as the call site it is
The finding rides the existing analysis-finding machinery end to end: the mute registry, the two-phase materialize → enrich → persist write, occurrence folding,
notify_severity, the incident-keyed cooldown and its worsening rule. It is, however, a new delivery call site on both SKUs — a different thing, and accounted for as one:ShouldNotifyAnalysisFindings(master AND family, alerts_enabled is not the master switch its contract claims: two delivery paths paged through a fleet-wide mute, 38 and 60 minutes in #3464's predicate) — production is unconditional, delivery alone is gated (the D0 split);AlertMasterSwitchSurfaceTestscensus entry (Inline), and the pins hold both in both directions, with the predicate's call count raised from two to three;alerts_enableda promise widened from "scheduled-analysis finding notifications" to "analysis finding notifications". The narrower wording described a promise smaller than the one the gates keep, and an operator could reasonably have read a non-scheduled finding as escaping the switch — the exact class of confusion alerts_enabled is not the master switch its contract claims: two delivery paths paged through a fleet-wide mute, 38 and 60 minutes in #3464 was about.One read-side change, in both stores, for the same reason
The latest-batch read now excludes pileup rows from defining
MAX(analysis_time)and overlays the ones newer than it. #2448's contract is that the newest batch is one indivisible statement about a server — the complete output of one pass — and a per-sweep row claiming the MAX would replace the last complete analysis with a single-row one, making a server read healthier mid-incident. That is #2448's own misreading arriving through the write cadence instead of through a torn batch. The overlay is self-limiting (visible from firing until the next scheduled pass completes, then living on in the windowed reads and its incident trail), and the epoch fallback keeps a pileup visible where no scheduled batch exists to anchor on — never a disabled-analysis deployment (the pileup sweep rides the same analysis-enabled gate, so analysis off means no pileup rows either), but a young install whose pileup fired before the scheduled pass's 24-hour data-span gate let a first batch exist, or analysis toggled off after pileup rows landed and before any scheduled batch did. Both stores carry the identical shape and are pinned against each other.Migration rung
None claimed, and none needed. The finding persists through
analysis_findings' existing columns, readsquery_snapshots' existing columns, and keeps its one piece of cross-sweep state — the last-evaluated snapshot instant — in memory, where a restart costs at most one repeated evaluation whose finding folds onto the same incident id and whose notification the cooldown (seeded from the alert log on first lookup) suppresses. The stamp advances only once the instant's outcome is durable — evaluated clean, fully muted, or persisted — so a transient store fault re-derives the same instant on the next sweep instead of losing a firing instant's finding, and the same folding-and-cooldown absorption covers the replay; the ordering is pinned from source on both call sites. The rung available to this lane is left unspent.Tests
The measured incident reconstructed verbatim from the store's own rows, and the shapes that must stay quiet:
long_running_query's job;severity × divisor == session-seconds) so the divisor cannot drift from its doc comment;Verification, and its limits
Every touched assembly builds, plus the whole solution:
PerformanceMonitor.Analysis,Darling.Analysis,Darling.Service,Lite,Darling.Tests,Lite.Tests.The test projects target
net10.0-windowsand cannot execute on this machine, so the assertions were additionally replayed out of band against the real code:PerformanceMonitor.Analysisisnet10.0, so a throwawaynet10.0console harness ran the detector itself over the same fixtures (102 checks, all passing), and a second harness replayed the census logic — the #3465 file-set derivation, per-member site counts,Inlinegate ordering, the notify pins, the #3467 QS-independence and store-SQL pins, and the dedup-stamp ordering pin — against the worktree with the test project's ownCSharpSourceWalkercopied in verbatim (93 checks, all passing), with its census table verified content-identical to the committed one. What that does not cover: the xUnit runner's own discovery/attribute handling, and anything requiring a live store — the two readers' SQL is verified by column-name audit against the collector definition and both generated schemas, and by shape-match against the existing long-running-query read of the same table, not by execution.What this cannot see
An episode whose own onset poisons its baseline. If the collector's first sample of an episode lands while the sessions are between the sub-second ceiling and the elapsed floor — 1–10 s in — that instant correctly does not fire (the floor), and its rows then sit in the 45-minute lookback as multi-second priors: every later instant of the same episode fails the sub-second baseline gate against them, and the episode never fires at all. It fails quiet — a missed page, never a false one — and the exposure is sampling phase: a 9-second landing band in a 60-second collection cycle, roughly 9-in-60 on the measured 90-second shape (under ~1 s the onset rows are themselves sub-second and reinforce the baseline; past 10 s the first sample fires directly). Accepted deliberately: both repairs are gate changes — onset-exclusion, or pricing on the qualifying subset past the floor — each with edges of its own, and neither ships ahead of a measured miss to calibrate against. The recurrence watch is the instrument that would produce one: episodes fold onto a statement-scoped incident id, so a statement whose trail holds fired episodes alongside snapshot-visible pileups that produced no finding is the reopen shape. The detector's remarks carry the full statement, and the poisoned sequence is pinned as accepted behavior, counterfactual included.
The same reopen shape covers the gate's fresh-arrival dilution: a caller that joins the convoy just before the sample resets the pack's minimum below the floor and suppresses detection while the pileup is growing. The qualifying-subset alternative fixes both edges at once; it waits on re-deriving the fleet calibration, which was measured under the every-session predicate.
Deferred, deliberately
WAIT_ON_SYNC_STATISTICS_REFRESH, plausibly the recompile trigger itself). The trigger does not need it —wait_typeandstatusride on the snapshot row, which is where the IO-class evidence comes from — so adding a second per-sweep read for drill-down enrichment alone was not worth doubling the reader surface on a rarely-fired finding. It is deliberately not in the census's permitted-tables set: pre-cleared, the read could arrive later without touching a census line, which is the opposite of what the census is for. Adding it means widening the permitted set in the same change — the census fails, names the file, and the dependency lands as a recorded decision.notify_severityis already the operator's knob over whether a given pileup pages. A control-plane knob rung is available if the fleet shows a deployment that needs a different shape.