Skip to content

Page on a same-statement pileup while it is happening, from the collectors that already saw it (#3467) - #3469

Merged
erikdarlingdata merged 3 commits into
devfrom
fix/3467-same-statement-pileup
Sep 16, 2026
Merged

erikdarlingdata merged 3 commits into
devfrom
fix/3467-same-statement-pileup

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Page on a plan-flip pileup from the two collectors that were already current while it ran

Closes #3467

A parameterized statement joining a #temp table flipped to a plan doing ~35x the logical reads. Every caller shares the parameterized plan, so ten tenants inherited the flip at once: concurrent executions convoyed on PAGEIOLATCH_SH, physical reads saturated the volume, and sixteen executions hit the client's 30-second command timeout together. The whole episode ran about ninety seconds and self-cleared when the plan churned out.

The collection side was never behind. The 20:10:31Z active-query snapshot held three concurrent copies of the one statement — 29,124 ms / 28,343 ms / 27,703 ms elapsed, 1,351 ms CPU and 287,049 logical reads on the leader, two suspended on page IO — sitting next to that same statement's 20:07:01Z observation at 54 ms. Evidence sufficient to describe the incident existed in the store while the incident was still running, and nothing could page on it:

  • the timely findings were the right shape and capped under the gate — ANOMALY_QUERY_DURATION 0.72, ANOMALY_BATCH_REQUESTS 1.0, ANOMALY_CPU_SPIKE 0.56, against notify_severity 1.5;
  • the finding that did clear 1.5 (PLAN_REGRESSION, 20:38:49Z) keys off query_store data that had not landed yet;
  • no alert family covers the shape: long_running_query triggers at 30 minutes and the victims died at 30 seconds, poison_wait does not class PAGEIOLATCH, blocking/deadlocks do not apply, CPU stayed near 50%.

So the tool watched an outage it could describe and could not report. This adds the finding the issue proposed, computed from the two collectors that were already current.

The finding

SAME_STATEMENT_PILEUP: N concurrent sessions on one statement at one snapshot instant, every copy past a floor of elapsed time, at least one suspended on an IO-class wait, where that same statement's own snapshots across the prior 45 minutes were sub-second. The conjunction is what makes it specific — on the fleet data it fired exactly once in four server-days, and that firing was this incident.

Severity prices magnitude rather than presence (#3462): sessions × min-elapsed-seconds ÷ 40, capped at the band ceiling.

Placement, and the achieved latency

The finding is evaluated on the collection cadence, not the analysis interval — the Darling worker's per-server sweep on the 30-second alert-sweep stamp, Lite's one-minute collection loop, in both cases immediately behind the collectors that write its input.

Achieved latency for the measured incident: inside one collection cycle of the snapshot that proved it. The 20:10:31Z snapshot is read and evaluated on the next sweep stamp (≤30 s on Darling, ≤60 s on Lite), so the finding exists at severity 2.0 against the 1.5 gate — and notifies — by roughly 20:11Z, while the episode that self-cleared around 20:11 was still alive. Against the measured baseline: the scheduled anomalies landed 20:31:48Z (21 minutes, all capped ≤1.0), and the notify-eligible PLAN_REGRESSION could not exist before ~20:38Z (~28 minutes). Detection moves from ~28 minutes after the damage to inside the damage.

It also deliberately does not inherit the scheduled pass's 24-hour data-span gate. That gate exists because fraction-of-period fact scoring distorts on short windows; this finding's evidence is one snapshot plus one statement's own recent history, so it works on a young install.

Alternatives rejected, and why the issue was right to reject them

Query Store independence is a requirement, not a convenience

On at least one production store class the Query Store readers are deliberately disabled (#2296) — which is the deployment this incident happened on, where the plan-count evidence had to come from outside the product. A pileup detector that reached for query_store identity would compile, test green against a QS-enabled fixture, and ship blind on exactly the servers that needed it.

So the trigger, the baseline, the statement identity and the incident fingerprint are computed from the active-query snapshot collector alone, and the statement key is the DMV's own query_hash off sys.dm_exec_requests rather than anything Query Store hands out.

SameStatementPileupSourceCensusTests pins this from source, for the detector and both editions' readers (enumerated rather than intersected — a census that checks one SKU certifies only that SKU):

  • no query_store identifier in code or in SQL literals on the trigger path (comments are exempt on purpose — the detector's own remarks have to be able to explain why the dependency is forbidden; this is the inverse of the delivery census's strip-the-strings rule, because here a table named inside a literal is the dependency under test);
  • every FROM/JOIN target in the readers' literals is one of the permitted collector tables, and the detector's literals name none at all (it is a pure function of rows);
  • the fingerprint is a pure function of (server, database, statement identity) — no store read, nothing a disabled Query Store could withhold.

PLAN_REGRESSION and the #2138 force-plan bot stay exactly what they were: follow-on enrichment where QS happens to be readable, never the trigger.

Thresholds, derived from fleet distributions

Pulled live from the Darling store's active-query snapshots: four production servers × 24 hours, ~4,160 snapshot instants. Same-statement concurrent groups: 484 at n ≥ 2, 103 at n ≥ 3 — but only four n ≥ 3 groups had every session past ten seconds, and three of those four are routine batch shapes. Each constant is named per #3060 with its derivation stated on it, in the unit the gate compares (#3463).

Constant Value Why that value
MinConcurrentSessions 3 n ≥ 2 groups number 484 in four server-days and are full of ordinary paired workloads (job pairs, two-lane ETL, index maintenance). Three concurrent copies is where a shared plan flip shows itself — which is what two coincidentally-slow sessions do not prove.
MinElapsedMs 10,000 The victims died at the client's 30-second timeout; a 10 s floor pages while they are still alive. The measured pack cleared it with 2.7× headroom (min 27,703 ms). Below it the fleet's 5–10 s band holds routine permission-cache rebuilds at n ≥ 3.
MinIoWaitSessions 1 The mechanism gate. One, not two, because page-IO waits are millisecond-scale — the incident's suspended copies showed wait_time_ms 5 — so sessions bounce between suspended and runnable and demanding two simultaneously suspended makes the gate a coin flip on sampling phase. The incident showed two; requiring one keeps the evidence without betting the page on sampling luck. Specificity comes from the conjunction.
IoClassWaitPrefixes PAGEIOLATCH_, IO_COMPLETION, ASYNC_IO_COMPLETION, WRITELOG, WRITE_COMPLETION The measured shape plus the same mechanism on other IO paths. BACKUPIO/BACKUPTHREAD deliberately excluded — a backup convoy is not a plan flip.
BaselineLookbackMinutes 45 Long enough to catch a fast statement in flight at the ~1-minute snapshot cadence (the measured statement was observed at 54 ms 3.5 minutes before), short enough that the window self-clears after an episode — a fired pileup leaves 27-second rows in it, so the same statement cannot re-fire until they age out. That quiet period is deliberate and sits near the notify cooldown's floor.
SubSecondBaselineCeilingMs 1,000 The issue's own wording, and the measured in-window observation was 54 ms — 18× under. A single recent multi-second observation means the statement was already slow, which is a different finding's beat.
MinBaselineObservations 1 A statement never seen before the pileup has no "normally sub-second" to betray. The fleet's loudest counterexample (a sync job at n = 5, all past 13 s) has zero priors and is excluded by exactly this. Thin alone; load-bearing only in the conjunction, which measured clean.
StaleSnapshotCutoffMinutes 10 The staleness rule the long-running-query alert already applies to this table (#1812's shape): a collector outage must not hold a level-triggered evaluation firing on frozen rows.

The measured incident clears every gate; ordinary workload does not. The three non-incident fleet shapes that reach n ≥ 3 past ten seconds each fail a different gate, which is the argument that no single threshold is carrying the finding:

  • an inventory-sync job (n = 5, all past 13 s) — zero IO-class waits, and no prior observation;
  • a permission rebuild (n = 3, past the floor) — priors already at 11.9 s, so never sub-second;
  • a nightly catalog scan (n = 3, 24 s, on page IO) — no baseline observation at all;
  • and a slow shipment statement with a 2-second prior — over the baseline ceiling: already-slow getting slower, not a flip.

All four are tests.

Severity calibration, in the unit the gate compares

severity = min(2.0, sessions × min-elapsed-seconds ÷ 40). Stated to the figures the arithmetic actually produces, because a calibration claim stated in a rounded unit is not the claim the gate evaluates (#3463):

Shape Session-seconds Severity Pages?
Measured incident (3 × 27.703 s) 83.109 2.0777 → capped 2.0 yes — 0.578 above the gate
Notify boundary (3 × 20 s, 4 × 15 s, 6 × 10 s) 60 1.5 yes (gate is >=)
Just under (3 × 19 s) 57 1.425 no
Minimum firing shape (3 × 10 s) 30 0.75 no — visible, persisted, drill-downable
Any two-session pileup — no finding at all no

The measured incident clears the gate even with two of its three sessions discounted to the 10-second floor. The notify boundary is three concurrent copies each past two-thirds of the timeout that killed the measured victims. And the issue's "a two-session brief pileup must not notify" holds one step stronger than asked: the session floor is a predicate, not a severity input, so two sessions produce no finding at 12 s, 29 s or 600 s.

Recurrence folds instead of surprising

The measured statement re-drew its regressed plan later the same evening after fully self-clearing, so the finding gets the incident-fingerprint treatment every alert family already has: the IncidentId recipe extended with the statement identity. Episode two lands on episode one's incident trail as an occurrence (occurrences / first_seen / last_seen via the existing (story_path_hash, incident_id) folding) rather than reading as a fresh surprise, and #2138 gets a trigger token stable across episodes, plan churn, restarts and QS resets. The story path carries the statement identity too, so muting or cooling down one statement's pileup cannot answer another statement's question (#3459's rule).

Stated plainly because it is a real behavior and not an accident: a recurrence inside the 6-hour notify cooldown folds silently rather than re-paging, because a band-saturating severity cannot satisfy the #2054 worsening step. The occurrence trail and the viewer show it; the channel does not repeat it. That is the house's standing-condition policy, and it is what "folds onto episode one rather than reading as fresh" means.

Both editions

Lite's active-query snapshots feed the same seam — one shared collector definition, one table shape — so the predicate, thresholds, severity arithmetic, fingerprint and noise filters live once in PerformanceMonitor.Analysis as a pure function of snapshot rows. Each edition contributes only its window reader (Npgsql over query_snapshots, DuckDB over v_query_snapshots) and its sweep call site. Both ship it.

It adds no delivery semantics — and is censused as the call site it is

The finding rides the existing analysis-finding machinery end to end: the mute registry, the two-phase materialize → enrich → persist write, occurrence folding, notify_severity, the incident-keyed cooldown and its worsening rule. It is, however, a new delivery call site on both SKUs — a different thing, and accounted for as one:

One read-side change, in both stores, for the same reason

The latest-batch read now excludes pileup rows from defining MAX(analysis_time) and overlays the ones newer than it. #2448's contract is that the newest batch is one indivisible statement about a server — the complete output of one pass — and a per-sweep row claiming the MAX would replace the last complete analysis with a single-row one, making a server read healthier mid-incident. That is #2448's own misreading arriving through the write cadence instead of through a torn batch. The overlay is self-limiting (visible from firing until the next scheduled pass completes, then living on in the windowed reads and its incident trail), and the epoch fallback keeps a pileup visible where no scheduled batch exists to anchor on — never a disabled-analysis deployment (the pileup sweep rides the same analysis-enabled gate, so analysis off means no pileup rows either), but a young install whose pileup fired before the scheduled pass's 24-hour data-span gate let a first batch exist, or analysis toggled off after pileup rows landed and before any scheduled batch did. Both stores carry the identical shape and are pinned against each other.

Migration rung

None claimed, and none needed. The finding persists through analysis_findings' existing columns, reads query_snapshots' existing columns, and keeps its one piece of cross-sweep state — the last-evaluated snapshot instant — in memory, where a restart costs at most one repeated evaluation whose finding folds onto the same incident id and whose notification the cooldown (seeded from the alert log on first lookup) suppresses. The stamp advances only once the instant's outcome is durable — evaluated clean, fully muted, or persisted — so a transient store fault re-derives the same instant on the next sweep instead of losing a firing instant's finding, and the same folding-and-cooldown absorption covers the replay; the ordering is pinned from source on both call sites. The rung available to this lane is left unspent.

Tests

The measured incident reconstructed verbatim from the store's own rows, and the shapes that must stay quiet:

  • the 20:10:31Z incident — fires at notify-eligible severity, with the statement identity, the session/elapsed evidence, the drill-down's three sessions (worst-first, with waits and reads) and its baseline row, and the frozen value-stated advice quoting the measured wait type;
  • the 20:07:01Z pre-incident snapshot — must not fire;
  • busy-but-healthy — eight concurrent sessions, all deep, all on page IO, all on different statements, each with its own sub-second baseline: must not fire;
  • a long-running single session (an hour deep) — must not fire; that is long_running_query's job;
  • two sessions at 12 s / 29 s / 600 s — must not fire;
  • the fleet's other three deep groups — each failing a different gate;
  • no-IO, stale-snapshot, noise-shape (sp_server_diagnostics / WAITFOR / backup), one-deep-copy-among-fast-siblings, and duplicate-session-rows — must not fire;
  • the onset-poisoned sequence — first sample at ~5 s (no finding: the floor), second sample past the floor (still no finding: the ceiling gate reads the onset rows) — pinned as the accepted blind spot, with the counterfactual (the identical second instant, onset sample absent) firing;
  • episode folding — episode two shares episode one's incident id and mute key, while the fingerprint still separates statements, databases and servers, and two statements piling up at once produce two distinct findings;
  • severity — pinned at six points with the notify boundary expressed in session-seconds, plus the formula's unit identity (severity × divisor == session-seconds) so the divisor cannot drift from its doc comment;
  • QS independence — pinned behaviorally (the snapshot row type is the input contract, enumerated member by member) and structurally (the three-file source census).

Verification, and its limits

Every touched assembly builds, plus the whole solution: PerformanceMonitor.Analysis, Darling.Analysis, Darling.Service, Lite, Darling.Tests, Lite.Tests.

The test projects target net10.0-windows and cannot execute on this machine, so the assertions were additionally replayed out of band against the real code: PerformanceMonitor.Analysis is net10.0, so a throwaway net10.0 console harness ran the detector itself over the same fixtures (102 checks, all passing), and a second harness replayed the census logic — the #3465 file-set derivation, per-member site counts, Inline gate ordering, the notify pins, the #3467 QS-independence and store-SQL pins, and the dedup-stamp ordering pin — against the worktree with the test project's own CSharpSourceWalker copied in verbatim (93 checks, all passing), with its census table verified content-identical to the committed one. What that does not cover: the xUnit runner's own discovery/attribute handling, and anything requiring a live store — the two readers' SQL is verified by column-name audit against the collector definition and both generated schemas, and by shape-match against the existing long-running-query read of the same table, not by execution.

What this cannot see

An episode whose own onset poisons its baseline. If the collector's first sample of an episode lands while the sessions are between the sub-second ceiling and the elapsed floor — 1–10 s in — that instant correctly does not fire (the floor), and its rows then sit in the 45-minute lookback as multi-second priors: every later instant of the same episode fails the sub-second baseline gate against them, and the episode never fires at all. It fails quiet — a missed page, never a false one — and the exposure is sampling phase: a 9-second landing band in a 60-second collection cycle, roughly 9-in-60 on the measured 90-second shape (under ~1 s the onset rows are themselves sub-second and reinforce the baseline; past 10 s the first sample fires directly). Accepted deliberately: both repairs are gate changes — onset-exclusion, or pricing on the qualifying subset past the floor — each with edges of its own, and neither ships ahead of a measured miss to calibrate against. The recurrence watch is the instrument that would produce one: episodes fold onto a statement-scoped incident id, so a statement whose trail holds fired episodes alongside snapshot-visible pileups that produced no finding is the reopen shape. The detector's remarks carry the full statement, and the poisoned sequence is pinned as accepted behavior, counterfactual included.

The same reopen shape covers the gate's fresh-arrival dilution: a caller that joins the convoy just before the sample resets the pack's minimum below the floor and suppresses detection while the pileup is growing. The qualifying-subset alternative fixes both edges at once; it waits on re-deriving the fleet calibration, which was measured under the every-session predicate.

Deferred, deliberately

  • The waiting-task collector as a second input. The issue names it among the permitted inputs, and the 20:10:34Z sample corroborates the episode (including a tempdb WAIT_ON_SYNC_STATISTICS_REFRESH, plausibly the recompile trigger itself). The trigger does not need it — wait_type and status ride on the snapshot row, which is where the IO-class evidence comes from — so adding a second per-sweep read for drill-down enrichment alone was not worth doubling the reader surface on a rarely-fired finding. It is deliberately not in the census's permitted-tables set: pre-cleared, the read could arrive later without touching a census line, which is the opposite of what the census is for. Adding it means widening the permitted set in the same change — the census fails, names the file, and the dependency lands as a recorded decision.
  • Operator-tunable thresholds. The four shape constants stay in code for now; notify_severity is already the operator's knob over whether a given pileup pages. A control-plane knob rung is available if the fleet shows a deployment that needs a different shape.

…current while it ran (#3467)

A parameterized statement joining a #temp table flipped to a plan doing
~35x the logical reads. Every caller shares the parameterized plan, so
ten tenants inherited the flip together, the copies convoyed on
PAGEIOLATCH_SH, physical reads saturated the volume, and sixteen
executions died at the client's 30-second command timeout. The episode
ran about ninety seconds and self-cleared when the plan churned out.

The store was AHEAD of it the whole time. The 20:10:31Z active-query
snapshot held three concurrent copies of the one statement - 29,124 ms /
28,343 ms / 27,703 ms elapsed, 287,049 logical reads on the leader, two
suspended on page IO - next to that same statement's 54 ms observation
from 3.5 minutes earlier. Nothing could page on any of it: the timely
findings were correct and capped under the threshold
(ANOMALY_QUERY_DURATION 0.72, ANOMALY_BATCH_REQUESTS 1.0,
ANOMALY_CPU_SPIKE 0.56, against notify_severity 1.5), the finding that
DID clear it keys off query_store data that had not landed yet (~28
minutes), and no alert family covers the shape - long_running_query
fires at 30 minutes and the victims died at 30 seconds, poison_wait
does not class PAGEIOLATCH, CPU stayed near 50%. The tool watched an
outage it could describe and could not report.

* The finding. SAME_STATEMENT_PILEUP: N concurrent sessions on one
  statement at one snapshot instant, EVERY copy past a floor of elapsed
  time, at least one suspended on an IO-class wait, where that same
  statement's own snapshots across the prior 45 minutes were sub-second.
  The conjunction is what makes it specific rather than any single
  threshold - on four production servers x 24 hours (~4,160 snapshot
  instants) it fired exactly once, on this incident.

* Placement is the point, not an implementation detail. It is evaluated
  on the COLLECTION cadence - the Darling worker's per-server sweep on
  the 30-second alert-sweep stamp, Lite's one-minute collection loop -
  immediately behind the collectors that write its input, so the
  20:10:31Z snapshot produces a notify-eligible finding inside one
  collection cycle instead of 21 minutes later at the scheduled pass. It
  is NOT on the 5-360 minute analysis interval, and it deliberately does
  not inherit the scheduled pass's 24-hour data-span gate: that gate
  exists because fraction-of-period fact scoring distorts on short
  windows, and this finding's evidence is one snapshot plus one
  statement's own recent history.

* Query Store independence is a requirement, not a convenience. On at
  least one production store class the Query Store readers are
  deliberately disabled (#2296) - the deployment this incident happened
  on, where tonight's plan-count evidence had to come from outside the
  product. A detector keyed on query_store identity would compile, test
  green against a QS-enabled fixture, and ship blind on exactly the
  servers that needed it. So the trigger, the baseline, the statement
  identity and the incident fingerprint are all computed from the
  active-query snapshot collector alone, and the statement key is the
  DMV's own query_hash off sys.dm_exec_requests rather than anything
  Query Store hands out. SameStatementPileupSourceCensusTests holds that
  from source for the detector and BOTH editions' readers: no
  query_store identifier in code or SQL, and the only tables the trigger
  path may read are the permitted collector tables. PLAN_REGRESSION and
  the #2138 force-plan bot stay what they were - follow-on enrichment
  where QS happens to be readable, never the trigger.

* Thresholds from fleet distributions, named per #3060, each derivation
  stated on its constant in the unit the gate compares. Three sessions,
  because n>=2 same-statement groups number 484 in four server-days and
  are full of ordinary paired workloads while n>=3 past the elapsed
  floor numbered four; a two-session pileup therefore produces NO
  finding at any duration, which is the issue's "must not notify" held
  one step stronger than asked. Ten seconds, because the victims died at
  the client's 30-second timeout and a 10 s floor pages while they are
  still alive (the measured pack cleared it with 2.7x headroom). One
  IO-waiting session, not two: page-IO waits are millisecond-scale - the
  incident's suspended copies showed wait_time_ms 5 - so demanding two
  SIMULTANEOUSLY suspended makes the gate a coin flip on sampling phase,
  and specificity is carried by the conjunction instead. A 45-minute
  sub-second baseline window with at least one observation in it, which
  is what turns "normally fast" into a measurement: the three fleet
  shapes that reach n>=3 past ten seconds and are NOT incidents fail
  precisely this gate or the IO one - a sync job at n=5 with no IO wait
  and no prior observation, a permission rebuild whose priors were
  already at 11.9 s, a nightly catalog scan with no baseline at all.

* Severity prices magnitude, not presence (#3462), in the unit the gate
  compares (#3463): sessions x the pack's minimum elapsed seconds, over
  40 session-seconds per severity point, capped at the band ceiling. The
  measured incident - 3 x 27.703 s = 83.109 session-seconds - scores
  2.0777 before the cap, 0.578 severity points above the 1.5 gate, and
  would still clear it with two of its three sessions discounted to the
  floor. The minimum firing shape (3 x 10 s = 30 session-seconds) scores
  0.75: a persisted, drill-downable, visible finding that does not page.
  Notify-eligibility begins at exactly 60 session-seconds - three
  concurrent copies each past two-thirds of the timeout that killed the
  measured victims.

* Recurrence folds instead of surprising. The measured statement
  re-drew its regressed plan later the same evening after fully
  self-clearing, so the finding gets the incident-fingerprint treatment
  every alert family already has: the IncidentId recipe extended with
  the statement identity, which keeps episode two on episode one's
  incident trail as an occurrence (first_seen / last_seen / occurrences,
  the drill-down an operator wants when deciding whether the plan
  deserves forcing) and gives #2138 a trigger token that is stable
  across episodes, plan churn, restarts and QS resets. The story path
  carries the statement identity too, so muting or cooling down one
  statement's pileup cannot answer another statement's question
  (#3459's rule).

* Both editions, one detector. Lite's active-query snapshots feed the
  same seam - one shared collector definition, one table shape - so the
  predicate, the thresholds, the severity arithmetic, the fingerprint
  and the noise filters live once in PerformanceMonitor.Analysis as a
  pure function of snapshot rows, and each edition contributes only its
  window reader (Npgsql over query_snapshots, DuckDB over
  v_query_snapshots) and its sweep call site.

* It adds no delivery semantics. The finding rides the existing
  analysis-finding machinery end to end - mute registry, two-phase
  materialize-enrich-persist, the (story_path_hash, incident_id)
  occurrence folding, notify_severity, the incident-keyed cooldown and
  its #2054 worsening rule - which also means a recurrence inside the
  6-hour cooldown folds silently rather than re-paging, because a
  band-saturating severity cannot worsen. It IS a new delivery call site
  on both SKUs, which is a different thing and is accounted for as one:
  each sits under ShouldNotifyAnalysisFindings (master AND family), each
  carries its own #3465 census entry, and AlertMasterSwitchSurfaceTests
  pins both in both directions, with the predicate's call count raised
  from two to three. The README sentence that makes alerts_enabled a
  promise widened from "scheduled-analysis finding notifications" to
  "analysis finding notifications" - the narrower wording described a
  promise smaller than the one the gates keep, and an operator could
  reasonably have read a non-scheduled finding as escaping the switch,
  which is the exact confusion #3464 was about.

* One read-side change, in both stores, for the same reason. The
  latest-batch read excludes pileup rows from DEFINING MAX(analysis_time)
  and overlays the ones newer than it. #2448's contract is that the
  newest batch is one indivisible statement about a server - the
  complete output of one pass - and a per-sweep row claiming the MAX
  would replace the last complete analysis with a single-row one, making
  a server read HEALTHIER mid-incident. That is #2448's own misreading
  arriving through the write cadence instead of a torn batch. The
  overlay is self-limiting (visible until the next scheduled pass
  completes, then in the windowed reads and its incident trail), and the
  epoch fallback keeps a pileup visible where analysis is disabled and
  no scheduled batch exists.

No migration rung is claimed and none is needed: the finding persists
through analysis_findings' existing columns, reads query_snapshots'
existing columns, and keeps its one piece of cross-sweep state
(the last-evaluated snapshot instant) in memory, where a restart costs
at most one repeated evaluation that folds onto the same incident id.

Tests: the 20:10:31Z incident reconstructed verbatim from the store's
own rows (notify-eligible severity, statement identity, session and
elapsed evidence, the drill-down's sessions and baseline, and the frozen
value-stated advice quoting the measured wait type); the 20:07:01Z
pre-incident snapshot must not fire; a busy-but-healthy eight-session
parallel workload on DIFFERENT statements must not fire; a long-running
single session must not fire; two sessions must not fire at 12 s, 29 s
or 600 s; the fleet's three other deep groups must not fire, each
failing a different gate; no-IO, stale-snapshot, noise-shape,
one-deep-copy-among-fast-siblings and duplicate-session shapes must not
fire; episode two folds onto episode one's incident id and mute key
while the fingerprint still separates statements, databases and servers;
the severity table is pinned at six points with the notify boundary
expressed in session-seconds; and QS independence is pinned both
behaviorally (the row type is the contract) and structurally (the source
census over three files).
…p from outrunning the persist (#3467)

Review of the pileup rung surfaced one behavior worth fixing and one worth
stating out loud instead of fixing.

The statement first, because it is the bigger admission. The detector's
baseline gate reads the statement's prior in-flight observations - and the
collector can make the episode itself one of them. If the first sample of
an episode lands while the sessions are between the sub-second ceiling and
the elapsed floor (1-10 s in), that instant correctly fires nothing, and
its rows then sit in the 45-minute lookback as multi-second priors: every
later instant of the same episode fails the sub-second ceiling against
them, and the episode never fires at all. It fails quiet - a missed page,
never a false one - and the exposure is sampling phase: a 9-second landing
band in a 60-second collection cycle, roughly 9-in-60 on the measured
90-second shape. This is accepted, deliberately: both repairs are gate
changes (onset-exclusion, or pricing on the qualifying subset past the
floor), each with edges of its own, and neither ships ahead of a measured
miss to calibrate against - the recurrence watch, which folds episodes onto
a statement-scoped incident id, is the instrument that would produce one.
The detector's remarks now carry the full paragraph, and the poisoned
sequence is pinned as accepted behavior: instant one at ~5 s (no finding,
the floor), instant two past the floor (still no finding, the ceiling reads
instant one's rows), and the counterfactual - the identical second instant
without the onset sample - firing. The every-session floor's fresh-arrival
dilution gets the same treatment beside its comment: the qualifying-subset
alternative is named, the reason it waits (the fleet calibration was
measured under this predicate) is stated, and the same reopen shape covers
it.

The fix: both call sites advanced the per-instant dedup stamp BEFORE
InsertFindingsAsync, so a transient persist fault marked a firing instant
"done" and lost its finding for good if the episode cleared before the next
collector write. The stamp now advances only once the instant's outcome is
durable - evaluated clean, fully muted, or persisted - and a fault leaves
it behind so the next sweep re-derives the same instant, where occurrence
folding absorbs the re-persist and the incident-keyed cooldown absorbs the
re-notify. The ordering is pinned from source in both files.

The polish, each item its own small honesty:

* The census's permitted-tables set pre-cleared waiting_tasks, which made
  "adding it later is a deliberate census change" false - the read could
  have arrived without touching a census line. The permission is removed:
  adding the read now means widening the set in the same change, so the
  dependency lands as a recorded decision.
* The worker's doc quoted the README's pre-#3467 contract sentence
  ("scheduled-analysis finding notifications"); the quote now matches the
  widened sentence this lane shipped.
* Both finding stores justified the epoch COALESCE with a deployment that
  cannot exist (analysis disabled writes no pileup rows either - the sweep
  rides the same gate); the comments now name the real anchors: a young
  install whose pileup fired before the data-span gate let a first
  scheduled batch exist, and analysis toggled off after pileup rows landed.
* PgPileupSnapshotReader's catch now uses the directory's own
  AnalysisShutdown.IsExpectedAbandon filter (#2299) instead of a bare
  OperationCanceledException carve-out, so shutdown residue propagates
  quietly and a cancellation that nobody requested keeps its log line.

Out-of-band verification (the test projects target net10.0-windows): the
detector harness replays 102 checks and the census harness 93, both clean,
including the onset pin and the stamp-ordering pin.
@erikdarlingdata
erikdarlingdata enabled auto-merge (squash) September 16, 2026 00:00
var baselineFloor = latest.AddMinutes(-BaselineLookbackMinutes);

/* The candidate groups: the newest instant's rows, by statement identity. */
foreach (var group in usable.Where(r => r.CollectionTime == latest).GroupBy(StatementIdentity))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness: grouping by StatementIdentity (query_hash) alone, without DatabaseName, can conflate unrelated slow queries in different databases into one bogus "shared plan" pileup.

query_hash is computed from the normalized query text and is not scoped to a database — two different user databases running textually-identical parameterized SQL (a very common shape in exactly the multi-tenant-per-database SaaS deployments this finding targets) produce the same hash. But the entire causal story this detector tells ("a shared parameterized plan flipped, every caller of that one plan piles up together") is only true within one database, because a SQL Server plan-cache entry is keyed per-database (dbid is part of the compiled-plan cache key) — a plan going bad in database A cannot be "the same plan" as one in database B, even for byte-identical text.

Concretely, three sessions in three different databases, each independently slow (unrelated causes) for >10s on a query shape that happens to share a query_hash, with one on an IO wait, would be merged here into a single SAME_STATEMENT_PILEUP finding — miscounted as "3 concurrent sessions piled up on one statement in {one arbitrarily-picked db}" and paged at a severity computed from the combined (unrelated) session count.

This also affects:

  • The baseline check at line ~317-320 (StatementIdentity(r) == identity, no database filter): a genuinely-new slow query in DB B could pass the "normally sub-second" baseline gate using DB A's unrelated fast history for the same hash — or a real same-DB pileup could be suppressed because an unrelated slow occurrence of the same hash in another DB pollutes the baseline's sub-second ceiling check.
  • databaseName at line 390 (pack.Select(r => r.DatabaseName).FirstOrDefault(...)): picks one database out of what could be a multi-database pack, non-deterministically (row order isn't guaranteed stable for ties at the same collection_time). Since ComputeIncidentId folds on (serverName, RootFactKey, databaseName, identity), this can also make the same recurring incident fold onto different incident IDs across evaluations if the "first" database picked changes — undermining the occurrence-folding guarantee this PR is built around.

Worth scoping the grouping (and the baseline lookup) to (DatabaseName, StatementIdentity) rather than StatementIdentity alone, matching how a plan-cache entry is actually scoped.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Real finding, all three parts — and it's now #3474 rather than a quiet default, because the fix has a fork in it: the measured incident's convoy may itself have spanned tenant databases ("ten tenants inherited the flip together"), in which case scoping the group key per-database would have missed the exact outage this detector was built for. The issue lays out both options plus the deciding evidence (the incident's own snapshots record DatabaseName per session — that lookup happens before any code moves). The FirstOrDefault fold-attribution nondeterminism is real under every option and flagged to ship first.

@claude

claude Bot commented Sep 16, 2026

Copy link
Copy Markdown

Reviewed the same-statement-pileup finding (#3467): detector, both SKUs' readers, worker/service wiring, finding-store overlay SQL, advice, and the extensive pinning tests (census, master-switch, calibration).

Overall: well-scoped and heavily self-verified — the source-census tests keep the trigger path off Query Store as claimed, the master-switch surface tests keep delivery gated identically to the scheduled path, and the calibration/severity math checks out against the fleet numbers cited in the docstrings. Lite and Darling stay in lockstep (same detector, same D0 production/delivery split, same dedup-stamp-after-persist discipline, same latest-batch carve-out SQL shape, verified by BothFindingStores_CarveThePileupRowsOutOfTheLatestBatch_Identically). Confirmed separately that Lite has no PostgreSQL target seam at all, so the missing engine guard in RunPileupSweepAsync (present in Darling's EvaluateSameStatementPileupAsync) isn't a parity gap — it's dead code that can't apply there.

Left one inline finding: SameStatementPileupDetector.Evaluate groups candidate pileups by StatementIdentity (query_hash) alone, without DatabaseName. Since query_hash isn't database-scoped but SQL Server's plan cache is (a plan flip in one database can't be "the same plan" as another database's), this can conflate unrelated slow queries across databases into one bogus finding, pollute the sub-second baseline check cross-database, and make incident-id folding non-deterministic when the pack spans databases. Recommend scoping the group/baseline key to (DatabaseName, StatementIdentity).

No SQL injection, secrets, or file/network/process concerns — all reads are parameterized ($1/$2 or DuckDBParameter), and no T-SQL was touched by this PR (the shared collector query itself is pre-existing).

…ps (#3467)

CI's only failure on 601ab8a (1 of 9,449) was FindingStoreSql_CarriesTheDashboardShape:
its GetLatestFindingsSql pin still asserted the pre-#3467 single-line anchor,
"SELECT MAX(analysis_time) FROM analysis_findings WHERE server_id = $1" — a string that
no longer exists, because the carve-out deliberately rewrote it: the MAX now excludes
SAME_STATEMENT_PILEUP rows from DEFINING the batch (a per-sweep pileup row claiming the
MAX would replace the last complete analysis pass and read the server healthier
mid-incident), and the subquery spans two lines.

The re-pin asserts MORE than the old pin, not less: both fragments as they actually
appear in the const — the MAX anchor line and the WHERE line carrying the exclusion
predicate. The full carve-out contract (overlay arm, epoch anchor, Lite twin parity)
already belongs to SameStatementPileupSourceCensusTests, which normalizes source
whitespace and passed on the same run; this pin's job stays what it was — the dashboard
shape of the Darling store's own consts — now stated against the SQL that ships.

Why the lane's out-of-band verification missed it: the detector and census harnesses
replay the NEW tests (102 + 93 checks, clean); this pin lives in the pre-existing suite
that only executes on Windows CI. A Contains pin between two compile-time strings is
fully verifiable by grep, which is how this fix was checked: each fragment appears
twice in GetLatestFindingsSql (anchor arm and COALESCE arm), and the stale single-line
form appears nowhere outside DELETE cleanup statements.
@claude

claude Bot commented Sep 16, 2026

Copy link
Copy Markdown

Review: SAME_STATEMENT_PILEUP finding (#3467)

Went through the full diff — detector (SameStatementPileupDetector.cs), both SKUs' readers (PgPileupSnapshotReader.cs / Lite/Analysis/PileupSnapshotReader.cs), both worker integration points (DarlingWorker.cs / CollectionBackgroundService.cs), the GetLatestFindingsSql carve-out in both FindingStores, the census tests, and both test suites (~1080 new test lines).

Correctness

  • Predicate logic (session floor, elapsed floor, IO-wait gate, sub-second baseline, staleness cutoff) checked against the test fixtures line-by-line — matches, including the distinct-session-vs-row counting, the worst-first/newest-first drill-down ordering, and the severity arithmetic (sessions × min-elapsed-seconds ÷ 40, capped at 2.0).
  • Verified PileupWindowSql's column list against QuerySnapshotsCollector's actual schema (session_id, query_hash, wait_type, total_elapsed_time_ms, logical_reads, reads, is_cdc_capture, etc.) — all present, and the session_id > 50 / is_cdc_capture filters match the existing long-running-query read of the same table.
  • GetLatestFindingsSql's carve-out (exclude pileup rows from defining MAX(analysis_time), overlay them while newer) is naive-UTC-consistent with the timestamp (not timestamptz) column, so the TIMESTAMP '1970-01-01' epoch fallback compares cleanly.

Lite/Darling parity

  • Detector and its constants live once in PerformanceMonitor.Analysis, so predicate/severity/fingerprint logic can't drift between editions by construction.
  • Darling's worker gates the sweep on Target.Engine != PostgreSql since Darling also monitors Postgres targets that never populate query_snapshots; Lite has no equivalent check, but Lite's ServerConnection/collector surface is SQL-Server-only (no Engine discriminator on the model), so there's no Postgres case for it to filter — not a parity gap.
  • The GetLatestFindingsSql carve-out is byte-identical in shape between PgFindingStore and Lite's FindingStore, and the census test pins them against each other.
  • Darling needs its own NextPileupSweep cadence stamp because its alert-sweep loop runs on a finer tick than the collector; Lite's collection loop tick is the collection cadence, so evaluating on every tick without a separate stamp is correct, not an oversight.

Security: no injection surface — both readers use parameterized queries throughout, no string-built SQL.

One minor nit (style, not a bug): PgPileupSnapshotReader.cs:99 binds the naive-UTC timestamp inline via DateTime.SpecifyKind(windowFloorUtc, DateTimeKind.Unspecified) rather than through a local AsNaive() helper. CONTRIBUTING.md calls this out explicitly under the Darling PostgreSQL conventions ("Several files keep a one-line Naive() helper for exactly this; use it rather than binding a DateTime straight through"). Functionally correct either way since there's only one timestamp parameter in this file, so take it or leave it.

Otherwise didn't find a correctness, parity, or security issue to flag — the census tests and the fleet-shape counterexamples in SameStatementPileupDetectorTests cover the edge cases I'd normally probe for (two-session pileups, mixed-statement concurrency, stale snapshots, duplicate session rows, null query_hash).

@erikdarlingdata
erikdarlingdata merged commit 1772add into dev Sep 16, 2026
8 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/3467-same-statement-pileup branch September 16, 2026 02:21
erikdarlingdata added a commit that referenced this pull request Sep 16, 2026
…nent ordinals (#3466)

CI's only failure on 2b45c77 (1 of 9,486, identical in both jobs) was the V122 rung's
round-trip pin: TheViewerRow_RoundTripsThroughTheBind_AtTheAppendedOrdinals asserted its
pair at Parameters[^2]/[^1] — "the END" — which was true only while V122 was the newest
rung on config_alert_settings. This lane's V124 append (fleet_sweep_enabled $65,
fleet_sweep_interval_minutes $66) followed the append rule exactly, and the first
appended knob is a bool: the V122 pin's IsType<NpgsqlParameter<int>> at [^2] met it and
failed — the correct failure, loudly refusing to read a bool seat as an int seat.

The append discipline itself held on all four parallel sequences (column list, upsert
placeholders, bind order, reader ordinals — verified from the diff: $65/$66 appended
last, reader GetBoolean(64)/GetInt32(65) appended last). No bind bug; a positional pin
aged out.

The fix seats V122's pin at its permanent absolute ordinals: indexes 62/63, matching
the upsert's $63/$64 and the reader's GetInt32(62)/(63) — verified mechanically from
the bind source (62 Adds precede the pair). Absolute is the point: appended columns
never move again, which is what "every earlier ordinal keeps its column" promises, and
which a distance-from-the-end pin structurally cannot assert past the next rung. The
END pin now lives where it belongs — in this lane's own FleetSweepCadenceKnobRungTests,
until V125 repeats this handover, for which the V122 test's comment is now the recipe.

Same failure class, third instance today (#3457's cross-merge arity, #3469's stale
anchor pin, now this): a pre-existing pin meeting newly appended state on the
Windows-CI-only suite that out-of-band harnesses do not replay. The sweep census tests
this lane ships are the pattern that ends the class — census from source, not pins at
a distance.
erikdarlingdata added a commit that referenced this pull request Sep 16, 2026
…d-have-paged ledger (#3466, lane 2 of 4) (#3471)

* Sweep the fleet with memory: the engine that writes lane 1's tables, on the operator's cadence (#3466)

Lane 2 of the approved four-lane build order, stacked on the V123 state
store. The sweep engine is the scheduled, stateful, Darling-only
evaluation the spec decomposes: collect through the daily-summary
scoring path, diff against the previous sweep's persisted rows, advance
watch items through lane 1's state machine, prove the instruments were
alive before believing the quiet, and write the would-have-paged ledger
when the fleet is muted. No feed, no delivery here; those are lanes 3
and 4 against the rows this engine now writes.

* The cadence knob lives where the analysis cadence lives

The spec says user-configured cadence, hourly by default, and the
product already has exactly one "run a whole-fleet evaluation every N
minutes" knob: analysis_enabled / analysis_interval_minutes on the
config_alert_settings control-plane row, wholesale-swapped into the
running config, exposed through get/update_alert_settings, editable in
the Viewer. V124 adds fleet_sweep_enabled / fleet_sweep_interval_minutes
on that exact path - with the one deliberate difference stated
everywhere it matters: the sweep's switch is its OWN, never the alert
master switch's, because sweeps under master-off are the muted-mode
contract's whole point. Bounds are FleetSweepCadence's named constants
([15, 1440], default 60) at every writer - the MCP write bound, the
Viewer's save gate, the worker's read-side clamp - so no accepted value
is ever silently rewritten, the parity every knob rung since V119
holds. The floor is where "two consecutive sweeps" still confirms a
sustained condition rather than one burst read twice; the ceiling is
one sweep a day, the grain lane 4's channel rollup is ruled to.
Darling-only structurally (Lite has no fleet to sweep), recorded as a
group-level omission in McpAlertSettingsKeyTests with the paying test
the self_alerts and health_bands omissions established.

* Verdicts reuse the daily-summary scorer - the same SQL, the same bands

Per-server verdicts come from DailySummarySql.RangeSql bound to the
sweep's own span (GetWindowSignalsAsync - the one aggregate the
calendar, get_daily_summary and now the sweep all read) folded by the
shared DailyHealthBandCalculator, with BuildReasons as every
non-Healthy card's stated reason. Nothing is forked: when a banding
threshold moves, the sweep and the day surfaces move together, and the
engine tests assert the verdicts against the calculator's own answers
rather than against literals. The span anchors on the previous sweep's
instant so no gap opens between documents, capped at the cadence
ceiling so a sweep waking from downtime never claims a window its
retention tiers might not hold; sub-day spans always sit inside the
4-day raw window, so the raw tier is correct by construction.

* The diff is against persisted rows, and a missing anchor says so

Sweep N joins its verdicts to sweep N-1's by server through the store's
new engine-grade verdict read (throwing, unlike the presentation read:
a fault swallowed into empty would make every server read as NEW - a
diff that restates absolutes because the store failed). Band
transitions, new servers and departed servers land in the document; a
first sweep declares no_previous_sweep rather than implying a quiet
history. This is the compare_analysis leg conceptually - two windows,
per-key deltas - applied to sweep state.

* Instrument liveness is judged before anything else is believed

The sweep proves its own instruments and says which: per-server signal
reads that faulted render as unreadable-not-quiet (No Data card, red
liveness, no fabricated staleness), fleet-wide quiet outside the settle
window is dead collection instruments, and the alert path's pass
counter is compared against the PREVIOUS run's recorded total - the
frozen-counter-beside-a-delivering-path reading that caught #3464, now
running on the sweep's own persisted memory. Every unjudgeable
comparison carries its stated cause instead of a verdict: master-off
freezes the counter by design (the #3464 gate stops the pass before
RecordPass), and a restart resets the baseline, detected from the
service's own counting_since.

* Watch items: narrow qualification, honest omission

Three item families in v1, the spec's own contracts: a server whose
verdict banded Critical, a server that reported nothing outside the
settle window, and the fleet-scope instruments-dead condition on the
store's sentinel id. Every active item is evaluated every sweep - a
miss is an evaluation - except where the sweep cannot honestly judge:
staleness items inside the settle window and any item on a server whose
read faulted are OMITTED from the upsert whole, so their counters stand
rather than advancing on evidence that does not exist. A miss hands the
store null evidence so the COALESCE keeps what opened the item. The
episode stamps always describe the current episode; history lives in
the documents that reported it.

* The would-have-paged ledger is derived honestly

Under master-off, each Critical verdict decomposes into the summary
scoring's own critical triggers - deadlocks, collection errors, severe
memory pressure, sustained CPU, heavy blocking - one row per (server,
family) with the measured figure and the threshold it crossed. This is
deliberately NOT an alert-engine replay: the families are the sweep's
vocabulary, and every evidence payload states the derivation, so nobody
reads the ledger as the engine's counterfactual. What it claims is what
it can prove, which on the evidence day was the only surface that
carried an 18-minute CPU pin through an operator mute.

* Restart awareness is a gate

Inside PostRestartSettleWindow (10 minutes - double the worst of the
three ~4-5 minute measurements) a stale fleet is classified startup
transient: no-data cards say so, staleness opens nothing, liveness is
not failed on it, and real data still bands and still advances. The
detector is the service's own start instant, the same counting_since
every health surface reports.

* The worker hosts it like the backlog sweep, and a failure costs a slot

Fire-and-track off the fleet loop with the IsCompleted stacking guard
and the stamp advanced before launch; RunAsync never throws (quiet on
cancellation, one error line otherwise), so a sweep failure costs
exactly one sweep slot and the collection loop nothing. The store's
one-transaction writer means a failed sweep leaves the PREVIOUS sweep
as the newest complete document. The engine makes no delivery call
anywhere - AlertMasterSwitchSurfaceTests' source census has nothing of
this path's to count, and the engine tests pin both the absence and the
master-off compose.

* Retention lands beside the writer, as lane 1's rung doc promised

Runs and their two children prune at the base data horizon (children
first, through their run's swept_at, with an orphan arm covering a
purge that failed between statements - the price of no FKs, paid as one
cheap NOT EXISTS instead of an ordering constraint); watch items prune
on last_seen_at, the backlog's rule, so a carried episode never loses
its row.

* The rung, the probe, and the handoff

V124 follows V122's knob-rung shape exactly: ADD COLUMN IF NOT EXISTS
with the constants as defaults, no GRANT, no second reload beacon (V17's
trigger already bumps config_version). The probe gains the sentinel at
all three sites (ordinal 99); FleetSweepStateRungTests hands the
top-rung claims to FleetSweepCadenceKnobRungTests and keeps everything
true of V123 wherever it sits - the V120->V121->V122->V123 note,
carried once more.

Tests: FleetSweepCadenceKnobRungTests (top-rung claims, every
settings-row surface names both columns, write-bound/clamp/gate parity
on the named constants, defaults named not restated, the worker gates
on the sweep's own switch with the master switch as data, the viewer
bind round-trip at the appended ordinals); FleetSweepEngineTests (pure
compose with a fixed clock: scorer reuse asserted against the
calculator itself, the two-sweep diff, the full watch lifecycle through
the real state machine, miss-on-unsighted, fleet-quiet and read-fault
liveness, the frozen-counter judgment in all four judgeability shapes,
master-off produces-and-ledgers vs alerts-on empty ledger, the settle
window's suppressions and its ten-minute figure, span arithmetic, and
the no-delivery-seam source pin); McpAlertSettingsKeyTests gains the
fleet_sweep group omission with its paying test; the MCP round-trip
census extends through the sample row's two new named arguments.

Development on macOS, where the test assemblies build but cannot run,
so verification was: the full solution builds clean (0 errors; all
warnings pre-existing in untouched files), and an out-of-band harness
referencing the built Service assembly ran 78 checks - every pure
compose behaviour above, plus the LIVE half against a real TimescaleDB
pg17: full-ladder migration to V124 with the knob columns' defaults on
the wire, rung replay idempotence, the probe's on-the-wire column count
against the map arity (100 == 100, mapping to 124 and to 123 without
the new sentinel), two real end-to-end FleetSweepEngine.RunAsync sweeps
against the store (first-sweep settle-window document, then a
master-off sweep anchored on it with the mute header on the run row),
and the real DarlingRetention.PurgeAsync pruning a planted 40-day-old
sweep, its children, a planted orphan child and a long-unseen watch
item while the fresh sweeps survive. All pass.

No changelog entry in this lane, deliberately, continuing lane 1's
stated disposition: nothing renders until the web feed (lane 3), and
the entry belongs to the lane that makes the feature visible rather
than to the engine that fills its tables.

Part of #3466 - lanes 3 (web feed at configurable spans) and 4 (daily
channel rollup ceiling + get_sweep_reports) follow against these rows.

* Last-seen means sighting: carry the watch-item stamps through a miss, and name the second cadence (#3466)

Review caught the sweep engine stamping last_seen_sweep_id/last_seen_at on
every evaluation, which quietly violated lane 1's last-seen-means-sighting
contract and made one class of row immortal: a pending item nothing ever
resights stays pending (the state machine resets its entry count and holds),
the active read re-evaluates it every sweep, and every evaluation re-stamped
last_seen_at fresh — so the retention arm keyed on that column could never
mature for exactly the rows nothing will ever close, while the readers'
ORDER BY last_seen_at DESC ranked "most recently evaluated", which every
active row is, every sweep. BuildItemImage now carries the row's standing
sighting pair through a miss unchanged and stamps only on a hit. Safe for
live episodes: an open item takes at most one standing miss before the exit
bar closes it, so a carried row is never more than one sweep from its last
sighting. The engine tests gain the discriminator (a miss does not move the
pair, a hit does), and the harness now plants a 40-day-old pending item that
a REAL sweep evaluates and misses — the sighting pair stands at its planted
values on the wire, and the real purge prunes the row the pre-fix stamping
kept immortal (85 checks, up from 78, all passing against TimescaleDB pg17).

Also from review: get_alert_settings' description walked self_alerts and
health_bands but never mentioned the fleet_sweep group it reports and called
the analysis cadence the only cadence — the READ tool now carries the update
tool's fleet_sweep paragraph in shorter form (the two knobs, default hourly,
and that the master switch deliberately governs delivery, never sweep
production). Verified before touching it: no census pins the get-side text,
and the cross-SKU byte-identity rule does not apply — Lite's copy is its own
deliberately different words, and its fleet_sweep payload omission is already
argued and paid for by McpAlertSettingsKeyTests. And the AdvanceWatchItems
omission doc now states the settle-window consequence as decided rather than
overlooked: a non-staleness pending item missed inside the window has its
entry count reset, bounded to one sweep because the window sits under the
floor cadence, and a single miss cannot close an open item under the exit bar.

* The end belongs to the newest rung: seat V122's bind pin at its permanent ordinals (#3466)

CI's only failure on 2b45c77 (1 of 9,486, identical in both jobs) was the V122 rung's
round-trip pin: TheViewerRow_RoundTripsThroughTheBind_AtTheAppendedOrdinals asserted its
pair at Parameters[^2]/[^1] — "the END" — which was true only while V122 was the newest
rung on config_alert_settings. This lane's V124 append (fleet_sweep_enabled $65,
fleet_sweep_interval_minutes $66) followed the append rule exactly, and the first
appended knob is a bool: the V122 pin's IsType<NpgsqlParameter<int>> at [^2] met it and
failed — the correct failure, loudly refusing to read a bool seat as an int seat.

The append discipline itself held on all four parallel sequences (column list, upsert
placeholders, bind order, reader ordinals — verified from the diff: $65/$66 appended
last, reader GetBoolean(64)/GetInt32(65) appended last). No bind bug; a positional pin
aged out.

The fix seats V122's pin at its permanent absolute ordinals: indexes 62/63, matching
the upsert's $63/$64 and the reader's GetInt32(62)/(63) — verified mechanically from
the bind source (62 Adds precede the pair). Absolute is the point: appended columns
never move again, which is what "every earlier ordinal keeps its column" promises, and
which a distance-from-the-end pin structurally cannot assert past the next rung. The
END pin now lives where it belongs — in this lane's own FleetSweepCadenceKnobRungTests,
until V125 repeats this handover, for which the V122 test's comment is now the recipe.

Same failure class, third instance today (#3457's cross-merge arity, #3469's stale
anchor pin, now this): a pre-existing pin meeting newly appended state on the
Windows-CI-only suite that out-of-band harnesses do not replay. The sweep census tests
this lane ships are the pattern that ends the class — census from source, not pins at
a distance.
erikdarlingdata added a commit that referenced this pull request Sep 16, 2026
… baseline, and the tempdb rule the evidence demanded (#3475)

The review on #3469 asked the question the merge had defaulted: the
same-statement pileup detector grouped concurrent sessions by query_hash
alone, and query_hash is not database-scoped - on a fleet of identical
tenant schemas, textually-identical SQL in different databases shares a
hash by construction - while a compiled-plan cache entry IS keyed per
database, so the causal story the finding tells (one shared parameterized
plan flipped, every caller of that one plan convoyed) is only strictly
true within one. #3474 pulled the deciding evidence from the store's own
snapshots before any code moved, and the evidence chose option A:

- The measured incident's three convoy sessions all ran in ONE tenant
  database (the "ten tenants" are application-level tenants inside it,
  sharing exactly one plan), so per-database scoping preserves the
  motivating incident intact - its group, its 2.08 severity, and its
  notify margin are unchanged.
- The conflation the review predicted was measured, not hypothetical:
  within the same hour, the same statement text ran 9.7 s / 7.4M reads
  in a DIFFERENT tenant database on the same server - one snapshot's
  timing away from joining the incident's pack under the server-wide key.
- The DMV sometimes attributes a copy's database context to tempdb (a
  16.5 s copy of the incident statement, the day before - the statement
  joins a #temp table and the request's context followed it), a shape
  neither option in the issue had named.

The change, in the detector only (both SKUs' readers already project
database_name and are untouched):

- The group key is (DatabaseScope, StatementIdentity). Tempdb-attributed
  rows form their own (tempdb, statement) group - honest about what the
  DMV said, with reattribution deliberately not attempted: the snapshot
  carries no reliable session-to-database mapping at read time, and a
  guessed attribution would poison exactly the per-database baseline this
  change exists to keep clean.
- The "normally sub-second" baseline reads only the same database's
  history, closing both pollution directions: another database's fast
  history cannot vouch for a statement that is new-and-slow where it
  actually runs, and another database's unrelated slow occurrence cannot
  suppress a real pileup - the direction that fails quiet.
- The evidence pack's database is the group key's database, structurally,
  retiring the first-non-empty pick over the pack that was
  nondeterministic for multi-database packs - ComputeIncidentId folds on
  databaseName, so the pick's wobble destabilized the occurrence folding
  the detector was built around and the stable token #2138 wants.
- The calibration carries over without a re-pull, by argument written
  into the derivation's doc comment: per-database scoping PARTITIONS the
  server-wide groups and can never grow one, so every gate is strictly
  harder to pass and the four-instants-in-four-server-days derivation
  stays conservative under the new predicate.

Tests pin the kill-shot (three same-hash sessions in three databases are
three groups, not a pileup), both baseline-isolation directions, the
tempdb rules (own group, own history, no dilution of the tenant pack's
arithmetic), fold determinism under row-order permutation, and the
null-database fold to one scope.

Closes #3474.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant