Repository navigation
Alerts: a Client Timeouts alert from attention events, with an always-on capture, for Lite and Darling (#5640) - #5714
Merged
Conversation
…ors, the web page and the MCP tools (#5640)
… Darling and Lite ensure, cleanup and removal (#5640)
…a 68, and tests (#5640)
… viewer's mute-rule list on V179, close the Azure session-name arm and add the V179, v68 and Capture Down pins (#5640)
…nto fix/5640-client-timeout-alert-b2
… the V179 probe sentinel in the cagg rung test (#5640)
…are samples (#5640) One count row per run (collection_time = the run, attention_count = the counter delta, event_time NULL), samples with attention_count 0 deduplicated by the event_time watermark, so no attention is counted twice. Drops the dead below-V179 mute read from the viewer. Adds the attention entry to CollectorEngineCapability and a census for shred-gated XE collectors.
…eRuleClientAppTests, and the LF-reader roster (#5640)
… Lite logs the ensure failure once, the probe compares the definition, Darling gates the ensure on its collector, Azure retries without statement text (#5640)
…nto fix/5640-client-timeout-alert-y
…ve zero; read AlwaysOnXeSessions.ReadOnlyDatabaseMessage whole (#5640)
…al pin the dev merge broke (#5640)
…, settings plumbing, adapter reads (#5640)
…e_rule param moved last, pins updated (#5640)
…us allow-list, ReadsPendingWiring exemption removed (#5640)
…application, tests (#5640)
…dows, viewer upsert, get/update_alert_settings, web admin, docs (#5640)
…owance past an as_of, sampled_events_returned never exceeds the limit (#5640)
…d lists share the count's ceiling, test/comment/UI fixes (#5640)
erikdarlingdata
marked this pull request as ready for review
October 13, 2026 06:19
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| /// groups (#5640), so a top list or a sampled event can never come from outside the rows the count came from.</summary> | ||
| public static string SpanFloor(bool bounded) => bounded ? "collection_time > $2" : "collection_time >= $2"; | ||
|
|
||
| /// <summary>Successful run times, newest first. Column 0 = collection_time. DISTINCT: an Azure SQL Database server logs one run per monitored database.</summary> |
There was a problem hiding this comment.
Non-blocking: the stated reason for the DISTINCT is the opposite of what both runners document.
A per-database collector writes one blended collection_log row per run, not one per database — DarlingCollectorRunner.cs:1087 ("same collection_log row"), :2896 ("a per-database split is N:1 against collection_log"), and Lite's RemoteCollectorService.DefinitionRunner.cs:836 ("the per-database cost the blended collection_log row cannot carry"). So on Azure SQL Database there is one run time per cycle and the DISTINCT is a no-op.
That matters because the DISTINCT is doing nothing to protect the rate: if per-database runs did log separate rows at distinct times, AttentionEvaluator.OldestTaken would treat N sibling databases of one cycle as N runs, collapse the span to seconds, and Take would divide a 3-minute count by Math.Max(1d, ~0) — a badly inflated rate on every Azure SQL DB server. The design is safe because the log row is blended, so that is the sentence worth writing here; as it stands a future reader is told the opposite and may "simplify" the DISTINCT away or rely on it for a guarantee it does not give.
| /// <summary> | ||
| /// A plain table on every store (#5640). The 73rd hypertable is a grid decision (#3035, #3044, #3107: the compression | ||
| /// band, the worker sizing and the refresh watch lines all derive from the hypertable count), and attention rows are | ||
| /// low volume: the ring buffer caps the detail at about 1,800 events per read, and a healthy server reads near zero. |
There was a problem hiding this comment.
Non-blocking: three different numbers for one ring-buffer cap, and this is the one the grid ruling rests on.
- here and
ICollectorSchemaInfo.cs:175: "caps the detail at about 1,800 events per read" - the class summary,
AttentionCollector.cs:35: "keeps at most max_events_limit events (1,000 by default)" - the
get_client_timeoutsguide tail in both SKUs: "keeps at most 1,000 events"
The session is created with SET max_memory = 1024 and no MAX_EVENTS_LIMIT (AlwaysOnXeSessions.AttentionTargets), so the binding cap is the 1,000-event default (or the 1 MB memory cap, whichever hits first) — not 1,800. The 1,800 figure is the stated evidence for the StaysPlainTable opt-out and the "73rd hypertable is a grid decision" argument, and AttentionPlainTableTests.TheOptOut_IsDeclaredOnTheAttentionCollectorOnly_AndNamesTheGridDecision asserts the literal phrase "about 1,800 events per read" is present in this file, so the number is now pinned in place. Worth settling on one figure (and if 1,800 was measured rather than derived, saying what it was measured on) before the pin outlives the reasoning.
Comment on lines
+128
to
+133
| public Task<AttentionWindow> GetAttentionWindowAsync(string serverKey, int lookbackMinutes, CancellationToken cancellationToken = default) => | ||
|
|
||
| Task.FromResult(AttentionWindow.Empty); | ||
|
|
||
|
|
||
|
|
There was a problem hiding this comment.
Nit: the inserted fake member carries an editing artifact — a blank line between => and its expression, then two blank lines and a line of trailing whitespace (132) before the next member. Every sibling in this fake is a one-liner.
Suggested change
| public Task<AttentionWindow> GetAttentionWindowAsync(string serverKey, int lookbackMinutes, CancellationToken cancellationToken = default) => | |
| Task.FromResult(AttentionWindow.Empty); | |
| public Task<AttentionWindow> GetAttentionWindowAsync(string serverKey, int lookbackMinutes, CancellationToken cancellationToken = default) => | |
| Task.FromResult(AttentionWindow.Empty); | |
There was a problem hiding this comment.
Verdict: LGTM
Claims
- VERIFIED — third always-on session, server-scoped on SQL Server, per database on Azure, shared Azure session never dropped —
AlwaysOnXeSessions.cs:683; the Azure catalog probe answers2only for a matching definition and1for another, which the driver treats as unusable, so the install falls back to its own session instead of dropping the shared one. - VERIFIED — event_counter is the exact count, ring_buffer the sample, one count row per run only when the count rose —
AttentionCollector.cs:321(is { } delta and > 0); the counter is staged either way andCountDeltareturns the current value on a backwards counter. - VERIFIED — the rate takes the newest runs until they cover 3 minutes and divides by those minutes —
AttentionEvaluator.OldestTaken+Take. I walked a 1-minute cadence, a 5-minute cadence, a failed run inside the span and a quiet run inside it: the count rows admitted by(boundary, ceiling]cover exactly the span they are divided by. - VERIFIED — a Muted fire starts no cooldown, records no tier, touches no failed-send backoff —
AlertEngine.cs:2246(own stamp/run dictionaries),:2267and:2292behindif (!finding.Muted). - VERIFIED — off by default at 10/60, clamped on read in both SKUs,
update_alert_settingsrefuses a crossed pair and a lone crossing bar in-transaction — V179FALSE/10/60,ClampBarson both settings classes, the two checks inDarlingMcpAlertTools. - VERIFIED — Lane F's
sampled_events_returnedfix —ClientTimeoutsRead.cs:133-134over asampleLimit + 1fetch, pinned atClientTimeoutsReadTests.cs:93with(3,false,3), (2,true,2), (1,true,1). - UNVERIFIED — "Lite 91, Darling 182" tool totals: docs and both read-only lists agree, but the totals come from a reflection walk; needs the budget run.
- UNVERIFIED — "DuckDB accepts
COLLATE \"C\"(tested)" and every live-rig result (sql2019, throwaway PostgreSQL, the 33,084 B measurement, the Azure 25744 retry the body itself calls unproven).AttentionWindowReadTestsdoes drive the realCOLLATE/array_agg FILTER/[1]text against a DuckDB file, but only a run settles it.
Riskiest lines
PerformanceMonitor.Collectors/AttentionCollector.cs:126— the batch (THROW 50000probe, event_counter XPath, two-counter shred gate,ring_buffer xml NOT NULL) — pinned byAttentionCollectorDefinitionTestsas TEXT, so the live batch is UNPINNED in CI.TRY_CASTinto aNOT NULLcolumn is the shapeDeadlocksCollector.cs:91and two siblings already ship.PerformanceMonitor.Alerting/AttentionEvaluator.csTake/Boundary/OldestTaken— the midpoint boundary and 30 s ceiling decide which count rows divide by which span — pinned byAttentionAlertTestsandARowPastTheNewestRunPlusSkew_...in both stores.Darling/PerformanceMonitor.Darling.Storage/TimescaleSupport.cs:173—HypertableTablesnarrowed fromCollectorCatalog.All; the conversion walk, phase order andHypertableCountread it — pinned byAttentionPlainTableTests.TheOptOutCensus_ListsExactlyTheAttentionTable.
Traps near this diff
- "Lite and Darling ship in parity: a fix to one SKU's reader, tool, alert or description without the counterpart's is a finding, not a nit" — does not apply:
McpClientTimeoutTools.cs:51andDarlingMcpClientTimeoutTools.cs:65both call oneClientTimeoutsRead.Payloadover oneAttentionWindowSqltext. - "A default interface member compiles a parity gap away" — does not apply:
IAlertReadAdapter.cs:144andIAlertEngineSettings.cs:82,178,184have NO default body. The four implementing-looking files not in the diff (BlockingDeadlockContextBuilderTests,TempDbReservedLabelProvenanceTests,FileGrowthRiseUnitCensusTests,DarlingAlertReadAdapterTests) only reference the types. The one default added,ICollectorSchemaInfo.cs:180, keeps the pre-existingfalseand its override is censused exactly. - "Truncation inferred from a full page ... Fetch
limit + 1; the flag isfetched.Count > limit" — does not apply:LocalDataService.Attention.cs:102,DarlingMcpClientTimeoutTools.cs:158,ClientTimeoutsRead.cs:134. - "A
total_*assigned from a.Countis a count of the page" — does not apply:sampled_events_in_rangeisSampleTotalover the un-LIMITed aggregate (ClientTimeoutsRead.cs:123); the page's figure issampled_events_returnedand nototal_*key exists. - "
hours_back,limit,topare refused when out of range, never clamped" — does not apply:ValidateWindow/ValidateTopreturn before any read (McpClientTimeoutTools.cs:35,38). The bars are clamped, but they are stored settings and both Settings windows refuse a bad pair rather than clamp it. - "An alert evaluated every ~30 s against an observation that changes once per COLLECTION ... remember the observation stamp per key, beside — never folded into — its last-seen stamp" — does not apply:
_lastAttentionRunTime(AlertEngine.cs:144) sits beside_lastAttentionAlertand the fire needsNewestRunTime > lastRun(:2247). - "A delta family that stores no interval cannot tell a restart's zero from a quiet interval" — does not apply for an unusual reason:
attention_counthas no interval column, but a quiet run writes NO ROW (AttentionCollector.cs:321), so no fabricated zero exists; the interval comes from thecollection_logrun times. - "An expression-bodied member holding a
{ }property pattern is read SHORT by the convention guard's member walk (#3667)" — applies, and is answered here:ReadOnlyDatabaseMessageis now a block body with the switch in a local (AlwaysOnXeSessions.cs:743) and theKnownTruncatedRangesexemption is gone.
Divergence (blind read vs PR body)
- A2/L4 says "The one
NpgsqlCommandinReadRangeAsync(it runs all seven statements)". There is no single command: the localQuery<T>builds a NEW command per statement — seven on one connection — each setting the deadline. The fix is right; the sentence is not. - Round 2 (B), UI Low 6 says "no visible text says 'attentions' any more".
ClientTimeoutsBarsInputand the viewer'sBuildAlertRowFromControlsstill say it in their validation messages. Labels were plainly what was meant. - Unmentioned:
PgServerConfigScopeRungTestsswappedCollectorCatalog.All.CountforHypertableTables.Countwhile its comment still says "the two censuses did not move". The catalog count moved 71 → 72 and is still pinned elsewhere, so no coverage is lost.
Three non-blocking inline notes posted separately.
Reviewer checklist
- Run both test projects: this verdict reads the pins, not green. The body leaves 91/182, the 33,084 B measurement and
COLLATE "C"on DuckDB to a run. - Azure SQL DB release pass. A 25744 on
client_hostnameis the unhandled case: it is in neitherAttentionAzureRejectedActions(empty) norAttentionAzureUnverifiedActions(sql_textonly), so the retry would not drop it. - A live SQL Server run of the batch: the
THROW 50000arm on a dropped session, the counter delta across a session restart, and a ring buffer at the 1 MB cap. - An Azure SQL DB server with several databases: confirm ONE blended
collection_logrow per attention run; theDISTINCTwould not save the 3-minute span if the times differ per database. - Open both Settings windows and both mute dialogs on screen — every UI claim is a source/XAML pin and the apps were not run.
- On a real fleet: the collector ships ENABLED, so confirm the new session on every SQL Server is acceptable volume and that
sql_texton every client cancel is the intended default.
…ne ring-buffer cap figure, the XE log-level pin follows the shared ensure, the StateKeys pin gains attention (#5640)
There was a problem hiding this comment.
Verdict: LGTM
Claims
- VERIFIED — "the attention cutoff filters with exist()" —
PerformanceMonitor.Collectors/AttentionCollector.cs:230now readsevt.exist('@timestamp[. > sql:variable("@cutoff_time")]') = 1, matching the patternXeTimeFilterSweepTests.NoCollectorFiltersEventsByTimeWithValuerequires of every SQL-Server-engine collector (that sweep'sCollectorTargetEngine.SqlServerscope already catchesAttentionCollector, since it does not overrideTargetEngine);AttentionCollectorDefinitionTests.cs:111pins the new text. - VERIFIED — "one ring-buffer cap figure" — the stray "about 1,800 events per read" in three doc comments (
AttentionCollector.cs:35,107,ICollectorSchemaInfo.cs:175) is now "1,000 events", matching the DDL (AlwaysOnXeSessions.cs:444,max_memory = 1024only, nomax_events_limitoverride, so the target's documented 1,000-event default applies) and every other reference to the cap already in the tree (ClientTimeoutsRead.cs:49, bothMcpClientTimeoutTools.csdescriptions,McpToolGuideHeads.ClientTimeouts.cs).AttentionPlainTableTests.cs:66pins the figure. - VERIFIED — "the XE log-level pin follows the shared ensure" —
XeSessionPermissionLogLevelTests.csnow asserts the Warn/Error split only onRemoteCollectorService.BlockedProcessReport.cs's singleEnsureServerScopedXeSessionAsynccatch (IsPermissionDenied→ Warn else Error, one match, confirmed by grep), and separately asserts each of the three session files (including the newRemoteCollectorService.AttentionXeSession.cs) hands its session constant to that shared ensure and does not re-catch/re-log.AttentionXeSession.cs:51-53matches the regex exactly, and has nocatch (SqlExceptionorAppLogger.Error("XeSession"of its own. - VERIFIED — "the StateKeys pin gains attention" —
DefaultTraceEventsCollectorDefinitionTests.cs'sdeclaringarray now includes"attention", matchingAttentionCollector.StateKeys(non-empty since before this range) and bringing it into parity with Darling'sCollectorStateContractTests.cs:150, which already listed"attention".
Riskiest lines
PerformanceMonitor.Collectors/AttentionCollector.cs:230— theexist()/sql:variable()rewrite changes the actual filter predicate run against live XML, not just a cosmetic string — pinned byAttentionCollectorDefinitionTestsandXeTimeFilterSweepTestsas TEXT only; no live SQL Server run exercises the new predicate (same gap the prior full review already flagged for this batch).Lite.Tests/XeSessionPermissionLogLevelTests.cs:38-75— the roster (SessionEnsures) is now the only thing stopping a fourth XE-session file from skipping the shared log-once path — checked against the repo: exactly three files callEnsureServerScopedXeSessionAsync, all three rostered, so the roster-blind-spot trap below does not currently have an unseen file.Lite.Tests/DefaultTraceEventsCollectorDefinitionTests.cs:432— the literaldeclaringarray and Darling'sCollectorStateContractTests.cs:150literal are two independent copies pinned to the same content, not aliased — pinned by both tests individually, but nothing stops them drifting again the way this very commit had to fix.
Traps near this diff
- "Two copies of one definition pinned EQUAL to each other fail only after BOTH have drifted (#3684). Alias, do not copy." — applies, and is in fact the root cause this commit fixes:
Lite.Tests/DefaultTraceEventsCollectorDefinitionTests.cs'sdeclaringliteral andDarling/Darling.Tests/CollectorStateContractTests.cs:150's literal are still two separate copies (now back in sync), not a shared source — a future collector that addsStateKeyscan red just one side again. - "A roster-based census does not see the file it does not roster:
EveryPagedTool_ObservesTruncation_…walked ten tools and missed three in another file (#3699)." — does not apply here: grepping the whole tree forEnsureServerScopedXeSessionAsync(callers returns exactly the three filesXeSessionPermissionLogLevelTests.cs's newSessionEnsuresrosters (Deadlocks.cs,BlockedProcessReport.cs,AttentionXeSession.cs), so the roster is complete as of this diff. - "
.gitattributessaystext=auto eol=crlf… a diff that touches every line of a file is a line-ending conversion until proven otherwise." — does not apply:git diff --statfor this range shows small, targeted hunks per file, not whole-file rewrites.
Divergence (blind read vs PR body)
None — this range's commit message ("CI reds, collector side: the attention cutoff filters with exist(), one ring-buffer cap figure, the XE log-level pin follows the shared ensure, the StateKeys pin gains attention") matches the diff on every point, and the prior full review (head=9a1d2f2cfd66de2aedddcdc200c8763ba1cdc2bd) already covers everything else in this PR; this range touches only tests, doc comments and one WHERE clause.
Reviewer checklist
- A live SQL Server run of the rewritten batch: confirm
evt.exist('@timestamp[. > sql:variable("@cutoff_time")]') = 1returns the same rows as the oldevt.value(...) > @cutoff_timecast across a ring buffer with mixed/odd timestamp precision, since no test executes real XML against a real engine.
…ath, resolve the precondition scan's collector const, name the statement-list Take and pin the blended run log row (#5640)
This was referenced Oct 13, 2026
Merged
Closed
erikdarlingdata
added a commit
that referenced
this pull request
Oct 13, 2026
…y run (#5754) (#5757) The Client Timeouts collector no longer fails on every run (#5754). The attention collector from #5714 read the event_counter target's count as `value('sum(.../@count)', 'bigint')`. An XQuery sum() over untyped attributes returns xs:double, and value() reaches bigint through that value's text ("0E0", "1.2E1"). So every run failed with "Error converting data type nvarchar to bigint", because event_counter lists the event even at count 0. The live check on DARLING01 saw 30 of 30 runs fail on SQL Server 2016 through 2025. The read is now CONVERT(bigint, value('sum(...)', 'float')), held in AttentionCollector.EventCountExpression and substituted into both the server and the Azure SQL Database forms. It is exact through 15 significant digits, far beyond any real count. Lite has no SQL of its own and runs the same definition, so this one change fixes both products. AttentionCollectorLiveSqlTests runs the real batch, through Darling's own SqlServerTargetProvider.CreateCommand, against a live SQL Server. It is gated on DARLING_TEST_SQL, creates and drops only its own test session, and has a read-only mode for servers that hold real data. It was red before the fix and green after on all five local servers. XQuerySumIntegerReadGuardTests is a CI-visible source scan that fails on any sum() or avg() read straight into an integer type; it would have caught the #5714 line.
This was referenced Oct 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Claude posting for Erik Darling.
Fixes #5640
Summary
A Client Timeouts alert warns when clients time out or cancel queries at a high rate, in Lite and Darling (#5640).
No signal counted client timeouts. A timed-out query leaves only a sqlserver.attention event, so a storm of timeouts never showed up in any grid or alert.
A third always-on Extended Events session, PerformanceMonitor_Attention, sits beside the deadlock and blocked-process sessions: server-scoped on SQL Server, per database on Azure SQL Database, where the shared session is never dropped. Its event_counter target gives the exact count and its ring_buffer target gives a sample of the events with database, client application, host, session, duration and statement text. The text goes through the existing statement filter. The new attention collector runs every minute in both apps. It stores one count row per run when the count went up, plus the sampled events. Capture Down reports the session as "Client timeouts".
Rung V179 and Lite schema 68 add attention_events (a plain table, kept 30 days), a client application pattern on mute rules, and the alert's switch and two bars on config_alert_settings.
The alert takes the newest successful collector runs until they cover at least 3 minutes and divides the counted events by those minutes. When no successful run is recent enough, it reports no reading and does not fire. A standing alert resolves after 30 minutes without a reading, and the notice gives that reason. Mute rules can match by database and client application. Because only a sample of the events carries those fields, the muted share is estimated from the samples. A fire whose full count crosses a bar but whose unmuted share does not is recorded as Muted. A Muted fire does not start the cooldown, record a tier or touch the failed-send backoff. A Warning that worsens to Critical is delivered at once. The alert card lists the top statements, databases and client applications from the minutes the alert measured, and the mute dialogs prefill the top database and application.
The alert is off by default, with bars of Warning at 10 and Critical at 60 per minute. #5736 sets the defaults from a week of field capture before it is turned on. The switch and the bars are in the Lite and viewer Settings windows, in get_alert_settings in both apps, and in Darling's update_alert_settings, which refuses a Warning above Critical. Both mute dialogs, the web Mute Rules page and create_mute_rule take a client application pattern. get_client_timeouts in both apps reports the rate, the exact count and the sampled events for a chosen time range.
The sections below are the lane-by-lane record, oldest first; later sections correct earlier ones where they say so.
Merge order: after #5706 (V176), #5709 (V177) and #5707 (V178); the dense-ladder pin enforces it.
Lane D: mute by client application
What changes
MuteRule.ClientAppPatternandAlertMuteContext.ClientAppName. Matching is a case-insensitive Contains like the other patterns, and a rule with the pattern never matches a context that has no client application. It shows inSummary(app≈...), is copied byClone, and counts as a constraint, so such a rule is not "matches every alert".PgMuteRuleStoreand the viewer'sViewerDataService.MuteRulesread and writeconfig_mute_rules.client_app_pattern(last column, so the existing$Norder is unchanged; insert is$13, update$12).DuckDbMuteRuleStorereads and writesconfig_mute_rules.client_app_pattern.MuteRuleDialog, the viewer'sMuteRuleEditDialog(new "Client Application (substring)" box; shown for "(any)" and a metric named "Client Timeouts"; seeded fromAlertMuteContext.ClientAppName; saving a rule that has only this pattern no longer asks "mute ALL alerts?"), and the web Mute Rules page (mute-rules.jsfield list).client_app_patternto be empty, so an app-only rule is not read as a server silence:ViewerDataService.IsWholeServerSilence, the fleet bell SQL inDarlingFleetReader, andfleet.js.create_mute_rulegains theclient_app_patternparameter (appended last onCreateMuteRuleOver, so existing callers compile);update_mute_ruleand the web create/update body accept it (sharedBuildMuteRuleUpdate);get_mute_rulesand the created-rule echo report it; the create duplicate check and the update "unchanged" check compare it, so two rules that differ only by client application are two rules. Lite:get_mute_rulesreports it (Lite has no mute write tools).MuteRuleClientAppTests(Darling),MuteRuleClientAppStoreTests(Lite), the viewer column pin, the fleet SQL pin, the tools-list budget (new parameter line; ceiling 197,466 -> 197,621 bytes, measured).Depends on Lane B (not in this commit)
I coded against the exact column names
config_mute_rules.client_app_pattern(Postgres, V179) and DuckDBconfig_mute_rules.client_app_pattern VARCHAR. Lane B had not pushed the DDL when I pushed, so on this commit alone:PgMuteRuleStore_RoundTripsClientAppPattern_AndNullForALegacyRule) needs V179 and skips withoutDARLING_TEST_PG;DuckDbMuteRuleStore_RoundTripsClientAppPattern_ThroughInsertUpdateAndClearneeds the DuckDB column, both on a fresh install (Schema.CreateMuteRulesTable) and as an additiveALTERfor existing files, plus the preserved-table restore carrying the column. I ran it green locally with a throwaway one-lineSchema.csedit that I reverted, so it fails until B's DDL merges.ViewerWave3Tests.MuteRuleSql_MatchesTheV3SchemaColumnsskipsclient_app_pattern(it is not in V3); it would be worth asserting the column exists in V179 once B lands.Test plan
Lite.TestsandDarling.Tests: 0 warnings, 0 errors.MuteRuleClientAppTests,ViewerWave3Tests,DarlingFleetReaderTests,MuteRulesBehaviourTests,DarlingMcpAlertToolsTests,MuteRuleServerId*,ViewerServerSilenceTests,ViewerTraySnoozeTests,FleetSilencePageTests,DarlingAlertingTests,DarlingSelfAlertTests: 404 run, 0 failed, 9 skipped (live, noDARLING_TEST_PG).*Census*,*Ratchet*,McpToolsListBudget*,McpPayload*class: all pass after the budget update (528 run).*Census*,*Ratchet*,*Mcp*classes: 812 run, 0 failed;MuteRuleClientAppStoreTestsandMuteRulesSurviveResetTests9 run, 0 failed with the throwaway column.For the coordinator
TotalCeilingBytes); expect a trivial conflict there. My dialog edits are in the pattern grid andMetricCombo_SelectionChangedonly.AlertMuteContext.ClientAppName(the alert producer lane does), and the "Mute this alert" prefill inmute-context.jsdoes not carry a client app for the same reason.create_mute_rule/update_mute_rule, so cancels from one tool (for example SSMS) can be muted without muting an application's own timeouts. A rule that names a client application never mutes an alert that has none.Lane B: collector and store
PerformanceMonitor.Collectors/AttentionCollector.cs(shared by Lite and Darling): reads thePerformanceMonitor_Attentionevent_counter (exact count) and ring_buffer (detail). The delta from the prior count is the number of attentions; a lower count means the session restarted (delta = current count). The prior is kept in the collector state (StateKeys, staged so it lands only after the run's rows are written, per server or per Azure database) rather than in process memory, so a restart does not lose it. First run with no prior: only rows newer than the watermark, no count row. (Superseded by the R1-X count model below: samples never feed the count.) Ring buffer is shredded only when execution_count or the counter moved. Each run writes onecountrow with the delta (R1-X below replaced theoverflowrow). Statement text goes through the PostgreSQL targets: the config collector stores superuser-only settings unredacted, including a standby's replication password #4348 filter, then is cut to 4,000 characters. The fingerprint is a hash of the filtered text, NULL when the filter withheld it. It is SHA-256 cut to 160 bits (40 hex characters), not SHA-1 as the plan said: no SHA-1 is used anywhere in the tree and the weak-algorithm analyzer would fire.RunXeTolerantAsyncrecords SESSION_MISSING. Checked live on sql2019 (throwaway sessionpm_test_attention_b, dropped): missing session throws; 3 sqlcmd timeouts gave counter 3 and 3 ring rows with app, host, database, session, request_id, duration and text; with the prior counts passed back the gate skips the shred.["attention"] = new(1, 30)enabled, Darling map entry throughRunXeTolerantAsync, Capture Down third arm (attentionreads as "Client timeouts"), Lite dispatch (RemoteCollectorService.Attention.cs) and its session-name arm inRemoteCollectorService.DefinitionRunner.cs.StorageVersion179):collect.attention_events,config_mute_rules.client_app_pattern, andclient_timeouts_enabled(false),client_timeouts_warning_per_minute(10),client_timeouts_critical_per_minute(60) onconfig.config_alert_settings. Viewer probe sentinelhasClientTimeoutSettings(ordinal 151) with the top arm returning 179. Lite DuckDB schema 68:config_mute_rules.client_app_patternthrough the added-columns list (the attention table is generated from the catalog).Lite.Tests/AttentionCollectorDefinitionTests.cs(batch pins, delta/restart, count row, first run, fingerprint, truncation, staged state) andDarling.Tests/StatementScrubAttentionTests.cs(statement filter census case, registered asattention.statement_textin both census tables). Existing top-rung probe pins moved by one ordinal (ServiceVersion, AwsPerServerRole, PlanRegressionDaily, LegacyPinCandidate, PlanRegressionRowcount). Passing: the new classes, census classes, SelfAlert, ViewerSchemaVersionGate.CollectorCatalog.AllholdsAttentionCollector, whichTimescaleSupport.HypertableTables(TimescaleSupport.cs:165) turns into the 73rd hypertable.MigrationLadderPinsand the "dense" rung classes) until V176-V178 merge.DarlingCollectorRunner.csis off limits, so its per-databaseAlwaysOnSessionNameswitch (~:2526) has noAttentionCollectorarm (Azure own-session fallback would read the shared name); a Lite schema-68 test is not written.Lane A: the always-on capture (Option A)
Seat-approved plan, Option A: a third always-on session,
PerformanceMonitor_Attention, beside deadlock and blocked-process. No RDS instance was created: the incident replica runs the deadlock and blocked-process sessions of the same server-scoped shape cleanly (observed by the coordinator), and Erik ruled RDS is covered by the client fleet. O = observed, I = inferred.What changes
PerformanceMonitor.Collectors/AlwaysOnXeSessions.cs:AlwaysOnXeSessionKind.Attention. One builder for both products:BuildAttentionServerCreateSql()(ON SERVER,MEMORY_PARTITION_MODE = NONE,STARTUP_STATE = ON, start included),BuildAttentionServerProbeSql(),BuildAttentionServerStartSql(); the Azure per-database statement comes from the existingBuildAzureCreateSql(ON DATABASE, no server-only option), so the existing per-database driver (AlwaysOnXeAzureEnsure) arms it with the shared/own fallback. Eventsqlserver.attention, actions client_app_name, client_hostname, database_name, session_id, sql_text (no query_hash: always 0, O). Targetspackage0.event_counterandpackage0.ring_buffer (SET max_memory = 1024).SharedNameForalready was; the Azure create's event/targets/retention, the catalog probe,ReadOnlyDatabaseMessage). A new kind cannot fall into the deadlock arm. The shared name is the attention collector's own constant (AttentionCollector.XeSessionName), so reader and lifecycle cannot disagree.AttentionAzureRejectedActionsis a closed list (empty today) the Azure DDL is generated from, like [BUG] long_query_completions XE session cannot be created on Azure SQL DB: 'sqlserver.server_principal_name' is not an available action #3753's: a 25744 reject found in the release's Azure run is a one-line edit.DarlingXeSessionsgets a third per-capture try block (on-prem) and an Attention ensure per database on Azure.DarlingXeSessionCleanup(drop list, own names, find-others patterns, stop, capture note) andDarlingRemovedServerSessions(fallback kinds) include it. The Darling README names it in the ensure list, the shared-name bullets and the remove/drop text.RemoteCollectorService.AttentionXeSession.csensures through the shared builder (no third DDL copy), gated on theattentionschedule likeDeadlocks.cs, throwsXeSessionEnsureExceptionthrough the shared server/database arms before the read; called inRemoteCollectorService.csbefore the read switch; removal forgets the Attention own sessions..claude/skills/release-checklist/SKILL.md: the Azure SQL DB run checks the three sessions and notes thatclient_hostnameandsql_texton Azure are unverified (a 25744 reject goes onAttentionAzureRejectedActions).Decision to check
The brief said to gate the Darling ensure on the
attentioncollector "the same way the other two are". Darling's two existing ensures are NOT gated on a collector (EnsureAllAsyncis documented as unconditional, create-only, run once per connect), so the Attention ensure is ungated too. Lite's is gated on the schedule, as Lite's two are. Say so if you want a Darling gate.Live check (sql2019 only; session dropped, 0 left)
Created through
BuildAttentionServerCreateSql()(O): session running; 3 forced timeouts (2 sCommandTimeouton aWAITFOR) gave event_counter = 3 and 3 ring-buffer events carrying app, host, database, spid,duration(343-400 us) and the batch text. A controlCancel()after 1.5 s added 1 (counter 4).DROP EVENT SESSIONleft 0 rows insys.server_event_sessions.SqlClient reader-close question (Microsoft.Data.SqlClient 7.1.1, the version this repo references)
Tests
Darling.Tests/AttentionXeSessionTests(13) andLite.Tests/AttentionXeSessionLiteTests(5): DDL text per scope, switch totality (an undefined kind throws in each switch), names, cleanup/removal lists, Darling third try block pinned, Lite uses the builder with no DDL copy and ensures before the read.DropXeSessionsVerbTests,DropXeSessionsPerInstallTests,AlwaysOnXeReadOnlyIntentTests(kinds), theAlwaysOnXeSessions*theories now include Attention.Lite.Tests.LlmsTxtCollectorCensusTests.LlmsTxtCollectorCount_MatchesTheCatalog("llms.txt reads 42 SQL Server collectors, but the catalog defines 43") comes from Lane B adding theattentioncollector;llms.txt(and any README collector count) needs the sentence updated. I did not touch it, to avoid a conflict with Lane B.For the coordinator
DarlingCollectorRunner.cs(~line 2526, off limits to this lane) hasBlockedProcessReportCollector => AlwaysOnReadSessionName(...); it needsAttentionCollector => AlwaysOnReadSessionName(server, databaseName, AlwaysOnXeSessionKind.Attention)so an Azure database whose install fell back to its own session reads that one (Lite'sDefinitionRunneralready has it).PerformanceMonitor_Attention(theattentionevent, counted exactly and sampled with application, host, database, session and statement text), created by Lite and Darling on SQL Server, Managed Instance, RDS and Azure SQL Database.--drop-xe-sessionsand server removal handle it like the deadlock and blocked-process sessions."sql_textandclient_hostnameon Azure SQL Database (release run); a Web/Express edition refuses the session as it does the other two (I).Lane B2: plain table, retention, gaps, merged-tree tests
Seat ruling (b), applied.
attention_eventsis a plain table on every store;HypertableCountstays 72 and no grid, workflow or runbook number changed.1. Opt-out. New
ICollectorSchemaInfo.StaysPlainTable(default interface memberfalse, plus avirtualtwin onCollectorDefinitionBase, asRequiredPgExtensionsdoes). OnlyAttentionCollectoroverrides it totrue; its doc names the reason (#3035, #3044, #3107; the ring buffer caps detail at about 1,800 events per read).TimescaleSupport.HypertableTablesis now the catalog less opted-out entries (71), so conversion, compression, the phase order, the name list andHypertableCountall stay as they were. Census testAttentionPlainTableTests.TheOptOutCensus_ListsExactlyTheAttentionTablelists the opt-outs (exactlyattention_events).2. Retention and storage.
DarlingRetentionskipsdrop_chunksfor a table that stays plain and runs the 30-day time-sliced DELETE directly, on both store kinds (no drop_chunks failure warning per sweep). V179's index(server_id, collection_time)is unchanged. Live test on TimescaleDB (OnATimescaleStore_TheStartupWalkLeavesItPlain_AndTheSweepPurgesItAtThirtyDays): afterConvertToHypertablesAsyncandApplyCompressionPolicyAsync(both return 71) the table is not intimescaledb_information.hypertables, has no job, keeps its index;PurgeAsyncremoves the 45- and 31-day rows and keeps 29, 1 and 0 days, with no attention warning logged.3. Gaps. (a)
DarlingCollectorRunnerper-databaseAlwaysOnSessionNameswitch has theAttentionCollectorarm (only that switch touched). (b) Live V179 rung test: idempotent, defaults false/10/60,client_app_patternnullable text, index, and an upgrade from a store with the columns and table removed. (c)CaptureDownChunkOrderTestsnow pins threeLIMIT 1arms and twoUNION ALL, and the live plan test seeds anattentionSESSION_MISSING row (answer:Client timeouts,Blocking). (d) Lite schema 68: newClientTimeoutsSchemaV68Tests(ladder text, fresh store, v67 database climbs and an old rule reads NULL). (e) Viewer:ViewerDataService.MuteRulesSelectBeforeClientAppSql(typed NULL for the column) is used when the probed store version is below 179 (MuteRuleClientAppMinSchemaVersion);ViewerWave3SqlTests.MuteRuleSql_MatchesTheV3SchemaColumnsnow asserts the column from V179, and a new test pins the two statements differ only in that entry; the live test shows the gated statement answers and the ungated one fails 42703 on a store below V179.Also fixed on the merged tree.
ScheduleManagerhad noattentionrow, so Lite would never have scheduled the collector andSharedCollectorDefaultsPinTestswas red: added (enabled, 1 minute, 30 days).llms.txtandREADME.md(Lite collector table) say 43 SQL Server collectors, with one line on the attention collector (coordinator request);Darling/README.mdcounts 72 collectors / 43 SQL Server, and the merge conflict in theremove_serversentence with dev kept both sides.CollectorCatalog.Allfor hypertable work now countHypertableTables(TimescaleSupportTests, CompressionPhaseAssignmentTests, RetentionReevaluationTests, the V136/V137-era rung pins);PgSchemaGeneratorTestspins the catalog at 72;ReadmeDerivedCountPinTestsworker-sizing pin compares the 4th figure toHypertableTables.Count; the cagg rung test's probe-sentinel helper drops V179's sentinel.Test results (local, sql2019 not needed; throwaway PostgreSQL with TimescaleDB, stopped)
DarlingOperatorGroupScriptTests.cs:182(not touched here).*Census*,*Ratchet*,*Pin*,*Readme*: 1168 run, only the known dense-ladder reds fail (MigrationLadderPins.TheLadder_IsDenseAboveTheHistoricalGap,CaggGroupIndexDropRungTests.TheRungIsRegistered_AndTheLadderIsDenseAboveTheHistoricalGap: V176-V178 unmerged).StatementScrubAttentionTestsandPgLogEventMetricsTestsname filters matched 0 (class names differ), so not run by name.Double-check. The last commit (
36f195f03: Lite schedule row, README pin, cagg helper) was pushed right at the 60-minute wall; its test runs are above, but the post-merge full CI is the real check.R1-X: count model (review round 1: H1, L2, L3)
Claude posting for Erik Darling.
This section supersedes the Lane B bullets about the
overflowrow and the "21 tests" count above.H1: the count no longer double-counts
The defect: the run read the event_counter before the ring buffer and wrote
overflow = delta - kept. The ring buffer receives an event up to 5 s after the counter counts it, so one attention could be counted as overflow in run N and again as a detail row in run N+1 (also across the two reads, and after a NULL counter read). With no detail rows the overflow row sat at the old watermark.The new model, in four lines (
AttentionCollector.cs; the DDL is unchanged,event_kindis the row-kind column):event_kind = 'count',attention_count= the event_counter delta from the staged prior,event_timeNULL.collection_timeis the run's time (the host stamps it), never a watermark. The alert and any rate readcountrows only. A quiet run writes no row (C0 below).event_kind = 'sample'rows (app, host, database, session, request, duration, scrubbed text, fingerprint, event time) withattention_count0, deduplicated by the existingevent_timewatermark (strict>). They are never summed into a count, and a sample never moves a count.event_timeis NULL, soMAX(event_time)is the newest sample and a count row cannot move it. A lost sample is possible (ring buffer cap, or a tie at the watermark); a lost or doubled count is not.Store shape: nothing changes in V179 or Lite schema 68 (same 11 payload columns), only the values and the V179 comment.
attention_eventsis still a plain table with 30-day retention. The fingerprint is nowQueryHashIdentity.StatementFingerprint(C0 below).Behavior check, superseded by C0 below: the zero-count row per run is gone. A quiet run writes no count row, so the table holds one row per run that saw an attention, not about 43,000 rows per server per 30 days.
Tests (
Lite.Tests/AttentionCollectorDefinitionTests.cs, 24; the test pinning the old overflow placement is gone):collection_timeis the run's time, never the watermark) and its payload carries no time.Lite.Tests/ClientTimeoutsSchemaV68Tests(DuckDB appender) andDarling.Tests/AttentionPlainTableTests(live PostgreSQL binary COPY):collection_time= the run's time for both rows, count rowevent_timeNULL, sum ofattention_count= the delta,MAX(event_time)= the sample's time.L2: the viewer's below-V179 mute read was dead, removed
Confirmed from the code:
MainWindow.xaml.cs(connect, about line 409) refuses any store belowViewerDataService.RequiredStoreSchemaVersion(=StorageVersion.SchemaVersion, 179 or more) before any window opens;GetMuteRulesAsyncis only called from windows opened after that gate (MainWindow.ServerManagement.cs,MuteRulesWindow), and the probe fails open to the current statement.HeadlessSelfTestdoes not read mute rules. No path reads an older store, so I removedMuteRulesSelectBeforeClientAppSql,MuteRuleClientAppMinSchemaVersion, the version branch and its live test (the 42703 skew check inAttentionPlainTableTests). Kept: theViewerWave3assertion that the column exists from V179, and the live check that a rule saved before the rung readsclient_app_patternNULL after it.L3: engine capability prose
["attention"] = "the client timeout and cancel capture"inCollectorEngineCapability.CapturePathByCollector. The existing census (every entry must be gated or asked about by a wired read) would reject an entry whose read (get_client_timeouts, lane E) is not on this branch, soEngineCapabilityMissTestscarries a one-nameReadsPendingWiringset and a test that fails the day a read is wired (the exemption cannot outlive its reason). New censusEveryShredGatedXeCollector_HasACapturePathEntry: every catalog collector withXeShredGatein its state keys (deadlocks, blocked_process_report, attention) must have an entry; that is the check that would have caught the gap.Small in-lane fixes found by the Guard run
EngineCapabilityMissTests: my first insert stacked two summaries (fixed).MuteRuleClientAppTests(already on the branch): added[Trait("Reads","Lite")](it reads Lite paths) and[Collection("live-postgres")](its live test migrates the shared database).RepoFileAdoptionTestsroster: addedAttentionPlainTableTests,AttentionXeSessionTestsandMuteRuleClientAppTests, which read LF-normalized text.Results
xUnit2013inDarlingOperatorGroupScriptTests.cs(not touched). Lite.Tests: 0 warnings.AttentionCollectorDefinitionTestsandClientTimeoutsSchemaV68Tests31 passed.-class "*V179*"/rung classes: only the known dense-ladder pins fail (V176-V178 unmerged; alsoQueryStoreIntervalPartitionRungTests, which counts rungs).DocCommentHygiene,RepoFileAdoptionTests,LiteReadsTraitGuardTests,LivePostgresCollectionHygieneTests). Two stay red, neither from this change:PgCpuCapacityHeadroomTests.TheRungIsRegisteredInADenseLadder(known) andTsqlConventionGuardTests.TheMemberScan_ReadsEveryDeclarationWhole(AlwaysOnXeSessions.cs ReadOnlyDatabaseMessagestops short; that file is R1-Y's, so I leftKnownTruncatedRangesfor them).Proposed CHANGELOG entry (replaces the capture entry's counting claim)
Added: "Client timeouts are captured by an always-on Extended Events session,
PerformanceMonitor_Attention(theattentionevent), created by Lite and Darling on SQL Server, Managed Instance, RDS and Azure SQL Database. Each collection that saw a timeout writes one exact count from the session's event counter (a quiet collection writes none), so a timeout is counted once however the ring buffer lags, and keeps the application, host, database, session and statement text of the recent events as samples that never change the count.--drop-xe-sessionsand server removal handle it like the deadlock and blocked-process sessions."R1-Y: session and fault paths
Review round 1 findings H2, L1, L4, L5, L6 (this part is merged with R1-X's push).
DarlingXeSessions.AttentionFaultFor, the [BUG] Lite: XE permission error 15247 is classified as ERROR instead of PERMISSIONS #5378 permission set).EnsureAllAsyncreturns the fault; the worker keeps it onServerLoopState.AttentionXePermissionFault(the connect path and the hourly ensure replace it), and the attention run raises the existingDarlingXeSessionDeniedException, so the row is PERMISSIONS. Capture Down reads SESSION_MISSING only, so none fires. On Azure the fault is set only when EVERY database refused the create (a partial refusal leaves the survivors' read real, as in [BUG] Azure collector failures are swallowed into status=SUCCESS, and get_collection_health then reports HEALTHY with a finding that says the opposite of what happened #3754). Any other failure, and a session dropped by hand on a login that may create one, are untouched: the ensure recreates it, and until then the read records SESSION_MISSING. No new exception type. Lite does not have H2: its ensure raisesXeSessionEnsureExceptionwith the inner SqlException, whichRunCollectorAsyncalready classifies PERMISSIONS (RemoteCollectorService.cs:912-925). The RDS Web/Express engine refusal is not a permission error number, so it stays SESSION_MISSING (no new status invented).EnsureServerScopedXeSessionAsync, which applies the Lite and Darling log a Warning on every cycle while the long-query session cannot be created #4964 rule (Warn or Error once, then Debug). Tests: the attention session is a third case inDeadlockAndBlockedProcessEnsureLogLevelLiteTests(a repeated SQL error and a repeated permission refusal log once, then Debug), plus a source pin inAttentionXeSessionEnsureLiteTeststhat the three inner methods hold noFailed to start/createtext and noAppLogger.Error. Limit: those log tests replace the open and the ensure, so the removal itself is proved by the source pin.BuildAttentionServerProbeSqlnow returnssession_state0, 1 or 2 (2 = another definition). It compares the oneattentionevent, the five actions (none extra) and theevent_counterandring_buffertargets (none extra), built from the same lists as the DDL. Darling and Lite drop (BuildAttentionServerDropSql) and recreate on 2, logged once, at most once per connect (Darling,ServerRuntime.AttentionXeDefinitionReplaced) or per app run and server (Lite). Azure: the catalog probe returns 2 (usable) only for a matching definition and 1 for another one, which the existing driver treats as unusable, so the install reads its own session. The shared Azure session is never dropped (another install may read it), the one place this differs from "drop and recreate". Live, sql2019: no session gave no row; a session created withoutsql_textgavesession_state2; the builder's session gave 1; dropped afterwards, 0 left.attentionis enabled for the server (StoreConfigProvider.ResolveSchedule("attention", ...).Enabled), on-prem and Azure, at connect and in the hourly pass. Nothing is dropped on disable (the cleanup verb exists). Turning the collector on is ensured at the next sweep, not after the hour (AttentionXeEnsuredWhileEnabled). Deadlock and blocked-process ensures are unchanged. Tests: disabled gives no attention statement and no DROP while the deadlock session is still ensured; enabling within the hour ensures at once; the worker call is pinned.client_hostnameis confirmed: Microsoft's Azure SQL Database profiler templateStandard_Azure(Azure Data Studio, src/sql/workbench/contrib/profiler/browser/profiler.contribution.ts) is anON DATABASEsession whosesqlserver.attentionevent carriesACTION(package0.event_sequence,sqlserver.client_app_name,sqlserver.client_pid,sqlserver.database_id,sqlserver.username,sqlserver.query_hash,sqlserver.session_id,sqlserver.client_hostname).sql_textis NOT confirmed: MS Learn (azure-sql/database/xevent-db-diff-from-svr) lists no per-action table and says "You can obtain available events, actions, and targets using this query" (oversys.dm_xe_objects); no Azure session in this repo or in that template uses it. So there is a newAttentionAzureUnverifiedActions = { sqlserver.sql_text }: an Azure attention CREATE refused with error 25744 is retried once without it, and the capture note is a Warning ("created and started ... without statement text: Azure SQL Database refused an action ... error 25744") fromAlwaysOnXeChange.CreatedWithoutStatementText, in Darling and Lite. The Azure definition probe treatssql_textas optional, so a session created without it is not read as another definition (no flip to the own session).AttentionAzureRejectedActionsstays empty. Test: a scripted database refusingsql_textgets two creates, the second without it and withclient_hostname, and the note.R1-Y results
DarlingOperatorGroupScriptTestsxUnit2013, is on the branch head).PgCpuCapacityHeadroomTests.TheRungIsRegisteredInADenseLadder, V176-V178 unmerged).TsqlConventionGuardTests.KnownTruncatedRangesgainedAlwaysOnXeSessions.cs ReadOnlyDatabaseMessage(the member scan now stops short there after the new probe text).R1-Y: proposed CHANGELOG additions (extend the capture entry)
"A login that cannot create Extended Events sessions records PERMISSIONS for the client-timeout collector instead of SESSION_MISSING, so it no longer fires a permanent Capture Down alert. Darling creates the client-timeout session only while the collector is enabled for the server, and never drops it on disable. A same-named session with an older definition is dropped and created again once. On Azure SQL Database, a create refused for an action is retried once without statement text. Lite no longer repeats a failed Extended Events ensure in its log every minute (client timeouts, deadlocks and blocked processes)."
R1-Y: for the coordinator
sql_textretry is unproven against a real Azure SQL Database; the release run should confirm which path it takes.C0: collector carries
Claude posting for Erik Darling.
origin/dev(V176, Darling: the raw procedure duration trend reads per-collection totals, written with each procedure_stats batch (#5638) #5706) merged into the branch; conflicts were only the ladder rung lists. V179 stays the last migration andStorageVersionis 179. The viewer probe now endshasProcedureStatsTotals(V176, ordinal 151) thenhasClientTimeoutSettings(V179, ordinal 152) with the V179 arm on top; each side's sentinel is kept in rung order. The probe-ordinal pins the merge itself broke were bumped by one (ServiceVersion, AwsPerServerRole, PlanRegressionDaily, PlanRegressionRowcount, LegacyPinCandidate, QueryStoreComposeStamp); the cagg rung test's artifact drop covers both V179 and V176. The dense-ladder pins stay red until V177 (Alerts: a Runnable Queue alert on runnable tasks per scheduler, for Lite and Darling (#5641) #5709) and V178 (Darling: the Query Store watermark read uses a per-batch side table and runs once per pass (#5684) #5707) merge.AttentionCollector.Fingerprint(SHA-256 cut to 40 hex) is deleted, with no wrapper;ReadAsynccallsQueryHashIdentity.StatementFingerprint("text:" plus 16 hex, lower-cased, whitespace-collapsed, NULL for no text and for the withheld marker). Checked: the helper testsSensitiveStatements.PlaceholderTextand the collector's scrub emits exactly that constant, so the marker is the same. The column istextin V179 andVARCHARin Lite v68 (generated from the catalog'sVarchar), so no width change. The old 40-hex test is replaced byTheStoredFingerprint_IsTheSharedQueryHashIdentityFingerprint_NotACollectorOwnHash(the collector's output equals the shared helper's for the same text, in two spellings of one text and a different text) andTheStoredFingerprint_IsNullForNoTextAndForTheWithheldMarker. No store holds rows yet, so the stored grouping key changes for free now.AttentionCollectorhas the single write path for both products (Lite'sRemoteCollectorService.Attention.csruns the shared definition and has no writer of its own), so Lite follows. Doc comments fixed: the class summary,KindCount, the V179 migration comment (comment only; the SQL text is unchanged), Lite's Attention comment. The R1-X text above that said "one count row per run" or "a quiet run writes a count of 0" is corrected in place. Tests:ReadAsync_QuietCycle_WritesNoCountRow_StagesTheCounts_AndMeasuresGated(was the zero-row case) and a newQuietRuns_WriteNoCountRows_AndAReset_OrAnAttention_WritesItsRow; the six existing count-model cases pass unchanged. The engine must judge freshness fromcollection_log, not from zero rows; no read was built here.AlwaysOnXeSessions.cs ReadOnlyDatabaseMessage). The member was an expression-bodied concatenation that opens with a bracedswitch, so the member scan closed it at the switch's brace and stranded four literals (the prose of the message and the error-number formatter). Readers of that file: the member-map censuses (TsqlConventionGuardTests,TempDbReservedLabelProvenanceTests,StoreApplicationNameCensusTests,StoreSessionTimeZonePinCensusTests,StoreSqlClockDisciplineTests, plus the Darling-tree ones) and Lite's source-text pins, which name members and not literals. The stranded text is English prose with no SQL keyword, tempdb label, application name, zone name or clock call, so nothing counted was hidden. To prove it rather than argue it, the member is now a block body with the switch in a local, the walk reads it whole, theKnownTruncatedRangesline is removed, and every member-map census passed with its pinned counts unchanged.CollectorStateContractTests(attention declares state),ConsumedTimestampFrameDisciplineTests(71 timestamp columns, 52 SqlServer, 15 Utc,attention_events.event_time=Utcin the sequence; the T-SQL reads the XE event's own@timestamp).ViewerCollectorCoverageTests.EveryCollectorTable_HasAViewerReader_OrIsAllowListed(attention_eventshas no viewer reader and no allow-list entry) andDarlingMcpAlertToolsSurfaceAndSqlTests.ParamContract_MatchesContract(create_mute_rule)(the contract CSV hasreasonwhere the tool now hasclient_app_pattern). Plus the known dense-ladder pins (31 tests).Tests:
Darling.Tests0 warnings,Lite.Tests0 warnings. Attention*, AlwaysOnXe*, QueryHashIdentity*, CaptureDown*, ClientTimeouts* and the Rung classes: only the dense-ladder pins failed. FullDarling.Testswithout PostgreSQL: 31,736 run, 39 failed before the census-pin fixes (31 dense-ladder or rung pins waiting on V177 and V178, the two still-red tests above, one timing flakeSensitiveStatementsTests.TheSecondJudgePart...that passed on rerun, and the five fixed here); the fixed classes were rerun green (118 tests, 0 failed).Lite.Tests:-trait Stage=Guard496 passed, Census and Ratchet classes 67 passed,AttentionCollectorDefinitionTests28 passed. No PostgreSQL was started.Lane C: alert engine
IAlertReadAdapter.GetAttentionWindowAsync(serverKey, lookbackMinutes, ct)inDarlingAlertReadAdapter(PostgreSQL, one-retry seam named "client timeouts") and Lite's adapter (DuckDB,LocalDataService.Attention.cs). Both run one shared SQL text (PerformanceMonitor.Common/AttentionWindowSql.cs). Success rule: a run counts as observed when the collection log has a row withcollector_name = 'attention'andstatus = 'SUCCESS'for the server, in both stores. Quiet runs write noattention_eventsrow, so the log is the only freshness signal; any other status is not a run.AttentionEvaluator: rate over the covered span (runs, midpoint boundary, 30 s skew ceiling), the not-observed rule reusing Alerts: a Runnable Queue alert on runnable tasks per scheduler, for Lite and Darling (#5641) #5709's gap helper, proportional mute (the share of sampled events whose group is muted scales the count), Warning 10 and Critical 60 per minute, off by default. The restart edge is documented in the class: the first run after a restart only stages the counter, so that span counts as quiet (under-counts, the safe direction).CheckAttentionAsyncafterCheckRunnableQueueAsync(cooldown, fresh-sample guard, 30 min stale resolve, TierEscalation opt-in, rollback on every-channel failure; corrected in round 2: the tier is recorded for a delivered, unmuted fire only, a Muted fire no longer records it, see "Round 2 fixes, engine (A1)"). Settings plumbing:IAlertEngineSettings,DarlingAlertSettings,StoreConfigProviderseed and read (columns at ordinals 73-75),DarlingConfig,AppAlertEngineSettings, LiteApp.xaml.csload keys;AlertFamily(Performance),AlertSeverity,AlertMetricClassifier("N.N per minute"). The mute context carriesClientAppNameper sampled group. Test fakes updated.create_mute_ruletakesclient_app_patternas its LAST parameter, so the existing order is unchanged and the contract pin is appended (...,server_id,client_app_pattern). Pins updated for the new metric and seam: escalation opt-in set, fire-site counts, read-seam counts (14 reads, 41 counted sites), notification census, V177 probe and rung pins (no longer the top), mute-tool parameter text.Darling.Tests.AttentionAlertTests(27: quiet, storm, covered span, not-observed, mute all/partly/none, Warning to Critical, off by default, engine)Darling.Tests.AttentionWindowReadLiveTests(3, throwaway PostgreSQL, quiet window included)Lite.Tests AttentionWindowReadTests(2, real DuckDB file, quiet window included)-trait Stage=Guard-trait Stage=GuardStill red, expected or handed off:
ViewerCollectorCoverageTests: lane E adds the viewer reader (left red on purpose).PgCpuCapacityHeadroomTests.TheRungIsRegisteredInADenseLadder,MuteRuleReloadBeaconTests.TheRungIsRegisteredAtTheTopOfADenseLadder): red until Darling: the Query Store watermark read uses a per-batch side table and runs once per pass (#5684) #5707 (V178) merges.RunnableQueueSettingsSurfaceTests.TheThreeColumns_...: the service seed now has 76 columns, the viewer upsert still 73. Lane E adds the three columns to the viewer upsert; the test then passes.LivePostgresCollectionHygieneTestswas red for my new live class and is fixed ([Collection("live-postgres")]).Notes and handoff:
Left to lane E: Lite SettingsWindow save lines and UI, the viewer settings upsert, and fire detail items through
AlertContextBuilders(the fire branch has the window's top statements, databases and applications available).Top lists are lookback-wide (60 min), not window-wide. The restart edge under-counts by design.
Not run: the full AlertEngine* class list was run as part of the Darling class batch; a full-suite run was not done, as briefed. The playwright browser was not used.
Proposed changelog entry (Added): Client Timeouts alert - a new alert on client timeouts (query attention events), off by default, Warning at 10 and Critical at 60 timeouts per minute, in Lite and Darling. Mutes by client application apply proportionally to the sampled events. A worsening from Warning to Critical is delivered at once.
Lane E2: reads, detail, mute prefill
get_client_timeouts(LiteMcpClientTimeoutTools, DarlingDarlingMcpClientTimeoutTools). One shared judgement,PerformanceMonitor.Alerting/ClientTimeoutsRead.cs, over the one shared SQL (AttentionWindowSql, which gained an opt-inboundedceiling$3for as_of and aSamplesstatement; the alert's own text is unchanged).AttentionEvaluator.ReadSpanis the alert'sReadfreshness and span rule over the whole range (the two share one privateTake). Answers: data (exactcount,rate_per_minute,minutes_covered, top statements with one filtered sample text, top databases, top client apps, newest sampled events, and asamples_notesaying samples are not the total),empty(collector ran, counter did not move),unavailable(no fresh run, or fewer than two),not_collected(PostgreSQL target, through the existing engine-capability question),precondition(the collector's latest run recorded a missing or denied session). Registered in both hosts; counts moved to Lite 91 (89 read) and Darling 182 (89 shared) in README, llms.txt and both instructions; tools/list budget files and ceilings added in both SKUs. TheReadsPendingWiringexemption is deleted fromEngineCapabilityMissTests, replaced by a test thatattentionis reached by a shipped read; the census passes on its own.ViewerCollectorCoverageTestsalready has an allow-list with reasons (KnownStoreOnlyOrUnbuiltTables, precedentpg_database_size_stats). Addedattention_events, naming the alert and the MCP read. (Round 2 correction: an earlier version also named Client Timeouts alert: set the default bars from a week of field capture, then turn it on #5736 as a client-timeouts grid tracker, which was false; no issue tracks a grid and none is planned. See Round 2 fixes, dialogs and UI (B).) Lite'sCollectorViewerCoverageTestsneeded no change (passes).AlertContextBuilders.BuildClientTimeoutItems(finding, window): rate, exact count, minutes covered, the "estimated from N sampled events" line only when the mute proportion applied, top 5 statements (sample text throughTruncateStatement, the statement filter), top databases and client applications. Added at the fire site inCheckAttentionAsync. No census requires an authored notebook per metric (the AlertNotebook classes pass; the mechanical conversion applies), so no Client Timeouts notebook was added.AlertMuteContext.PopulateFromDetailTextnow parsesClient Application:(first wins), which both desktop dialogs seed fromClientAppName(Round 2 correction: the dialogs had no Client Timeouts metric item and hid the Database box, so the database was dropped on save and the rule differed from the web page's; fixed in Round 2 fixes, dialogs and UI (B)); the webmute-context.jsparses the same label and sendsclient_app_pattern.ClientTimeoutsReadTests(4, real DuckDB: storm exact count and rate, quiet window, stale collector, as_of ceiling)ClientTimeoutAlertDetailTests(6: detail items, estimate line, top 5 cap, mute prefill of app and database, web prefill pin, classification)EngineCapability*,McpToolsListBudget*,McpPayloadContract*,ViewerCollectorCoverage*,Attention*,AlertContext*,AlertNotebook*,Mute*, Census, Ratchet (1235)MuteRuleReloadBeaconTestsdense ladder (#5707, known)-trait Stage=Guard(1595)PgCpuCapacityHeadroomTestsdense ladder (#5707, known)-trait Stage=Guard(498)Both test projects build with 0 warnings.
Not done, for the next lane: no live PostgreSQL test for the Darling tool (a quiet window, a stale collector and a PostgreSQL target through
DarlingMcpClientTimeoutTools); the shared judgement is covered on DuckDB, and the Darling store path (ReadRangeAsync) is only compiled, not run. Noplaywright-cliuse. No MCP page-contract or tool-guide head entry was added for the new tool.Lane E1: settings surfaces
Lite/Windows/SettingsWindow.xaml, with load, save, reset-to-defaults, preview-line and master-switch lines in the code-behind. The save goes through the newClientTimeoutsBarsInput(twin ofRunnableQueueBarsInput): Warning at least 1, Critical never below Warning, a bad pair refused with a message and applied both-or-neither. The three keys are written by the window, loaded byApp, and added tosettings.sample.json(alert off). Client Timeouts is on the red (Critical bar) and orange (Warning) legend lines in both SKUs, the linesAlertSeverity.ForMetricreturns for it.AlertLegendParityTestsnow runs for both metrics in both SKUs and pins the legend name toAttentionEvaluator.MetricName.SettingsWindow.xamland code-behind, the legend lines, and the three columns in the viewer's select, upsert ($74to$76, ordinals 73 to 75),AlertSettingsRow(default off, 10, 60) andValueEquals. The bars go throughAttentionEvaluator.ClampBarson save, and a bad pair is refused whatever the checkbox says (as Alerts: a Runnable Queue alert on runnable tasks per scheduler, for Lite and Darling (#5641) #5709 did). No per-version gate:ViewerDataService.RequiredStoreSchemaVersionisStorageVersion.SchemaVersionandMainWindowrefuses an older store at connect, as for V177.RunnableQueueSettingsSurfaceTests.TheThreeColumns_...was red on the width; it is now green. I moved the row-width totals (76) to the newClientTimeoutsSettingsSurfaceTests, because the newest append owns the total, and renamed the old method (noOfA73ColumnRowsuffix).client_timeoutsgroup (admin.jslabel, section list, field list).get_alert_settingsreportsclient_timeouts(enabled,warning_per_minute,critical_per_minute) throughAttentionEvaluator.ClampBarsin both Lite and Darling (the Darling reader gains the three columns at ordinals 73 to 75). Darlingupdate_alert_settingsaccepts the group with floor 1, refuses a crossed pair in one body, and refuses a lone bar that would cross the stored other inside the write transaction. No tool signature changed:get_alert_settingstakes no input andupdate_alert_settingstakes onlysettings_json, so there is no new MCP parameter and no ParamContract pin to append (the existing pins still pass). The tool descriptions say the alert is off by default.llms.txthas no alert-settings list (line 8 already names the alert), so it needed no change.README.mdhas a Client timeouts row in the alert types table.Darling/README.mdhas the threeclientTimeouts*settings rows beside Runnable Queue, a row in the engine table, andClient Timeoutsin theperformancecategory list. All say the alert is off by default because no measured normal rate exists yet, and Client Timeouts alert: set the default bars from a week of field capture, then turn it on #5736 sets the bars from field data. The plain-English checker ran on a copy of the new rows; its only remaining hits are the semicolons and the long sentence inherited from the Runnable Queue row style.darling.sample.jsoncomment lists the new default.Darling.TestsClientTimeoutsSettingsSurfaceTests(9: defaults,ValueEquals, 76-column ordinals on viewer/service/MCP/bind, clamp theory, window pins)Lite.TestsClientTimeoutsSettingsSurfaceTests(19: save rules, file keys, sample file off, window controls, engine clamp over statics, MCP group, description)AlertLegendParityTests(both SKUs, both metrics, in both projects)DarlingMcpAlertToolsTests.ClientTimeouts_IsReportedThroughTheClamp_AndWrittenWithTheEnginesValidation,AdminPageTests.ThePage_DrawsTheClientTimeoutsGroup_WithItsThreeFieldsRunnableQueueSettings*,AdminPage*,DarlingMcpAlertTools*,McpAlert*,Attention*,ClientTimeout*,Settings*,AlertSettings*, Census (548), Ratchet (15)-trait Stage=Guard: Darling 1592 run, 1 red; Lite 501 run, 0 redPgCpuCapacityHeadroomTests.TheRungIsRegisteredInADenseLadder(waits for #5707, V178), as lane C notedNot run: the live lone-bar arm of
update_alert_settings(a PostgreSQL class likeUpdateAlertSettings_RunnableQueueBars_...) and a live viewer upsert round trip against a real store; no PostgreSQL rig was started. The lone-bar check is a copy of the Runnable Queue transaction code, but no live test covers it yet. The full suites were not run, as briefed. The Settings windows were not opened in a browser or on screen (WPF; the pins are source and XAML checks).Lane F: live read tests, reading guide
get_client_timeouts(Darling/Darling.Tests/ClientTimeoutsToolLiveTests.cs, a throwaway PostgreSQL with TimescaleDB, UTC). They call the real tool from the server resolver to the serialized answer over seededattention_eventsandcollection_logrows, soReadRangeAsyncnow runs. They found two defects, both fixed:as_oflanding between a run's log time and its rows' time dropped that run's rows. The collection log stamps a run when it starts; its rows carry the time the runner took a moment later (a few seconds). The bounded read cut the rows at the range end, so a run counted as present but without its rows was an undercount (16 instead of 20 in the test), while the alert's ownReadSpanallowsRowSkewSeconds(30). Fix in both SKUs: only the run list stops at the range end; the count and sample-group reads get the end plusAttentionEvaluator.RowSkewSeconds, andReadSpanstill cuts the rows at the newest run plus the skew. (Correction after review round 2, L3: this part originally also gave the top-list and sample reads the allowance, and the review showed that leaked the base run's samples and rows past the span into them. They are now read over exactly the counted span; see "Round 2 fixes, reads (A2)".) Lite got the same fix (LocalDataService.Attention.cs) with a DuckDB test.sampled_events_returnedsaidlimit + 1when the page was truncated (the reader fetches one extra event as the truncation signal, and the payload reported the fetched count while the array heldlimit). NowMath.Min(samples.Count, limit)in the sharedClientTimeoutsRead.Payload. The boundary pair is tested on both SKUs (limit equal to the sample count: not truncated; one below: truncated).<<GUIDE>>plus a tail served byget_tool_guide: the count is exact (event_counter delta per run), samples are a ring-buffer subset (1,000 events), the rate covers only the minutes the collector covered, whatempty,unavailable,not_collectedandpreconditionmean here, and that an attention is a client cancel or timeout, so a high rate is a client-side symptom and the top statements and their waits (get_wait_stats,get_waiting_tasks,get_active_queries) are the next look. Head pins are in the newMcpToolGuideHeads.ClientTimeouts.csin both test projects (head at most 620 characters, points to the guide, keeps the status words, long-form text stays off the head).not_collected = engine without the collector(Lite already answersnot_collectedthrough the engine-capability question), and Darling'sPostgreSQL targetbecame the same words. Thetools/listbudget moved on purpose: the tool line 394 to 422 (Darling) and 359 to 422 (Lite), which is the 31-byte guide pointer plus the status words; total ceilings 198,604 to 198,632 (Darling) and 96,170 to 96,233 (Lite), each with a comment.get_client_timeoutswas NOT added to thePagedToolstuple lists. That census matches the tool body forvar truncated = x.Count > limit;, and this tool's truncation lives in the sharedClientTimeoutsRead.Payload(one place for both SKUs), so adding it would mean moving that code into both tools. Instead each SKU'sMcpPageContractTestshas its own fact: the description says samples are a subset, thelimitdescription says it never cuts the count or top lists, the cap is observed (sampleLimit + 1fetch,Take(limit),sampled_events_truncated), and the page count is*_returned, nevertotal_*.ClientTimeoutsSettingsLiveTests.cs, same rig).update_alert_settingswith theclient_timeoutsgroup: a valid enabled/5/20 saves and echoes; a lone Critical below the stored Warning and a lone Warning above the stored Critical are refused with nothing written; a crossing pair in one body is refused; a pair that moves both across the old one saves; a bar below 1 is refused; a pair stored crossed by a direct write is echoed through the engine's clamp (30/30). The viewer'sconfig_alert_settingsselect and upsert (ordinals 73 to 75): off/10/60 written back unchanged leaves every column equal (ValueEquals), a changed pair (on/7/33) reads back with the other columns untouched, and the bind's clamp stores 0/0 as 1/1. The original three values are restored. Both passed on the first run (no defect found there).ClientTimeoutsToolLiveTests(7: quiet, busy with the limit boundary pair, stale, single run, PostgreSQL target, range cuts a span vsReadSpanfor hours_back 1 and 2 andas_of,as_ofjust after a run)ClientTimeoutsSettingsLiveTests(2)ClientTimeoutsReadTests(+2: sample limit boundary,as_ofrow allowance)ClientTimeout*,Attention*,McpPageContract*,ToolGuide*,McpTool*(354)-trait Stage=Guard(1598)PgCpuCapacityHeadroomTestsdense ladder (#5707, V178, known)-trait Stage=Guard(501), Census and Ratchet (67)The PostgreSQL rig is stopped. Not run: the whole suites; no
playwright-cliuse. Known gap that remains: anas_ofread where two collector runs are closer than 30 seconds could count a later run's early rows (the alert's own rule has the same allowance, and the attention collector runs about a minute apart).Round 2 fixes, engine (A1)
Claude posting for Erik Darling.
CheckAttentionAsyncnow gives a Muted finding its own cooldown stamp and run marker (_lastAttentionMutedAlert,_lastAttentionMutedRunTime, same cooldown length). A Muted finding never stamps the delivery cooldown, never records an escalation tier and is never an escalation. It still writes its Muted history row, once per cooldown. An unmuted fire right after a Muted one goes out at once, and a later Warning-to-Critical worsening keeps its bypass. Tests (AttentionAlertTests, through the shared harness):Engine_MutedCritical_DoesNotHoldBackTheNextUnmutedCritical,Engine_MutedCritical_ThenUnmutedWarning_ThenCritical_KeepsTheEscalationBypass,Engine_MutedStorm_WritesOneMutedRowPerCooldown_NotOneAMinute(11 minutes of a muted storm, 3 rows).CheckRunnableQueueAsync(Alerts: a Runnable Queue alert on runnable tasks per scheduler, for Lite and Darling (#5641) #5709) has the same shape, but its mute changes only when a user edits a rule, so it is left alone.AttentionEvaluator.Evaluatenow asksisGroupMuted(null, null)when the span kept no samples; if muted, the finding is Muted at the full count's grade.MuteRule.Matches(PerformanceMonitor.Notifications/MuteRule.cs) returns false for a null database or app when the rule has a pattern, so only server-wide rules apply (confirmed). Tests: the old pin becameMute_WithNoSamplesInTheSpan_ServerWideMuteStillMutesAtTheFullCountsGrade; newMute_WithNoSamplesInTheSpan_RuleWithADatabaseOrAppPatternDoesNotMuteand..._NoMuteJudgesTheFullCount.AttentionEvaluator.SpanBoundary/TopListFloor, the same midpointReaduses to cut rows); when the span kept no samples they fall back to the lookback andAttentionWindow.TopsFromLookback(new optional last parameter) is true. The card then says "in the last 60 minutes" in the headings and labels the lines "Recent Database" and "Recent Client Application", so the mute dialogs and the web prefill take the span's top app and database only, never an hour-old one. Contract change is engine-side only:AttentionWindow, both alert adapters (DarlingAlertReadAdapter,LocalDataService.GetAttentionWindowAsync); the shared SQL text and the MCP range read are untouched. Tests:AttentionAlertTestsTopListFloor_*(3),ClientTimeoutAlertDetailTests(span headings stay plain; lookback headings and no prefill),AttentionWindowReadTests(Lite, DuckDB) andAttentionWindowReadLiveTests(PostgreSQL) each gainedTheTopLists_CoverTheSpansSamples_NotTheWholeLookbackandASpanThatKeptNoSamples_FallsBackToTheLookbackLists_AndSaysSo. The "estimated from N sampled events" line is unchanged.BuildRunnableQueueRequestItems;BuildClientTimeoutItemshas its own. Also fixed the stale "Runnable Queue is the only opted-in alert" field comment inAlertEngine.cs.Run: Darling.Tests and Lite.Tests build with 0 warnings. Darling: Attention*, ClientTimeout*, AlertEngine*, RunnableQueue*, AlertContext*, MuteRule*, IncidentCooldown*, Census* and Ratchet* classes (1274 tests),
-trait Stage=Guard(1600), plus the live attention classes on a throwaway PostgreSQL (55 pass, stopped). Lite: the same class set (270), Stage=Guard (501), the two attention read classes (10): all pass. Two Darling failures, both ladder pins on this branch head and not from this change (lane S's dev merge owns them):MuteRuleReloadBeaconTests.TheRungIsRegisteredAtTheTopOfADenseLadderandPgCpuCapacityHeadroomTests.TheRungIsRegisteredInADenseLadder(expected top rung 178, actual 179). Not run: whole suites. Not shown: reverting the engine change to watch the H1 tests fail.Dev merge after #5707
Merged
origin/dev(V176 #5706, V177 #5709, V178 #5707collect.query_store_watermarks) into this branch. #5714's rung stays V179 on top;StorageVersion.SchemaVersionis 179. The dense ladder is now 176-179 and its pins are green.Conflicts and resolutions (8 files, both sides kept)
StorageVersion.cs: 179 kept.PgMigrations.cs:Migration(178, "query-store-watermarks")thenMigration(179, "client-timeouts-alert").ViewerDataService.cs: probe SQL gets V178's table sentinel (ordinal 153) then V179's column sentinel (ordinal 154, last);MapProbedSchemaVersiongainshasQueryStoreWatermarksthenhasClientTimeoutSettings(last);GetBooleanlist ends at 154; arms newest first: V179, V178, V177, V176. The V178 arm's "TOP rung" prose was rewritten.CaggGroupIndexDropLiveTests.cs: both rewind blocks kept (V179 then V178 helpers fromRungRewindArtifacts, which already had all three).AwsPerServerRoleRungTests,PlanRegressionDailyRungTests,PlanRegressionRowcountRungTests,RunnableQueueSettingsRungTests: comments and parameter-name pins merged to the 154-ordinal shape.Pins bumped (several merged silently, so only the run found them)
withoutLateralso clears +4), RunnableQueueSettings (+2, adds a without-V179 step that answers 178), ServiceVersion, LegacyPinCandidate, QueryStoreComposeStamp.QueryStoreWatermarksRungTests: last three versions 177/178/179, V178 is no longer the top; probe and live tests now rewind V179 first (DropV179SentinelAsync).QueryStoreIntervalPartitionRungTests: nine rungs 171-179, migrate count 9,Scripts[^8]/[^9], V179 sentinel dropped inRewindToV170Async.attention_eventswas added by schema v68 but the frozen oracleGoldenCollectorSchema(table and index), its count 42 to 43, andDuckDbSchemaTeststable count 55 to 56 were not extended. Done by hand, as the oracle's own comment requires.Tests (Darling.Tests on a throwaway PostgreSQL, Lite.Tests; no whole suites; both projects build with 0 warnings)
*Rung*,*Ladder*,*StoreTests*,*MuteRule*, Attention, ClientTimeout, AlwaysOnXe, CaptureDown, AlertLegend, ViewerCollectorCoverage, QueryStore* probe readers, CollectionLog, PgLogEvent and the other classes that read the probeQueryStoreComposeStampafter fix-trait Stage=Guard(Darling)*Census*+*Ratchet*(Darling)-trait Stage=Guard(Lite)*DuckDbSchema*,*ClientTimeoutsSchema*,*Attention*after fixNot run: whole suites (CI is the proof). After the final two pushes,
origin/devand the PR branch each moved once more (clean merges, no conflicts); the Darling classes were not rerun on those last merges.Round 2 fixes, dialogs and UI (B)
Claude posting for Erik Darling.
One line per finding, with its fix. Part numbers are the brief's. Pushed to this branch from
fix/5640-b.Part 1, the mute dialogs (R2 M2, UI Medium 1)
Lite/Windows/MuteRuleDialog.xamlandDarling/PerformanceMonitor.Darling.Viewer/MuteRuleEditDialog.xamlhad no Client Timeouts item, so the Client Application box could not be reached by choosing a metric andSelectMetricstayed on "(any)" for an alert seed. Fix: an item after "Runnable Queue",Content="{x:Static n:AlertFamily.ClientTimeoutsMetric}"(the one declaration the engine's evaluator also uses), so there is no new literal in XAML or code.showDatabasedid not list the metric, so the Database box was hidden and the save dropped the pre-fill. Fix:showDatabaseandshowClientAppin both dialogs nameAlertFamily.ClientTimeoutsMetric(the old"Client Timeouts"literal inshowClientAppis gone). A Client Timeouts alert now shows both boxes, pre-fills both and saves database + app.mute-context.js(mutePrefillParams) sendsdatabase_pattern: d.databaseandclient_app_pattern: d.clientAppfor the alert, andmute-rules.jshas both as editable fields. The same "Mute this alert" click now makes the same rule (database + app) in Lite, the viewer and the web page.MuteDialogClientTimeoutsParityTests(Darling.Tests, 4: metric item from the constant after Runnable Queue in both dialogs, both gates and the prefill/save lines in both, the saved database + app rule mutes that alert and not another database or app, the web page's two patterns) andTheMuteDialog_OffersClientTimeouts_ShowingDatabaseAndClientApp_FromTheSharedMetricConstant(Lite.Tests). No census lists the dialogs' metrics, so there was none to extend. The dialogs are WPF windows, so these are source pins plus aMuteRule.MatchesAtcheck of the rule they save, not a window instance.Part 2, the Settings rows (UI Medium 2, 3 and the Lows)
VerticalAlignment="Center".sqlserver.attentionevent, and the box tooltips say "client timeouts and cancels (attentions) per minute".SettingsRowParityTests(Darling.Tests, 10: the eight label/tooltip attributes equal across the two files, the unit said once and no "attention" in the visible label, the viewer checkbox centred). Not checked visually: the Runnable Queue label is now as long in Lite as it already was in the viewer, at the same 750 px window; I did not run the app.Part 3 (R2 L2)
ViewerCollectorCoverageTestsno longer claims a client-timeouts grid is tracked in Client Timeouts alert: set the default bars from a week of field capture, then turn it on #5736 (Client Timeouts alert: set the default bars from a week of field capture, then turn it on #5736 is the default bars). The reason now says the table is read by the Client Timeouts alert andget_client_timeouts, neither app has a grid, and none is planned.Corrections to the Lane E2 section above (the review refuted them)
Results
Mute*,Settings*,AlertLegend*,ViewerCollectorCoverage*,XamlStaticResource*, Census, Ratchet,RunnableQueue*,ClientTimeouts*(1080 before the last merges),-trait Stage=Guard(1598),RepoFileAdoptionTestsandLiteReadsTraitGuardTests(both caught my first version of the new file and are green): all green except the two dense-ladder rung pins (MuteRuleReloadBeaconTests,PgCpuCapacityHeadroomTests), which are lane S's rung work and failed the same way before my change. Lite.Tests: the same class set (308) and-trait Stage=Guard(501) green.Mute*,Settings*,AlertLegend*,ViewerCollectorCoverage*,RepoFileAdoption*,LiteReadsTrait*(Darling 501, 0 failed) and LiteClientTimeoutsSettings*/Mute*(103, 0 failed); the very last fetch/merge (3 files) was pushed without a further run.Round 2 fixes, reads (A2)
Claude posting for Erik Darling. Review round 2, findings L3, L4 and L5 (the read-side ones), checked against the code first. Pushed to this branch only; dev not merged by this lane.
as_ofallowance leaked into the top lists and the sampled events: fixed in shared code, both SKUs. The top lists and the samples were read fromwindowStarttoas_of + 30 s, wider than the spanReadSpancounts, so they could hold the base run's samples from before the span and rows past the newest run, and sum pastsampled_events_in_range.AttentionEvaluator.RowSpannow returns the span's exclusive boundary and the newest run plus the skew (the same two valuesTakecuts with),ClientTimeoutsRead.DetailSpanrounds the boundary down to a microsecond, andAttentionWindowSqlreads the top lists andSampleswithcollection_time > $2 AND collection_time <= $3when bounded (newSpanFloor; the alert's unbounded reads keep>=). Both SKUs read runs, counts and groups first, then the detail over the span; with fewer than two runs the answer is "not observed" and the detail is not read. Tests: LiteClientTimeoutsReadTests.AnAsOf_DoesNotLetTheRowAllowanceLeakIntoTheTopListsOrTheSampledEvents(red when the old range is restored, checked), DarlingClientTimeoutsToolLiveTestswith the same name (throughReadRangeAsyncand the tool).NpgsqlCommandinReadRangeAsync(it runs all seven statements) now setsCommandTimeout = McpCommandDeadlines.ReadSeconds. The census already caught it:McpReadCommandTimeoutTests.EveryMcpAndWebReadCommand_SetsAnExplicitDeadlinefailed onDarlingMcpClientTimeoutTools.cs:113before the fix and passes now, so no census change was needed (it is a PR-branch failure that CI would have shown, not a census hole).AttentionWindowSql:Samplesbreaks ties onattention_event_id DESC(the PostgreSQL column isNOT NULL, the DuckDB one is the primary key); the sample text per fingerprint orders bycollection_time DESC, event_time DESC NULLS LAST, attention_event_id DESCinsidearray_agg; count ties break byCOLLATE "C"on the fingerprint and name columns. DuckDB acceptsCOLLATE "C"in the same shared text (tested) and its default is already binary, so both stores list the same fifth entry; other shared SQL does the same (COLLATE "C"inDatabaseNameHistoryReader,ViewerDataService.FileIo). Tests, one per store: LiteTies_AreOrderedTheSameWayInBothStores_BinaryNamesAndNewestIdFirstand DarlingClientTimeoutsToolLiveTestswith the same name (the rig's collation is English_United States, whereapp_asorts beforeApp_b, so the PostgreSQL test fails without theCOLLATE).get_client_timeoutswas missing from both read-only tool lists (DarlingMcpToolClassification,McpToolClassification), and so the Darling instructions' "Default 159" became "Default 160".ClientTimeout*,Attention*,Mcp*,Census,Ratchetclasses (2693)-trait Stage=Guard(1600)PgCpuCapacityHeadroomTests.TheRungIsRegisteredInADenseLadder(the rung pins, lane S's)McpReadToolBudgetTests.EveryReadToolAnswersUnderTheDefaultBudget_UnlessListedAsA4198Offender(get_collection_healthanswers 33,084 B on the Lite seed, over the default budget; not touched by this lane, the Lite collector list grew with the attention collector)-trait Stage=Guard(501)The PostgreSQL rig is stopped. Not run: the whole suites; no
playwright-cliuse.Last reds (T)
Lite
McpReadToolBudgetTests(get_collection_health, 33,084 B). Built the payload the test builds, at this head and at origin/dev (same test code in both):Cause: the new
attentioncollector is the 43rd row, +512 B (one partial_detail row, 514 B compact; sweep_pressure also grew 13 B). No description or note grew (output_note,collector_detail_note,alert_read_healthare byte-identical). The row has the same fields as its 42 neighbours, so there is nothing to trim. This fixture seeds mixed SUCCESS/ERROR history for every collector so none can compact, which makes it a worst case. Fix:get_collection_healthis listed inExemptOffenders(33,084) with a comment giving the cause, the sizes and #5640.CollectionHealthPayloadBudgetToolTests(the 80% ceiling on a realistic mix) still passes.Darling's twin
McpReadToolBudgetLiveTestsis not over (its fixture is the PostgreSQL collector set); it andCollectionHealthPayloadBudgetLiveTestspass (4 of 4) on a throwaway PostgreSQL, so Darling needed no change.Classes run at 8caf01f (head plus this commit), none failing:
*Rung*,*Ladder*,*Schema*,*Mcp*,*Census*,*Ratchet*-trait Stage=Guard*Rung*,*Ladder*-trait Stage=Guard*Census*,*Ratchet*The one red,
StatementCollectionCensusTests.ABlockingRingBufferOfAboutFourMegabytes_IsJudgedInTimeLinearInItsSize, is a timing test; it failed during a loaded laptop run and passed alone (1 of 1). The rung/ladder pin is green at the head. Build: 0 warnings, 0 errors, both projects. No whole suites run; CI decides.Final round fixes (U)
Claude posting for Erik Darling.
Each new test was run red on the code before its fix (fix reverted locally, run, restored). The L3 test is the exception: it pins a rule the code already met, so it was green first; it goes red if the span lists are relabelled "Recent ...".
AlertEngineno longer callsAfterFirefor a Muted Client Timeouts finding (it did nothing else a Muted fire needs: its own stamp and run marker are set at the call site). TestAttentionAlertTests.Engine_AMutedFire_LeavesTheUnmutedFailedSendBackoffAlone: unmuted fail, unmuted fail, Muted fire, unmuted fail waits the third failure's 4 minutes (red: it retried after 1).AttentionEvaluator.TopListFloornow asks> boundary AND <= newest + RowSkewSeconds(newTopListCeiling), and both alert adapters (LiteLocalDataService.GetAttentionWindowAsync, DarlingDarlingAlertReadAdapter) bind the three top readsbounded: truewith(topFloor, newest + RowSkewSeconds), the same shape the MCP read uses. The lookback fallback has the ceiling too. Tests:AttentionAlertTests.TopListFloor_APastTheCeilingSampleIsNotASpanSample_...(unit), and per storeARowPastTheNewestRunPlusSkew_IsInNeitherTheCountNorTheCardsListsandASpanWhoseOnlySamplesArePastTheCeiling_StillFallsBackToTheLookbackLists_WithoutTheminLite.Tests/AttentionWindowReadTestsandAttentionWindowReadLiveTests(throwaway PostgreSQL). The>=to>change at the lookback floor itself is immaterial (an instant an hour back).ClientTimeoutAlertDetailTests.ASpanWithSamples_ListsAreTheSpans_HeadingsStayPlain_AndThePrefillTakesTheirTopEntriesnow assertsPopulateFromDetailTexttakes the top database ("sales") and app ("etl-job"); the lookback-fallback test already asserts it takes neither.ViewerDataService.csnow says V179 is the top rung (comment text only).ForegroundBrushand Margin 6. The viewer's other threshold labels use Margins 2, 4 and 24, so there is no common muted-brush-and-8 rule to match, and the ruling's exception does not apply. Pinned bySettingsRowParityTests.TheCriticalAtText_OfTheRunnableQueueAndClientTimeoutsRows_HasTheSameBrushAndGapInBothWindows.ExemptOffenderscomment inLite.Tests/McpReadToolBudgetTests.csgained the sentence "get_collection_health: the worst-case answer grows ~512 B per collector and passed the 32 KB default budget at 43 collectors #5748 tracks keeping the worst case under the budget as collectors are added." Comment only.Ran after merging the branch head (8caf01f), 0 failed, 0 skipped: Darling.Tests, the Attention / ClientTimeout / AlertEngine / FailedSend / Backoff / AlertContext / MuteRule / AlertMuteContext / SettingsRowParity / Census / Ratchet classes (1290) with the live attention reads on a throwaway PostgreSQL, and
-trait Stage=Guard(1612); Lite.Tests, the same classes plus McpReadToolBudget (254) and-trait Stage=Guard(501). Both projects build with 0 warnings. No whole suites. The PostgreSQL rig is stopped.CI reds, collector side (V)
AttentionCollectorfiltered the ring buffer withevt.value('(@timestamp)[1]', 'datetime2') > @cutoff_time, which shreds every event before the filter. Fix:evt.exist('@timestamp[. > sql:variable("@cutoff_time")]') = 1, the shape the deadlock, blocked-process, long-query and system-health collectors use. The bound stays strictly greater than the cutoff, as before. The ORDER BY still sorts on the value, after the filter.AttentionCollectorDefinitionTestspinned the old text (> @cutoff_time); it now pins theexist()text.RemoteCollectorService.Deadlocks.csso a failed create propagates to the sharedEnsureServerScopedXeSessionAsync(Lite and Darling log a Warning on every cycle while the long-query session cannot be created #4964, inRemoteCollectorService.BlockedProcessReport.cs). The behaviour moved and still holds: that ensure logs Warn whenSqlServerPermissionErrors.IsPermissionDenied, Error otherwise, and Debug on repeat cycles. Fix: both tests now read the shared ensure, and assert that the deadlock, blocked-process and attention (RemoteCollectorService.AttentionXeSession.cs) files each hand their own session constant to it and, for deadlock and attention, carry nocatch (SqlExceptionorAppLogger.Error("XeSession"of their own. Nothing was lost.attentionnow declares StateKeys (the prior event_counter value). Fix: added"attention"first in the sorted pin, plus a NINE note in the comment.AttentionCollector.cs:105) - cause: three figures for one cap. One figure now everywhere: at most 1,000 events. Basis: the ring_buffer target's documentedmax_event_limitdefault of 1000, "existing events are dropped based on the limit that is first reached: max_event_limit or max_memory" (https://learn.microsoft.com/en-us/previous-versions/sql/sql-server-2012/ff878182(v=sql.110); the current targets page, https://learn.microsoft.com/en-us/sql/relational-databases/extended-events/targets-for-extended-events-in-sql-server, coversMAX_MEMORYand the 4 MB XML cut-off but not this default). The session setsmax_memory = 1024and noMAX_EVENTS_LIMIT, so the count cap is 1,000 and the 1,024 KB cap binds first only when events average over about 1 KB each (a longsql_text). So 1,000 is a ceiling, never a floor. The "about 1,800" figure was not a measurement: it appears in no commit message, doc or test beyond its own mentions, and it would need events of about 580 bytes, which does not fit an event with five actions and a statement; it is dropped. Changed:AttentionCollector.cs(class remarks and theStaysPlainTablesummary),ICollectorSchemaInfo.cs:175, and theAttentionPlainTableTestspinned phrase (1,000 events per read). Theget_client_timeoutsguide tail ("keeps at most 1,000 events", both SKUs) andClientTimeoutsRead.csalready said 1,000 and are unchanged. The plain-table decision stands; only its stated evidence changed. I did not measure on a server (that would create an XE session).GetAttentionWindowAsyncfake is now the two-line=>shape of its siblings, with the stray blank lines and trailing whitespace removed.Verified: Lite.Tests and Darling.Tests build with 0 warnings, 0 errors. Lite.Tests: the 4 failing tests pass;
*Attention*,*Xe*,*Sweep*,*DefinitionTests*(933),*PlainTable*,*LogLevel*,*McpToolGuide*,*Census*,*CollectorState*and-trait Stage=Guard(501) all 0 failed. Darling.Tests: the same filters plus*Ratchet*all 0 failed (Stage=Guard 1612). No whole suites; CI decides.CI reds, Darling read and web (W)
RuntimePreconditionReadWiringTests.EveryWiredRead_NamesARealCollector(both Mcp directories). Cause: not a dead branch. Both tools already ask about"attention", the real catalog name, through a private constCollector; the census scanner read the identifierCollectoritself instead of its value (the capability scanner already resolves such consts). Fix:WiredReadsinRuntimePreconditionMissTests.csresolves aprivate const string X = "..."in the same file. The tools keep the literal const (a type-qualified const would not scan). New per-SKU pinGetClientTimeouts_AsksThePreconditionQuestionAboutTheAttentionCollector(wired set is exactlyAttentionCollector.Instance.Name). Branch behavior pinned per SKU: a session-missing latest run givesprecondition(names the attention collector andALTER ANY EVENT SESSION), a healthy latest run or no log row (disabled, never ran) gives nothing, so the tool's ownunavailablestands. Darling: liveAMissingCaptureSession_OnTheLatestRun_IsAPrecondition_NotTheGenericUnavailable(ran on a throwaway PostgreSQL, 10/10 in the class). Lite:AMissingSession_OnTheLatestRun_IsAPreconditionSentence_AndNoLogRowFallsThroughon real DuckDB.get_client_timeoutswas never wired into the Darling web. Fix inDarlingWebEndpoints.cs: a catalog entry (CatData, server, hours default 1, limit 20, as_of) and a dispatch entry that callsDarlingMcpClientTimeoutTools.GetClientTimeoutsthrough the shared read path;plain-text.jstool-name list gainsget_client_timeouts;WebReadReach.csgainsList("attention")(the reach censusWebReadReachTestsfailed on the new hours read until it had a row). No documented exclusion. The wider*Web*classes (1853 tests) pass.StatementCutSourceScanTests. Cause: the scan matched the receiver namewindow.TopStatementsin.Take(AttentionTopListCap).Takecounts list items (top fingerprints) and cuts no text. Checked:AttentionCollectorruns the statement filter, then fingerprints, then the 4000-char cut at write time, a withheld text has no fingerprint and so no item, and the card prints each text throughTruncateStatement, which judges whole before it cuts. So no text reaches the card unjudged. Fix: a named census exception (PerformanceMonitor.Alerting/AlertContextBuilders.cs,window.topstatements.take, count 1) with that reason, the census's own documented route.AttentionWindowSql.csDISTINCT comment. Verified in both runners: neither fan-out runner (DarlingCollectorRunner.cs,RemoteCollectorService.DefinitionRunner.cs) callsLogCollectionAsync; Darling logs the run's one result (the per-database cost rides as the fan-out rollup on that row), Lite has exactly oneawait LogCollectionAsync(across the service partials. So the bot is right: one blendedcollection_logrow per run, the DISTINCT removes nothing, and the real guarantee is the blended row (per-database rows would collapseAttentionEvaluator.OldestTakenand inflate the rate). The comment now says that. DISTINCT is kept as a free guard against two rows with an identical stamp, and the comment says it is not the guarantee. NewAttentionRunLogCardinalityPinTests(Stage=Guard, Reads=Lite) pins the shape on both runners and the comment.Tests run (no whole suites): the 4 reds, then
*Precondition*,*Web*(1853),*Reach*,*StatementCut*,*SourceScan*,*ClientTimeout*,*Attention*,*Census*,*Ratchet*,-trait Stage=Guard(1616) in Darling.Tests, and*ClientTimeout*,*Attention*,*XeSession*,*Census*,*SourceScan*,-trait Stage=Guard(501) in Lite.Tests: all green after merging the branch at f2cb222; both test projects build with 0 warnings.*Web*Tool*matches no class. The throwaway PostgreSQL is stopped.