Skip to content

Gate the deprecated Dashboard's XE ring-buffer shred on execution_count (#4213) - #4371

Merged
erikdarlingdata merged 4 commits into
devfrom
fix/4213-dashboard-xe-ring-buffer-gate
Sep 26, 2026
Merged

erikdarlingdata merged 4 commits into
devfrom
fix/4213-dashboard-xe-ring-buffer-gate

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Sep 26, 2026 •

Copy link
Copy Markdown
Owner

Closes #4213.

Why

#4200 found that the blocked-process and deadlock collectors convert their whole Extended Events
ring buffer to XML and shred it on every scheduled run, even when almost every run finds nothing
new. #4212 fixed this for Darling and Lite by reading the ring_buffer target's own
execution_count before the cast and skipping the conversion when it has not changed since the
last cycle. #4212 could not reach the deprecated Dashboard's own copy of the same collectors,
which Erik ruled (on #4213) still gets this fix: it is not in release assets since v3.3.0, but
existing v3.2.0 users run these collectors on their monitored servers, and it stays buildable.

What changes

  • install/22_collect_blocked_processes.sql and install/24_collect_deadlock_xml.sql: both the
    Azure SQL DB and server-scoped branches now read xet.execution_count before the cast, compare
    it against the prior cycle's stored value, and only run the TRY_CAST + .nodes() shred when
    the count is unavailable, no prior count is stored, or the count has changed (including gone
    down, which means the session restarted). The rows each collector stores are unchanged; only the
    cast+shred step is now conditional.
  • install/03_create_config_tables.sql: adds config.xe_shred_state (one row per collector name,
    keyed on collector_name, storing last_execution_count and last_checked_time) as the durable
    place to carry that count between runs. Both collectors persist their own read there after every
    run, regardless of which branch the gate took.
  • deprecated/Installer/README.md:3: brought in line with the root README's v3.3.0 note — the
    Full edition is no longer in release assets, still supported and buildable for existing users.

Test plan

  • Build: dotnet build Darling/Darling.Tests/Darling.Tests.csproj -p:EnableWindowsTargeting=true
    and the same for Lite.Tests — 0 warnings / 0 errors on both (this fix doesn't touch either
    project, but the brief requires the check). Both projects target net10.0-windows and build but
    cannot run here; CI decides them, unaffected by this change.
  • Pins (deprecated/Installer.Tests/XeShredGateSqlTests.cs, a source guard reading the install
    scripts as text, the same pattern as QueryStoreSliceAggregationSqlTests): 7 assertions across
    6 tests, run with dotnet run --project deprecated/Installer.Tests/Installer.Tests.csproj --no-build -- -class "Installer.Tests.XeShredGateSqlTests" (this bypasses the runtime/DB-touching
    harness entirely — no server needed). All 7 PASS against the fixed files. Against the pre-fix
    files (verified with git stash over just the three changed SQL files), 5 of 7 FAIL:
    BlockedProcessCollectorReadsExecutionCountBeforeShredding,
    DeadlockCollectorReadsExecutionCountBeforeShredding,
    CollectorsPersistTheGateResultAfterEveryRun (both cases), and
    ConfigTablesScriptCreatesTheShredStateTable — because the old files have no
    execution_count read, no @shred_needed, no config.xe_shred_state table, and no persistence
    step at all.
  • sql-validation.yml equivalent: NOT run locally — the workflow spins up four SQL Server
    containers (2017/2019/2022/2025) and installs the full install/ tree end to end. I did not run
    this locally within this lane's time budget; CI's own sql-validation run on the PR is the
    equivalent check and should be treated as required here.
  • Before/after ring-buffer timing (blocked-threshold + real blocked-process/deadlock events, a
    second no-change run measured with SET STATISTICS TIME ON): NOT completed. I reached the
    context-wall handoff point before starting the container measurement step. This is the one piece
    of the brief I could not fit — see "For the coordinator" below.

CHANGELOG entry

SECTION: Fixed
ENTRY:

For the coordinator

  • Measurement not done. The brief asked for a before/after timing run on a throwaway SQL
    Server container (blocked-process threshold + real events, then a no-change second run, CPU/
    elapsed ms via SET STATISTICS TIME ON, before and after the fix). I hit this lane's context-wall
    handoff point right as I was about to start the container and could not fit it in. The pins and
    the source diff establish correctness (unconditional shred before, gated after); the
    before/after numbers are still missing. A short follow-up lane can do just this: start
    mcr.microsoft.com/mssql/server:2022-latest on a free port, install install/01 through
    install/24 (plus 21_setup_blocked_process_xe.sql), set the blocked-process threshold low,
    generate a lock-holding session and a few dozen deadlocks, run each collector twice (first run
    populates, second run should be near-zero cost under the gate), and diff that second-run timing
    against the same script run on origin/dev (pre-fix). No code changes needed for that lane —
    just the measurement.
  • I did not run the sql-validation GitHub Actions equivalent locally (four SQL Server
    containers, full install tree). CI's own run on this PR is the actual check; please don't merge
    until it's green.
  • No migrations were added or touched. config.xe_shred_state is created via the existing
    config.ensure_config_tables self-healing pattern, same as every other config table in that
    file, so existing installs pick it up on their next run of that procedure (already called from
    several collectors and the scheduled master collector).

pm-pr: measured before/after

No-change collector run (5x, median): blocked-process 150 ms → 35 ms, deadlock 56 ms → 28 ms; a run with a fresh event still collects it. The measurement run also caught a defect in this PR's first commit (@shred_needed undeclared inside the dynamic batches, so every run errored into CATCH); it's fixed, and pinned by DynamicSqlBatchesDeclareOrParameterizeEveryVariableTheyReference.

Re-shred dedupe: both collectors already skip events already stored, via NOT EXISTS … event_time (install/22_collect_blocked_processes.sql ~432/507, install/24_collect_deadlock_xml.sql ~389/464). That predates this PR and matches dev, and it's now pinned.

pm-pr lane report (dedupe check + missing-DECLARE pin)

Head sha: 454a37cb (branch fix/4213-dashboard-xe-ring-buffer-gate, draft PR #4371). git merge origin/dev was already up to date — no merge commit was created, so head is my one commit on top of the prior 3e30ec9b.

Dedupe question (RULING-CHANGED scope)

Both collectors already dedupe re-shredded events at head — no fix was needed, contrary to the original brief's open question.

  • install/22_collect_blocked_processes.sql:432 (Azure branch) and :507 (server-scoped branch): AND NOT EXISTS (SELECT 1/0 FROM collect.blocked_process_xml AS bx WHERE bx.event_time = evt.value('(@timestamp)[1]', 'datetime2(7)'))
  • install/24_collect_deadlock_xml.sql:389 (Azure branch) and :464 (server-scoped branch): same predicate against collect.deadlock_xml AS dx.

Traced with git log/git show: this NOT EXISTS predicate predates PR #4213 entirely — present in d0f9dc22 (an unrelated #1086 fix, well before this PR's first commit f8ded1ee) and unchanged in origin/dev. So dev already deduped on every full shred; the gate in #4213 doesn't add duplicate risk and doesn't need a fix. Since it exists at head, no container re-measurement for duplicate rows is needed — I did not start a container.

Added a source-text pin, CollectorsDedupeReShreddedEventsOnInsert (theory, both collectors), asserting the NOT EXISTS ... event_time predicate appears exactly twice per file (one per platform branch) against the correct target table/alias.

Missing-DECLARE pin

Added DynamicSqlBatchesDeclareOrParameterizeEveryVariableTheyReference (theory, both files) to deprecated/Installer.Tests/XeShredGateSqlTests.cs. It extracts each SET @sql = N'...' / EXECUTE sys.sp_executesql @sql, N'...params...' pair (comment-stripped, doubled-''-aware), and for every @variable referenced in the SQL text (with quoted string literals removed first, so XQuery @timestamp/@name attribute refs inside ''...'' don't false-positive) requires a local DECLARE or an sp_executesql parameter.

  • RED against f8ded1ee (this PR's own first commit, pre-DECLARE-fix): both files failed with references @shred_needed without a local DECLARE or an sp_executesql parameter — exactly the live defect pm-worker found. Verified by temporarily swapping in git show f8ded1ee:... for both files, running the filtered class, then restoring head's files (confirmed clean via git status --short).
  • GREEN at head (454a37cb).

Filtered run totals

dotnet run --project deprecated/Installer.Tests/Installer.Tests.csproj -- -class "Installer.Tests.XeShredGateSqlTests"

  • At head: Total: 11, Errors: 0, Failed: 0, Skipped: 0
  • Against f8ded1ee swap-in: Total: 11, Errors: 0, Failed: 2 (both new-DECLARE-pin cases, as expected; the 9 pre-existing pins still passed since the dedupe predicate and everything else pre-existed).

Build

dotnet build deprecated/Installer.Tests/Installer.Tests.csproj: Build succeeded, 0 Warning(s), 0 Error(s).

Anything else found

Nothing else. No GitHub writes were made; git push only.

…nt (#4213)

install/22_collect_blocked_processes.sql and install/24_collect_deadlock_xml.sql cast the whole
ring_buffer target to XML and shred it on every scheduled run, even when nothing new has arrived
(#4200's defect, fixed for Darling and Lite in #4212). Both collectors now read the target's own
execution_count before the cast and compare it against a value carried in a new
config.xe_shred_state table; a match skips the conversion and shred entirely. A missing count, no
stored count, or a count that has gone down (session restart) still does a full read.

Adds config.xe_shred_state (install/03_create_config_tables.sql) and a source-guard test class
(deprecated/Installer.Tests/XeShredGateSqlTests.cs) that pins the gate's presence in both
collector scripts and fails against the pre-fix file. Also brings
deprecated/Installer/README.md:3 in line with the root README's v3.3.0 release-assets note.

Closes #4213.
…XE gate (#4213)

XeShredGateSqlTests gains two source-text assertions: config.xe_shred_state
is created with the repo's existing IF OBJECT_ID(...) IS NULL idiom, and a
missing state row means both collectors fail open to a full read rather
than skipping real data.

Measuring the gate against a throwaway container turned up a real defect
the pins do not cover: @shred_needed was referenced inside the dynamic SQL
string in both collectors' server-scoped and Azure branches but never
declared there, so every live run threw 'Must declare the scalar variable'
and fell through to the CATCH block. Declared it locally in each dynamic
batch.
… coverage (#4213)

XeShredGateSqlTests gains two source guards:

CollectorsDedupeReShreddedEventsOnInsert pins the NOT EXISTS predicate both
collectors' payload INSERT already carries (unchanged since before this PR,
matching origin/dev) so a changed execution_count's full re-shred does not
re-insert events already stored. Confirmed present at head for both
platform branches of both collectors, so no dedupe fix was needed in this
PR -- the predicate already exists.

DynamicSqlBatchesDeclareOrParameterizeEveryVariableTheyReference parses
each sp_executesql call's SQL text and params string and asserts every
@variable referenced inside the SQL text is either DECLAREd there or
listed as a parameter. Confirmed RED against f8ded1e (this PR's own
first commit, before the DECLARE fix) with the exact failure the live run
hit: @shred_needed referenced without a local DECLARE in all 4 dynamic
batches. GREEN at head.
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review September 26, 2026 01:46
@erikdarlingdata
erikdarlingdata merged commit dfe9288 into dev Sep 26, 2026
21 of 22 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/4213-dashboard-xe-ring-buffer-gate branch September 26, 2026 01:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant