Skip to content

Long-running query completion trace (opt-in XE) in Lite + Darling (#1496) - #1572

Merged
erikdarlingdata merged 2 commits into
devfrom
feature/1496-long-query-trace
Jul 18, 2026
Merged

erikdarlingdata merged 2 commits into
devfrom
feature/1496-long-query-trace

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Closes #1496.

What

A new opt-in long_query_completions collector — the Extended Events equivalent of the classic long-query trace (SSMS's QuickSessionStandard completion trio), modeled end-to-end on the blocked_process_report collector family, shipped in both Lite and the Darling viewer.

  • Captures rpc_completed + sql_batch_completed filtered by duration >= threshold, plus attention unfiltered — a cancelled/timed-out long query's attention.duration is the cancellation-handling time (tens of ms), not the query runtime, so a duration predicate would discard exactly that evidence (MS Learn-verified). Completed events also carry result (Abort = the long query was killed).
  • Captures the 9 QuickSessionStandard actions + duration/cpu/reads/writes/row_count/result + statement text (SET collect_statement/collect_object_name/collect_batch_text = 1 — these are off by default or arrive NULL).
  • Reads its own app-managed ring-buffer XE session exactly like blocked-process/deadlock: server-scoped on-prem/MI/RDS, database-scoped per monitored DB on Azure SQL DB ([QUESTION] About deadlocks, generic webhook #1535 pattern).

Erik's dedicated switch (default OFF)

Seeded disabled via a new CollectorScheduleDefaults.Entry.DefaultEnabled flag both SKUs honor (Lite's ScheduleManager; Darling's StoreConfigProvider.ResolveSchedule falls back to it, so "reset to defaults" returns it to OFF rather than silently ON). The XE session lifecycle follows the flag: enabling creates the session on the monitored servers; disabling DROPS it there (the session itself is the busy-server cost). The reconcile is state-tracked (zero steady-state cost for the default-off fleet) and driven from shared session-DDL builders on the collector (BuildCreateSessionSql/BuildStartSessionSql/BuildDropSessionSql) so Lite and Darling can't drift on the 3-event/9-action DDL. Drop-on-disable had no precedent in the codebase (both apps only ever created XE sessions) — it is net-new and test-covered.

Both apps

  • Shared collector + CollectorCatalog registration → Lite's catalog-driven storage registrations (table/index/archive view/retention) and Darling's PgSchemaGenerator schema come for free; Darling store migration V29 for upgraded stores + StorageVersion bump 28→29 (viewer schema-probe extended with a 13th sentinel).
  • A Long Queries viewer tab in each app: a grid of completed long queries + attentions, default-sorted DESCENDING by duration (view-only SortDescription, never flips the SQL ORDER BY), aborted/attention rows tinted.
  • Empty-state banner (per the mid-flight requirement): when the tab is opened while the collector is disabled, a WarningTextBrush banner (the msdb-permissions idiom) names the exact switch — Lite: "Settings → Collector Schedules → Edit → tick the Enabled box"; Darling: "Settings → Collection Schedule → Edit Collector Schedules…".
  • A get_long_query_completions MCP read tool in both apps (Lite reads DuckDB, Darling reads Postgres) — added to both to keep the cross-app MCP inventory pin green with no allow-list debt (parity).

Key decisions

  • Threshold default = 2,000,000 µs (2 s) — the value the Full Dashboard's long-running-query SQL Trace shipped (install/30_collect_trace_management.sql:37, which captured the same RPC/batch/attention trio), NOT the 30 s fallback. Hardcoded const with doc comment: the schedule model carries only frequency/retention (no per-collector numeric-knob surface), so this follows the blocked-process-threshold hardcoded precedent + defaults-over-speculative-config.
  • Azure SQL DB variant = per-database session (mirrors blocked-process), not gated off. MS Learn confirms rpc_completed/sql_batch_completed/attention are all available in an Azure database-scoped session (used directly — no database_-prefixed variant, unlike deadlocks). nt_username is omitted from the Azure DDL — Azure uses Entra/SQL auth so it's meaningless and may be absent database-scoped; the column simply lands NULL there.
  • Extra fields beyond the issue's representative list: result (flags timed-out/aborted long queries — verifier-recommended), physical_reads, and object_name (the proc that ran) — all free in the completed-event payload.

Tests / gates

  • Builds: Lite, Darling service, Darling viewer all succeed.
  • Lite.Tests: 1388 passed, 0 failed, 0 skipped. Darling.Tests: 2197 passed, 0 failed, 136 skipped (gated live-Postgres/live-SQL tests; no env vars set — expected). No Installer.Tests, nothing run against live SQL contexts.
  • New LongQueryCompletionsCollectorDefinitionTests pins identity, the ring-buffer read/event filter, the 20-column payload ordinals (this caught a real ReadAsync off-by-one before it shipped), attention duration→NULL, and the DDL builders (predicate on completed-only, attention unfiltered, SET collect_*, Azure nt_username omission, guarded idempotent DROP).
  • Count pins bumped 35→36 (DuckDbSchemaGeneratorTests, DuckDbSchemaEquivalenceTests, DuckDbSchemaTests 42→43, PgSchemaGeneratorTests incl. 33→34 indexes, DarlingObservabilityTests migration count/version); golden schema oracle gains the table+index; shared-defaults pin strengthened to enforce Enabled↔DefaultEnabled parity; viewer schema-probe test extended to 13 sentinels.

Capability pins / baselines

The ServerTab / ViewerServerTab capability pins are subset ratchets, so the additive tab + grid + filter-manager + event-wire capabilities kept them green with no baseline file changes (baselines only ever shrink on intentional removals). No baseline lines added or deleted.

Noticed / not changed

  • The old Dashboard SQL Trace also OR'd a CPU >= 1000 ms predicate; the issue design specifies duration >= threshold only, so this follows the design (CPU is still captured as a column).
  • Darling reconcile re-ensures the session per sweep when enabled (self-healing, incl. restarting a stopped Azure database-scoped session); this is more frequent than blocked-process's per-connect ensure but only for opted-in servers, and the disabled default incurs zero connections after the first drop-confirm.

🤖 Generated with Claude Code

erikdarlingdata and others added 2 commits July 18, 2026 10:20
)

New long_query_completions collector: an app-managed Extended Events
ring-buffer session capturing rpc_completed + sql_batch_completed
(duration >= 2s, the Full Dashboard's SQL Trace default) plus attention
unfiltered (cancelled/timed-out long queries - attention.duration is
handling time, not query runtime). Modeled end-to-end on the
blocked-process collector family.

Dedicated switch, default OFF: seeded disabled via a new
CollectorScheduleDefaults.DefaultEnabled flag both SKUs honor; enabling
creates the session on monitored servers, disabling DROPS it there (the
lifecycle reconciles to the flag both ways, driven from shared
session-DDL builders on the collector so Lite and Darling can't drift).

Both apps: shared collector + catalog registration + Lite's catalog-driven
storage registrations + Darling store migration V29; a Long Queries viewer
tab (grid default-sorted DESC by duration, aborted/attention row tints)
with an empty-state banner naming the exact switch when disabled; a
get_long_query_completions MCP read tool. Azure SQL DB gets the
per-database session variant (#1535 pattern; nt_username omitted there -
Azure uses Entra/SQL auth).

Closes #1496

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The per-sweep reconcile is act-on-change, so a Darling-monitored Azure SQL
DB database-scoped session that stops on reconnect would not be restarted
(the applied flag stays true and the reconcile skips). Reset the flag to
null on every successful connect so the first post-connect sweep
re-ensures — matching blocked-process's per-connect ensure. The reconcile
still no-ops unless the desired state differs (ensure is IF-NOT-running
START; drop is IF EXISTS), so the disabled default stays near-zero cost.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@erikdarlingdata
erikdarlingdata merged commit d2a9321 into dev Jul 18, 2026
5 checks passed
erikdarlingdata added a commit that referenced this pull request Jul 18, 2026
Rebase onto dev brought #1572's new read tool get_long_query_completions; the endpoint<->catalog
parity test caught it as unwired (exactly its purpose). It is a stored data-read, so add its
/api/read dispatch entry (server, hours_back=24, limit=30). Also carries the web migration renumber
V29->V30 (dev took V29 = long_query_completions): PgMigrations V30Sql + Migration(30), StorageVersion
30, the 14-column store schema probe (long_query->29, web->30), and the migration/probe test pins.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@erikdarlingdata
erikdarlingdata deleted the feature/1496-long-query-trace branch September 12, 2026 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant