Long-running query completion trace (opt-in XE) in Lite + Darling (#1496)

CHANGELOG.md

@@ -33,6 +33,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **Long-running query completion trace (opt-in, default OFF) in Lite and the Darling viewer** ([#1572]) - a new `long_query_completions` collector captures the Extended Events equivalent of the classic long-query trace (SSMS's QuickSessionStandard completion trio): `rpc_completed` + `sql_batch_completed` filtered by `duration >= threshold` (default 2,000,000 µs = 2 s, matching the Full Dashboard's SQL Trace default in `install/30_collect_trace_management.sql`), plus `attention` captured UNFILTERED so cancelled/timed-out long queries are still evidenced (the attention event's own duration is the cancellation-handling time, not the query runtime, so a duration predicate would discard exactly those rows - MS Learn-verified). Each completion carries the 9 QuickSessionStandard actions plus duration/CPU/reads/writes/row_count/`result` (Abort = the long query was killed) and the statement text. It reads its own app-managed ring-buffer XE session exactly like the blocked-process/deadlock collectors - server-scoped on on-prem/MI/RDS, database-scoped per monitored database on Azure SQL DB (#1535 pattern; `nt_username` omitted there since Azure uses Entra/SQL auth). **Erik's dedicated switch:** the collector is seeded DISABLED and opted into per fleet (a completion trace is not free on a busy server); enabling it CREATES the session on the monitored servers and disabling it DROPS the session there (the session itself is the cost - the lifecycle reconciles to the flag both ways, mirrored across both SKUs from shared session-DDL builders so the two can't drift). Both apps get a **Long Queries** tab (a grid of completed long queries + attentions, default-sorted DESCENDING by duration, aborted/attention rows tinted) with an explicit empty-state banner naming the exact switch to turn it on when the collector is disabled, plus a `get_long_query_completions` MCP read tool. Retention rides the shared 30-day collector horizon; the Darling store adds it via migration V29.
+
 - **Darling viewer: ships as a Velopack `Setup.exe` for remote seats, like Lite and Dashboard** ([#1571]) - the standalone viewer that connects to a central Darling store from an operator's own machine now has a real install affordance: a per-user `Setup.exe` with Start Menu / desktop shortcuts, Apps & Features registration, and delta auto-updates from the releases feed, on its own `darlingviewer` Velopack channel. This required making the viewer a first-class Velopack app (it never was - it only shipped co-located inside the service zip): a `Program.Main` entry point that calls `VelopackApp.Build().Run()` before any window (wired via `<StartupObject>`, exactly as Lite/Dashboard), the `Velopack` package pinned to the suite's `1.2.0`, and Lite's "Check for Updates" affordance ported into the About dialog (Velopack self-update, falling back to opening the releases page). Nightly builds publish the `Setup.exe` UNSIGNED as a plain release asset alongside the existing zips (which are unchanged); the release pipeline is wired to publish self-contained, sign, `vpk pack`, and upload to the channel - but the signed release path is inert until the `DarlingViewer` SignPath artifact-configuration slug is created (the standing #1340 signing prerequisite, Erik's side). The co-located viewer inside `PerformanceMonitorDarling-*.zip` is untouched (zero-config parent-directory `darling.json` discovery), and the Velopack install id (`%LocalAppData%\PerformanceMonitorDarlingViewer`) does not collide with the viewer's per-user state (`%APPDATA%\PerformanceMonitorDarling`), so both can coexist for one user.
 
 - **Queries: Top Queries by Duration now attributes each statement to its named module — or "ad hoc" — in Lite and the Darling viewer** ([#1569]) — the Full Dashboard has long tied ad-hoc-looking `dm_exec_query_stats` rows back to the procedure/function/trigger they belong to (its `install/10` collector stamps `object_name`/`schema_name` onto `collect.query_stats` by joining `collect.procedure_stats` on `sql_handle`), but Lite and the Darling viewer never surfaced it — every Top Queries row read as an anonymous statement. Both apps now do the stitch at READ time (Option B — zero collector/schema changes, and it retroactively attributes already-collected data): the Top Queries by Duration reader LEFT JOINs `query_stats.sql_handle` to `procedure_stats.sql_handle` — the SAME normalized `CONVERT(varchar(130), ..., 1)` handle text both collectors already persist — and surfaces the matched module's `database.schema.object` in a new **Module** column, or the literal `ad hoc` when the handle matches no cached module. The join is deduped to ONE module identity per `sql_handle` (latest `collection_time` wins, via `ROW_NUMBER() ... WHERE rn = 1` in both the DuckDB and Postgres readers — Postgres has no `QUALIFY`), so a query row can never fan out across a module's many collected rows. One caveat inherent to read-time attribution: a very new module the procedure-stats collector has not captured yet will read as ad hoc until its next collection cycle. The Darling reader stays pure Postgres dialect (positional params, no T-SQL) and the additive column trips no capability-pin baseline. Same change in both apps (parity), pinned by new reader round-trip tests (Lite real-DuckDB; Darling gated live-Postgres) plus SQL-shape, PG-dialect, and row-model display pins.
@@ -411,6 +413,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 [#1567]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1567
 [#1569]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1569
 [#1570]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1570
+[#1572]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1572
 [#1571]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1571
 [#1536]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1536
 [#1534]: https://github.com/erikdarlingdata/PerformanceMonitor/pull/1534

Darling/Darling.Tests/DarlingObservabilityTests.cs

@@ -32,9 +32,9 @@ public sealed class DarlingObservabilityTests
     private const int TestServerId = -424242;
 
     [Fact]
-    public void MigrationScripts_TwentyEightVersions_V27DeadlocksDatabaseName_V28QueryStoreReplicaRole()
+    public void MigrationScripts_TwentyNineVersions_V28QueryStoreReplicaRole_V29LongQueryCompletions()
     {
-        Assert.Equal(28, PgMigrations.Scripts.Count);
+        Assert.Equal(29, PgMigrations.Scripts.Count);
         Assert.Equal(1, PgMigrations.Scripts[0].Version);
         Assert.Equal(2, PgMigrations.Scripts[1].Version);
         Assert.Equal(3, PgMigrations.Scripts[2].Version);
@@ -63,7 +63,8 @@ public void MigrationScripts_TwentyEightVersions_V27DeadlocksDatabaseName_V28Que
         Assert.Equal(26, PgMigrations.Scripts[25].Version);
         Assert.Equal(27, PgMigrations.Scripts[26].Version);
         Assert.Equal(28, PgMigrations.Scripts[27].Version);
-        Assert.Equal(28, StorageVersion.SchemaVersion);
+        Assert.Equal(29, PgMigrations.Scripts[28].Version);
+        Assert.Equal(29, StorageVersion.SchemaVersion);
 
         /* V26 (#1506) adds the generic webhook channel's four columns to the V17 control-plane table.
            Schema-qualified config.* and IF NOT EXISTS, per the file's additive-ALTER idiom. */
@@ -87,6 +88,14 @@ depends on it. */
         Assert.Contains("ALTER TABLE query_store_stats ADD COLUMN IF NOT EXISTS replica_role text;", v28, StringComparison.Ordinal);
         Assert.Contains("CREATE OR REPLACE VIEW v_query_store_stats AS SELECT * FROM query_store_stats;", v28, StringComparison.Ordinal);
 
+        /* V29 (#1496) creates the long_query_completions collector table on an upgraded store (a fresh
+           store gets it from V1's catalog walk). CREATE TABLE IF NOT EXISTS in the collect schema, plus its
+           retrieval index; no v_* view (a post-V14 collector — the viewer reads the base table). */
+        var v29 = PgMigrations.Scripts[28].Sql;
+        Assert.Contains("CREATE TABLE IF NOT EXISTS collect.long_query_completions (", v29, StringComparison.Ordinal);
+        Assert.Contains("duration_microseconds bigint", v29, StringComparison.Ordinal);
+        Assert.Contains("CREATE INDEX IF NOT EXISTS idx_long_query_completions_time ON collect.long_query_completions(server_id, collection_time);", v29, StringComparison.Ordinal);
+
         /* V5 completes the v_* twin of Lite's DuckDB view layer -- the copy-parity tail tabs
            (Running Jobs, Configuration, Daily Summary, Collection Health) read these five, so
            their ported SQL stays byte-identical to Lite's. */

Darling/Darling.Tests/PgSchemaGeneratorTests.cs

@@ -26,10 +26,10 @@ public sealed class PgSchemaGeneratorTests
     [Fact]
     public void Catalog_CoversAllCollectors_WithUniqueTablesAndNames()
     {
-        /* 33 through default_trace_events + job_history + agent_status (#1433 Job History tab) = 35. */
-        Assert.Equal(35, CollectorCatalog.All.Count);
-        Assert.Equal(35, CollectorCatalog.All.Select(s => s.TargetTable).Distinct().Count());
-        Assert.Equal(35, CollectorCatalog.All.Select(s => s.Name).Distinct().Count());
+        /* 35 through agent_status + long_query_completions (#1496 long-query trace) = 36. */
+        Assert.Equal(36, CollectorCatalog.All.Count);
+        Assert.Equal(36, CollectorCatalog.All.Select(s => s.TargetTable).Distinct().Count());
+        Assert.Equal(36, CollectorCatalog.All.Select(s => s.Name).Distinct().Count());
     }
 
     [Fact]
@@ -439,11 +439,11 @@ public void GenerateFullSchema_EmitsEveryTableAndIndex()
         var script = PgSchemaGenerator.GenerateFullSchema();
 
         var tableCount = CollectorCatalog.All.Count(s => script.Contains($"CREATE TABLE IF NOT EXISTS {s.TargetTable} (", StringComparison.Ordinal));
-        Assert.Equal(35, tableCount);
+        Assert.Equal(36, tableCount);
 
-        /* 35 tables minus the two index-less config tables (server_config, database_config) = 33 indexes. */
+        /* 36 tables minus the two index-less config tables (server_config, database_config) = 34 indexes. */
         var indexCount = script.Split("CREATE INDEX IF NOT EXISTS").Length - 1;
-        Assert.Equal(33, indexCount);
+        Assert.Equal(34, indexCount);
 
         /* The precision guard can never regress silently. */
         Assert.DoesNotContain("numeric(0,0)", script, StringComparison.Ordinal);

Darling/Darling.Tests/ViewerDataServiceTests.cs

@@ -400,24 +400,25 @@ public void StoreSchemaProbeSql_ProbesInformationSchema_ForTheV17ToV25Sentinels(
     }
 
     [Theory]
-    [InlineData(true, true, true, true, true, true, true, true, true, true, true, true, 28)]  // fully migrated V28 (query_store_stats replica_role, #1546)
-    [InlineData(true, true, true, true, true, true, true, true, true, true, true, false, 27)] // V27: deadlocks database_name present, replica_role not yet (#1535)
-    [InlineData(true, true, true, true, true, true, true, true, true, true, false, false, 26)] // V26: generic webhook present, deadlocks column not yet
-    [InlineData(true, true, true, true, true, true, true, true, true, false, false, false, 25)] // V25: agent_status present, generic webhook not yet
-    [InlineData(true, true, true, true, true, true, true, true, false, false, false, false, 24)] // V24: job_history present, agent_status not yet
-    [InlineData(true, true, true, true, true, true, true, false, false, false, false, false, 23)]    // V23 (collection_log hypertable, or plain-PG at V23)
-    [InlineData(true, true, true, true, true, true, false, false, false, false, false, false, 22)]   // Timescale store still at V22 (index present, collection_log not yet a hypertable)
-    [InlineData(true, true, true, true, true, false, false, false, false, false, false, false, 21)]  // pre-V22: has default_trace_events, no V22 index
-    [InlineData(true, true, true, true, false, false, false, false, false, false, false, false, 20)] // pre-V21: no default_trace_events
-    [InlineData(true, true, true, false, false, false, false, false, false, false, false, false, 19)]// pre-V20: no alert-tuning knobs
-    [InlineData(true, true, false, false, false, false, false, false, false, false, false, false, 18)]// pre-V19: no analysis_state
-    [InlineData(true, false, false, false, false, false, false, false, false, false, false, false, 17)]// pre-V18: no delivery-override column
-    [InlineData(false, false, false, false, false, false, false, false, false, false, false, false, 16)]// pre-V17: no config control plane at all
+    [InlineData(true, true, true, true, true, true, true, true, true, true, true, true, true, 29)]  // fully migrated V29 (long_query_completions, #1496)
+    [InlineData(true, true, true, true, true, true, true, true, true, true, true, true, false, 28)]  // V28: query_store_stats replica_role present, long_query_completions not yet (#1546)
+    [InlineData(true, true, true, true, true, true, true, true, true, true, true, false, false, 27)] // V27: deadlocks database_name present, replica_role not yet (#1535)
+    [InlineData(true, true, true, true, true, true, true, true, true, true, false, false, false, 26)] // V26: generic webhook present, deadlocks column not yet
+    [InlineData(true, true, true, true, true, true, true, true, true, false, false, false, false, 25)] // V25: agent_status present, generic webhook not yet
+    [InlineData(true, true, true, true, true, true, true, true, false, false, false, false, false, 24)] // V24: job_history present, agent_status not yet
+    [InlineData(true, true, true, true, true, true, true, false, false, false, false, false, false, 23)]    // V23 (collection_log hypertable, or plain-PG at V23)
+    [InlineData(true, true, true, true, true, true, false, false, false, false, false, false, false, 22)]   // Timescale store still at V22 (index present, collection_log not yet a hypertable)
+    [InlineData(true, true, true, true, true, false, false, false, false, false, false, false, false, 21)]  // pre-V22: has default_trace_events, no V22 index
+    [InlineData(true, true, true, true, false, false, false, false, false, false, false, false, false, 20)] // pre-V21: no default_trace_events
+    [InlineData(true, true, true, false, false, false, false, false, false, false, false, false, false, 19)]// pre-V20: no alert-tuning knobs
+    [InlineData(true, true, false, false, false, false, false, false, false, false, false, false, false, 18)]// pre-V19: no analysis_state
+    [InlineData(true, false, false, false, false, false, false, false, false, false, false, false, false, 17)]// pre-V18: no delivery-override column
+    [InlineData(false, false, false, false, false, false, false, false, false, false, false, false, false, 16)]// pre-V17: no config control plane at all
     public void MapProbedSchemaVersion_TakesTheHighestSatisfiedSentinel(
-        bool hasConfigControlPlane, bool hasAlertDeliveryOverride, bool hasAnalysisState, bool hasAlertTuningKnobs, bool hasDefaultTraceEvents, bool hasIndexObjectStatsLatestIndex, bool hasCollectionLogHypertableOrPlainPg, bool hasJobHistory, bool hasAgentStatus, bool hasGenericWebhook, bool hasDeadlocksDatabaseName, bool hasQueryStoreReplicaRole, int expected)
+        bool hasConfigControlPlane, bool hasAlertDeliveryOverride, bool hasAnalysisState, bool hasAlertTuningKnobs, bool hasDefaultTraceEvents, bool hasIndexObjectStatsLatestIndex, bool hasCollectionLogHypertableOrPlainPg, bool hasJobHistory, bool hasAgentStatus, bool hasGenericWebhook, bool hasDeadlocksDatabaseName, bool hasQueryStoreReplicaRole, bool hasLongQueryCompletions, int expected)
     {
         Assert.Equal(expected, ViewerDataService.MapProbedSchemaVersion(
-            hasConfigControlPlane, hasAlertDeliveryOverride, hasAnalysisState, hasAlertTuningKnobs, hasDefaultTraceEvents, hasIndexObjectStatsLatestIndex, hasCollectionLogHypertableOrPlainPg, hasJobHistory, hasAgentStatus, hasGenericWebhook, hasDeadlocksDatabaseName, hasQueryStoreReplicaRole));
+            hasConfigControlPlane, hasAlertDeliveryOverride, hasAnalysisState, hasAlertTuningKnobs, hasDefaultTraceEvents, hasIndexObjectStatsLatestIndex, hasCollectionLogHypertableOrPlainPg, hasJobHistory, hasAgentStatus, hasGenericWebhook, hasDeadlocksDatabaseName, hasQueryStoreReplicaRole, hasLongQueryCompletions));
     }
 
     [Fact]
@@ -427,7 +428,7 @@ public void MapProbedSchemaVersion_V23CompositeIsGatedBehindV22_NotAStandaloneTo
            but must NOT be reported as 23 — the composite only counts once the engine-agnostic V22 index is
            present. Here: pre-V22 (no index) with the composite true still maps to 21 (its real V21 sentinel),
            proving the composite is gated behind V22 rather than treated as a newest-first arm. */
-        Assert.Equal(21, ViewerDataService.MapProbedSchemaVersion(true, true, true, true, true, false, true, false, false, false, false, false));
+        Assert.Equal(21, ViewerDataService.MapProbedSchemaVersion(true, true, true, true, true, false, true, false, false, false, false, false, false));
     }
 
     [Fact]
@@ -437,11 +438,11 @@ public void RequiredStoreSchemaVersion_TracksTheBuildSchemaVersion_AndTheProbeCo
         Assert.Equal(StorageVersion.SchemaVersion, ViewerDataService.RequiredStoreSchemaVersion);
 
         /* Pin: a fully-migrated store (all sentinels present) must map to exactly the required version. If a
-           future migration bumps StorageVersion past 28, this fails until a matching sentinel + map arm is
+           future migration bumps StorageVersion past 29, this fails until a matching sentinel + map arm is
            added — the guard against the probe silently under-reporting a newer store as skewed. */
         Assert.Equal(
             ViewerDataService.RequiredStoreSchemaVersion,
-            ViewerDataService.MapProbedSchemaVersion(true, true, true, true, true, true, true, true, true, true, true, true));
+            ViewerDataService.MapProbedSchemaVersion(true, true, true, true, true, true, true, true, true, true, true, true, true));
     }
 }