Skip to content

Bucket the blocking-trend reads: lock waits, waiting tasks, blocked sessions (#4349) - #4362

Merged
erikdarlingdata merged 6 commits into
devfrom
fix/4349-blocking-trend-buckets
Sep 26, 2026
Merged

erikdarlingdata merged 6 commits into
devfrom
fix/4349-blocking-trend-buckets

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

Refs #4349.

Why

The BLOCKING trio of trend charts read one row per collection at the 7-day window, unbucketed unlike #4353's (Darling) and #4340's (Lite) twins for the other trend charts. The issue's own measured 7-day row counts at 1-minute cadence: Lock waits 20,160; Waiting tasks 30,240; Blocked sessions 20,160.

Measurement was not completed inside this lane's 30-minute limit (build/test priority came first, per the brief's own ordering). The before/after row counts on a live rig are deferred to a follow-up comment or a sibling lane; the code change is proven by build and by the same bucketing shape #4353/#4340 already ship.

What changes

Darling (ViewerDataService.BlockingTrends.cs):

Lite (LocalDataService.Blocking.cs, LocalDataService.WaitingTasks.cs):

  • GetLockWaitTrendAsync: SQL pulled into a named LockWaitTrendSql string (matching Bucket Lite's memory clerk and File I/O trend reads (#4234) #4340's pattern for checkability), bucketed via time_bucket/to_minutes/TrendBuckets.OriginSql, same rated-CTE and singleton-stamping rules as Darling.
  • GetWaitingTaskTrendAsync / GetBlockedSessionTrendAsync: bucketed inline (matching the reads' existing dynamic-SQL style for the ignore-list/database-filter clauses), same "every row rated" and singleton-stamping rules.

No new shared helper: every read reuses TrendBuckets/TrendBudget (already shared by both SKUs) and keeps its bucket expression local, per the ruling.

Test plan

  • dotnet build Darling/Darling.Tests/Darling.Tests.csproj -p:EnableWindowsTargeting=true — 0 Warning(s), 0 Error(s)
  • dotnet build Lite.Tests/Lite.Tests.csproj -p:EnableWindowsTargeting=true — 0 Warning(s), 0 Error(s)
  • Before/after row-count measurement on a throwaway rig (deferred — see Why)
  • New bucketing unit tests mirroring ViewerCpuTempDbTests / LiteMemoryFileIoTrendBucketingTests (deferred — not added in this lane; listed unchecked per the brief's priority order)
  • Full Windows test suite (CI decides; Darling.Tests/Lite.Tests target net10.0-windows and cannot run on this Mac)

CHANGELOG entry

SECTION: Fixed
ENTRY:

For the coordinator

  • Measurement (before/after row counts on a seeded rig) was not completed inside the 30-minute lane window; build/test/PR took priority per the brief's own ordering. A follow-up run of the harness described in the brief (throwaway net10.0 console referencing PerformanceMonitor.Darling.Storage, timescale/timescaledb:2.30.1-pg18 on port 55471) would confirm the counts before marking this ready.
  • Bucketing unit tests (mirroring ViewerCpuTempDbTests / LiteMemoryFileIoTrendBucketingTests) are not yet added. Both would assert: unbucketed SQL returns one row per collection per series at a 7-day/1-minute seed (fails the "coarsens at wide windows" expectation); bucketed SQL groups by date_bin/time_bucket and returns far fewer rows; a single-collection window stamps points at the raw collection time, not the bucket grid.
  • ViewerDataService.OverviewLanes.cs/ViewerServerTab.Blocking.cs callers were not touched — they already pass (serverId, startUtc, endUtc) and don't post-filter on endUtc client-side (unlike the Bucket the Darling viewer's TempDB file I/O trend (#4234) #4353 LoadTempDbAsync fix), so no caller-side change was needed. Same check on Lite's ServerTab.Refresh.cs call sites — no post-filtering there either.

pm-pr lane report (tests)

Head after this lane: 96318d76 (merged origin/dev in, keeping both sides — no conflicts).

Lite-parity check (coordinator's follow-up): the Lite twins were already present in this PR's 3-file diff — Lite/Services/LocalDataService.Blocking.cs (GetLockWaitTrendAsync) and Lite/Services/LocalDataService.WaitingTasks.cs (GetWaitingTaskTrendAsync, GetBlockedSessionTrendAsync), both bucketed in DuckDB's time_bucket/to_minutes shape. Nothing to add.

Tests added, mirroring ViewerCpuTempDbTests (#4234) / LiteMemoryFileIoTrendBucketingTests (#4340):

  • Darling/Darling.Tests/ViewerBlockingTrendBucketingTests.cs: a source check per read (bucket-width parameter, date_bin/GREATEST, the Measurement-layer campaign: the delta honesty contract (11 findings, one keystone) #3540 rated-CTE no-delta rule for lock waits, plain COUNT(*)-then-CAST(SUM...) for waiting-task duration, and the new per-collection-then-AVG shape for blocked-session count) plus 8 live-Postgres tests (gated on DARLING_TEST_PG, [Collection("live-postgres")]): a 7-day/1-minute-cadence row-budget cap and a budget-covers-every-collection raw-timestamp/point-equality pin for each of the three reads, plus a merged-bucket average-not-sum pin for blocked sessions.
  • Lite.Tests/LiteBlockingTrendBucketingTests.cs: the matching source check plus 7 live-DuckDB tests against SharedDuckDbFixture ([Collection("server-time-helper")], ServerTimeHelper.UtcOffsetMinutes pinned to 0) — same budget-cap / singleton-stamping / merged-bucket-average coverage for all three reads.

RED/GREEN evidence: unchecked. The test projects are net10.0-windows and this lane's host is macOS — WPF-hosted xunit can't run here (confirmed: dotnet exec on the built DLL fails with "No frameworks were found," Microsoft.WindowsDesktop.App isn't installed). I did not stand up the docker timescale/timescaledb:2.30.1-pg18 throwaway harness in the time this follow-up left; CI's Windows build job is the first real run of both new files. Both projects build locally with 0 Warning(s), 0 Error(s).

SQL defect found and fixed (in-lane): BlockedSessionTrendSql (and its Lite twin) computed COUNT(*) AS blocked_count over the bucket's raw waiting_tasks rows. A blocked-session count is a PER-SNAPSHOT gauge, not a delta — the un-bucketed read's COUNT(*) was already "sessions blocked at this one collection." Bucketing by summing that count across every collection a wide bucket merges double- (or N-tuple-) counts purely from how many snapshots got merged, with no more blocking having happened — the same shape as the CPU tab's gauge columns (#4234), which the PR's own doc comments already correctly averaged. Fixed both reads to keep the un-bucketed per-collection count in an inner CTE/subquery, then AVG (rounded, CAST/ROUND to bigint on Postgres) it across the merged collections — never SUM. collection_count still counts the merged collections via COUNT(*) over that CTE. Covered by the new merged-bucket tests in both files.

Left for pm-pr to check by hand: the RED/GREEN Docker proof (item 4 of the brief) and the full Windows suite — both need CI or a Windows/WSL host. The before/after row-count measurement is explicitly out of this lane's scope per the brief (a separate lane's job).

pm-pr: measured before/after

Darling, measured live (docker TimescaleDB 2.30.1-pg18, a fresh migrated store, generate_series at each collector's cadence, one server, 7-day window; dev's SQL vs this PR's SQL at 80507a2, run verbatim): Lock waits 20,160 → 2,016; Waiting tasks 30,240 → 3,024; Blocked sessions 20,160 → 2,016 (10-minute buckets). The lane's later blocked-sessions averaging fix changes the value, not the row count. Lite: computed, not run (no DuckDB on the measuring host); the expectation is the same 2,016 / 3,024 / 2,016.

pm-pr lane report (CRLF locator fix)

New head: 9980f73
Test fixed: Darling.Tests.ViewerBlockingTrendBucketingSqlTests.BlockedSessionTrendSql_CarriesABucketWidth_AndAveragesThePerSnapshotCount_NeverSums
Pattern: sql.ReplaceLineEndings("\n") before the ")\n SELECT" locator search (house pattern, matches ReplaceLineEndings usage elsewhere in Darling.Tests); scanned branch diff for other \n-joined multi-line locators, found none else.

pm-pr lane report (CI pin fixes, assigned to pm-worker)

Both failing tests were mis-aimed pins, and the SQL is unchanged: the NeverSums DoesNotContain is now scoped to the outer select (the inner per-collection CTE rightly counts), and the LockWaitTrendSql pin now asserts the rated-CTE CASE expressions and the time-weighted SUM/SUM. Dev's per-row rate already excluded non-positive intervals and negative deltas, so single-collection results are unchanged.

…essions (#4234) (#4349)

GetLockWaitTrendAsync, GetWaitingTaskTrendAsync, and GetBlockedSessionTrendAsync
returned one row per collection for both products, unbucketed at the 7-day window
unlike #4353's (Darling) and #4340's (Lite) already-bucketed twins for the other
trend charts. Each now buckets to TrendBudget.Chart's per-series point budget via
TrendBuckets.AutoMinutes (date_bin on Darling, time_bucket on Lite, both anchored to
the shared bucket origin), stamps points at their own raw collection time instead of
the bucket grid when every returned bucket holds exactly one physical collection, and
carries the #3540 unrated-collection rule for the lock-wait rate (nulled into a rated
CTE, not filtered, so collection_count still counts an unrated row).

Waiting-task duration and blocked-session count have no interval-based delta to rate
(a waiting-task row is a point-in-time snapshot, not a counter), so every row in those
two reads is unconditionally "rated" and collection_count is a plain COUNT(*).

Darling: ViewerDataService.BlockingTrends.cs. Lite: LocalDataService.Blocking.cs,
LocalDataService.WaitingTasks.cs.
…double-count

Adds source-check and live-store tests for the lock-wait, waiting-task
and blocked-session trend reads, mirroring ViewerCpuTempDbTests (#4234)
and LiteMemoryFileIoTrendBucketingTests (#4340): a 7-day row-budget
cap, point equality for the rate/count formulas, and singleton-window
raw-timestamp stamping, for both products.

Fixes a defect the tests surfaced: BlockedSessionTrendSql/its Lite
twin summed a per-snapshot gauge (COUNT(*) blocked sessions) across
every merged collection in a bucket, double- (or N-tuple-) counting
purely because a wide bucket merged more snapshots, with no more
blocking having happened. Both reads now average the per-collection
count over the merged bucket (matching the CPU tab's gauge-averaging
idiom), via a per_collection CTE that keeps the un-bucketed read's own
count unchanged.

Refs #4349.
…#4349)

BlockedSessionTrendSql_CarriesABucketWidth test asserted no COUNT(*) AS
blocked_count anywhere in the SQL, but the inner per_collection CTE
legitimately keeps that per-snapshot COUNT(*) - it's what the outer AVG
averages. Scoped the assertion to the outer SELECT (after the CTE closes)
instead of the whole SQL text.

LockWaitTrendSql_FiltersLckPrefix_PerSecondRateViaLag pinned the old
per-row "delta_wait_time_ms / interval_seconds" text. The bucketed SQL now
computes the same per-second rate through a time-weighted rated CTE
(SUM(rated_wait_ms) / SUM(rated_seconds)), matching WaitStatsTrendsSql's
existing idiom. Updated the pin to the new text and kept every behavioural
assertion.
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review September 26, 2026 01:45
@erikdarlingdata
erikdarlingdata merged commit 293a74b into dev Sep 26, 2026
15 of 16 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/4349-blocking-trend-buckets branch September 26, 2026 01:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant