Skip to content

Perfmon trend charts and get_perfmon_trend set aside a one-sample Wait Statistics spike instead of plotting it (#4476) - #4490

Merged
erikdarlingdata merged 7 commits into
devfrom
fix/4476-waitstats-single-sample-artifact
Sep 27, 2026
Merged

erikdarlingdata merged 7 commits into
devfrom
fix/4476-waitstats-single-sample-artifact

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner

Fixes #4476.

Why

SQL Server occasionally returns a cumulative-sized value for ONE sample of a SQLServer:Wait Statistics "per second" instance, which is stored as a gauge (cntr_type 65792).

  • The field evidence: one production store, 3 days, 43 servers: 141 samples over 10,000,000 on 18 servers (max 274,700,202; typical medians 76–504).
  • Each one is isolated: one server, one sample, with normal neighbours (e.g. 318 → 214,396,451 → 1,021; a 64 s gap). 3–4 wait families spike at that same sample, and there's no restart or failover nearby.
  • The magnitude matches the counter's count since instance start.
  • A second store had none.
  • The effect: the Viewer's perfmon chart plots the stored value, so one such point sets the y-axis and flattens every other line to 0. get_perfmon_trend reports it as the peak.

The stored value stays what SQL Server returned. The READ path recognises the isolated sample and sets it aside visibly.

What changes

  • PerformanceMonitor.Common/WaitStatisticsArtifact: one rule, in C# (IsIsolatedSingleSampleArtifact) and as a SQL twin (ArtifactPredicateSql) built from the same constants. A sample is set aside when all of these hold:
    • it's a gauge;
    • its object ends :Wait Statistics (so a named instance's MSSQL$X:Wait Statistics is included);
    • BOTH neighbours in the same (object, counter, instance) series exist;
    • the value is ≥ 1,000,000;
    • the value is > 1,000 × the larger neighbour.
    • The first and last samples of a window are never judged.
  • Both trend reads, the Viewer's ViewerDataService.PerfmonTrendsSql and MCP's DarlingTrendReader.PerfmonTrendSql/PerfmonTrendBucketedSql, read the raw rows as a UNION ALL:
    • rows of any other object carry is_artifact = false and open no window;
    • Wait Statistics rows get lag/lead and the predicate.
    • The per-collection sums use FILTER (WHERE NOT is_artifact), so the peak, average and last value exclude the sample.
    • A trailing artifacts_set_aside column counts them, and existing ordinals don't move.
    • A bucket whose only sample was set aside is skipped, not plotted as 0.
  • The Viewer's perfmon chart titles the count ("1 one-sample Wait Statistics spike set aside").
  • get_perfmon_trend reports artifacts_set_aside and a notes line when it's non-zero.
  • PerfmonBucketPoint gains ArtifactsSetAside, appended with a default of 0, so Lite compiles unchanged. Lite's own reads get the same treatment in a follow-up PR stacked on this one.
  • get_perfmon_stats is unchanged. It returns the latest sample, which never has a next neighbour, so the rule can't judge it.

Cost

EXPLAIN (ANALYZE, BUFFERS), warm, 6 runs averaged, on a rig with 7 days of data: 12 non-Wait-Statistics counters (120,960 rows) plus one Wait Statistics counter with 4 instances (40,320 rows), hourly buckets.

SQL 12 non-Wait-Statistics counters the Wait Statistics counter alone
before this PR 105.0 ms 28.5 ms
a window over every row (an intermediate shape, not shipped) 191.8 ms 75.0 ms
this PR 153.0 ms 71.8 ms
  • The non-Wait-Statistics arm is a plain scan with no window; the remaining ~46% is the two-arm append and the aggregate over it.
  • The Wait Statistics arm pays for the per-instance window it needs.

Test plan

  • WaitStatisticsArtifactTests (pure, 12/12) and WaitStatisticsArtifactSqlEquivalenceLiveTests (the SQL twin against the C# rule over the same 12 rows, 12/12). The rows are:
    • the four field shapes → set aside;
    • a real level step, a 667× burst, a sub-floor spike, missing neighbours, a non-Wait-Statistics object and a rate type → kept;
    • a named instance's object → set aside.
  • PerfmonWaitStatisticsArtifactViewerLiveTests, through GetPerfmonTrendsByCountersAsync, on a plant of 4 instances × 5 collections with the 214,396,451 spike:
    • the spike collection plots 62. RED on dev: Expected: 62, Actual: 214396513;
    • its ArtifactsSetAside = 1;
    • a spike with no other instance leaves no point.
  • PerfmonWaitStatisticsArtifactMcpLiveTests, through the get_perfmon_trend tool: the peak is < 10,000, artifacts_set_aside = 1, and the notes carry the caption. RED on dev at runtime (the spike's bucket publishes: Expected: 4, Actual: 5).
  • PerfmonChartArtifactSummaryTests (the title helper).
  • The census: PerfmonCounterTypeRungTests is updated for both SQL shapes.
  • Mutations:
    • the SQL predicate replaced by false → the Viewer and MCP pins RED;
    • the object-suffix constant broken → the Viewer pin RED;
    • all reverted.
  • Totals: the Viewer set is 110/110 on a live TimescaleDB 2.30.1 rig; the MCP set (with McpPayloadContractCensusTests, McpToolsListBudgetTests and the trend tool classes) is 222/222. Darling.Tests and Lite.Tests build with 0 warnings.

CHANGELOG

SECTION: Fixed
ENTRY:

)

PerformanceMonitor.Common.WaitStatisticsArtifact.IsIsolatedSingleSampleArtifact
is the shared test for a Wait Statistics gauge row whose value is a
one-sample cumulative-count artifact: both neighbours present, value
>= 1,000,000, and > 1,000x the larger neighbour, scoped to gauge rows
whose object name ends in ":Wait Statistics". The read-path wiring
(Viewer perfmon chart, MCP get_perfmon_stats) and pins are not yet in
this commit; see the PR body for what remains.
…for the artifact check (#4476)

PerfmonTrendsSql opened its lag/lead artifact-detection window over every raw
row it scanned, even though the isolated single-sample artifact can only
occur on a SQLServer:Wait Statistics object. Split the raw layer into a
UNION ALL: non-Wait-Statistics rows carry a plain false AS is_artifact with
no window; only Wait Statistics rows open the lag/lead window. Mirrors the
shape the MCP trend read already has.
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review September 27, 2026 18:53
@erikdarlingdata
erikdarlingdata merged commit 084bae8 into dev Sep 27, 2026
16 of 18 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/4476-waitstats-single-sample-artifact branch September 27, 2026 18:53
erikdarlingdata added a commit that referenced this pull request Sep 27, 2026
…t Statistics spike (#4492)

Lite's perfmon chart and get_perfmon_trend set aside a one-sample Wait Statistics spike, and say how many they set aside. This is the same rule the Darling Viewer and Darling's get_perfmon_trend use since #4490.

- LocalDataService.Perfmon.cs (the chart read) and LocalDataService.TrendBuckets.cs (GetPerfmonBucketsAsync) apply the shared WaitStatisticsArtifact predicate in their DuckDB SQL, so a sample is set aside under the same conditions:
  - a Wait Statistics gauge;
  - both neighbours present;
  - at least 1,000,000;
  - more than 1,000 times the larger neighbour.
  A bucket holding only artifacts is dropped.
- Lite's chart reports the count, and Lite's get_perfmon_trend reports artifacts_set_aside through the shared payload, which had always read 0 on Lite.
- The stored values are unchanged.
- Tests:
  - LitePerfmonWaitStatisticsArtifactSourceTests (Darling.Tests; runs on every platform) checks that Lite's two SQL builders use the shared predicate;
  - LitePerfmonWaitStatisticsArtifactLiveTests (Lite.Tests; Windows) reads a planted spike through GetPerfmonTrendsByCountersAsync. The spike is set aside, the neighbours are unchanged, and a collection holding only artifacts is dropped.

Refs #4476
erikdarlingdata added a commit that referenced this pull request Sep 27, 2026
…tract

# Conflicts:
#	Darling/Darling.Tests/ViewerPerfmonRunningJobsTests.cs
#	Darling/PerformanceMonitor.Darling.Viewer/ViewerDataService.Perfmon.cs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant