Skip to content

A Wait Statistics "per second" counter occasionally stores ~98 million as a gauge and flattens the throughput chart #4476

Closed

Description

@erikdarlingdata

Claude posting for Erik Darling

A Wait Statistics "per second" performance counter occasionally stores a value near 98 million, and the Viewer's General Throughput chart then flattens every other counter to zero.

Seen

One monitored server on a production store, perfmon_stats, counter Network IO waits, instance Waits started per second (stored cntr_type 65792):

  • normally small;
  • 98,332,865 at 2026-09-27 00:22:58Z and 97,619,588 at 2026-09-26 22:25:51Z, two isolated points.

The Darling Viewer's Performance → General Throughput chart plots the stored value. Those two points set the y-axis, so every other plotted counter reads as 0 across the window.

Why (to confirm)

  • cntr_type 65792 is PERF_COUNTER_LARGE_RAWCOUNT, a gauge the product stores as read. Wait Statistics' "per second" instances are normally a rate type (272696576, a cumulative counter the collector turns into a per-second delta), so a 65792 row for a "per second" instance is suspect on its own.
  • A value near 98 M looks like a cumulative total leaking through the gauge path: either the counter's type as reported at that instant, or the collector's per-instance type handling.
  • V132 added cntr_type so the store can tell the two kinds apart. This needs a look at how the collector picks the gauge path for this instance.

To do

  1. Check fleet-wide how many servers show it (a read over perfmon_stats for Wait Statistics "per second" instances stored as 65792, with the max value per server).
  2. Find the cause in the collector: the type reported by sys.dm_os_performance_counters for that instance, and how the collector routes it.
  3. Fix it so a cumulative value never lands as a gauge. Separately, consider a plausibility guard in the chart (a single point many orders of magnitude above its neighbors shouldn't flatten the chart).

Related: #3717 (perfmon_stats value handling), V132 (cntr_type).

Activity

  1. erikdarlingdata commented on Sep 27, 2026

    @erikdarlingdata
    OwnerAuthor

    Closed by the watcher: delivered in PR #4490, merged to dev.

  2. erikdarlingdata commented on Sep 27, 2026

    @erikdarlingdata
    OwnerAuthor

    Claude posting for Erik Darling.

    Fixed by #4490 (084bae8) for the Darling Viewer's perfmon chart and get_perfmon_trend. A single Wait Statistics sample that jumps to a cumulative-sized value is now set aside instead of plotted, and the count of set-aside samples is reported. The sample must be a gauge, have both neighbours present, be at least 1,000,000, and be more than 1,000 times its larger neighbour. The stored values are unchanged. The same treatment for Lite follows in its own PR. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions