[Bug] Incorrect Offline CPUs and NUMA Node count reported due to DAC / Internal Schedulers #5656

Activity

1. nashalj

   added

   bugSomething isn't workingSomething isn't working

   on Oct 11, 2026
2. nashalj commented on Oct 11, 2026

   

   nashalj

   on Oct 11, 2026

   Author

   More actions

   
3. erikdarlingdata commented on Oct 11, 2026

   

   erikdarlingdata

   on Oct 11, 2026

   Owner

   More actions

   Claude posting for Erik Darling.

   Triage notes for the agent who picks this up. The report is right, and the defect is wider than one box. The formula counts the DAC node as an offline node on every SQL Server instance. So every instance shows at least "Offline CPUs: 1" and one extra NUMA node. Nobody has changed any code yet.

   Code read on dev at 2d9709215.

   Where the defect is

   • PerformanceMonitor.Collectors/CpuSchedulerStatsCollector.cs, the nodes derived table (lines 173 to 197) and offline_cpu_count (line 154). This is the shared collector, so Lite and Darling both get the wrong values.
   • install/17_collect_cpu_scheduler_stats.sql has the same SQL (lines 183 and 211 to 221), for the Dashboard. The Dashboard is on bug-fix support, so fix it too.
   • The SQL has been the same since v1.0.0.

   Three faults

   1. The DAC node is excluded by the wrong id. The filter is n.node_id <> 32767. The DAC node's id is 64. The reporter's scheduler query shows the DAC scheduler (1048576, VISIBLE ONLINE (DAC)) on parent_node_id 64.
   2. "Online" is an exact match. The query counts a node as online only when node_state_desc = N'ONLINE'. The sys.dm_os_nodes docs say that the state combines values: one of ONLINE, OFFLINE, IDLE and IDLE_READY, and then DAC, THREAD_RESOURCES_LOW or HOT ADDED. So the DAC node ("ONLINE DAC") counts as offline. So would an IDLE node, or a node that is low on thread resources.
   3. "Offline CPUs" counts nodes, not CPUs. offline_cpus adds 1 for each node that is not exactly ONLINE. The warning flag offline_cpu_warning uses another rule: schedulers with is_online = 0. So the count and the flag disagree. That is why the reporter's 1 is not highlighted. The Dashboard's debug message (line 396 of the install script) also prints the node count as "offline nodes".

   Reproduction

   On a local SQL Server 2025 container with nothing offline, sys.dm_os_nodes returns node 0 ONLINE, node 1 ONLINE and node 64 ONLINE DAC. The collector's nodes query returns total_nodes 3, online_nodes 2 and offline_cpus 1.

   Candidate fix

   I checked this read-only on the same container. It returned 2 of 2 nodes online and 0 offline CPUs.

   /* Nodes: leave out the DAC node by its state, and count every state except OFFLINE as online. */
   SELECT
       total_nodes = COUNT_BIG(*),
       online_nodes = SUM(CASE WHEN n.node_state_desc NOT LIKE N'OFFLINE%' THEN 1 ELSE 0 END)
   FROM sys.dm_os_nodes AS n
   WHERE n.node_state_desc NOT LIKE N'%DAC%';
   
   /* Offline CPUs: count the schedulers that SQL Server has taken offline. */
   SELECT
       offline_cpus = COUNT_BIG(*)
   FROM sys.dm_os_schedulers AS s
   WHERE s.status = N'VISIBLE OFFLINE';

   • Drive offline_cpu_warning from the same scheduler count, so that the count and the flag always agree.
   • Not tested: a box with an affinity-masked CPU or an offline node. A test should cover a node state with a combined value and the DAC node.
   • No schema change. The column names stay the same and only the values change. Rows that are already stored keep the old values.
   • The readers need no change: PerformanceMonitor.Common/CpuSchedulerMetrics.cs (lines 52 and 53), the Darling viewer, and the Lite MCP tool's offline_cpu.
4. erikdarlingdata

   added

   in-progressActively being worked by a local session or its agents (PR open or in flight)Actively being worked by a local session or its agents (PR open or in flight)

   on Oct 11, 2026
5. erikdarlingdata

   added 2 commits that reference this issue on Oct 11, 2026

   Review round 1 for #5656: collection_log text, lite CI filter for ins…

   ...

   377161c

   Count offline CPUs and NUMA nodes correctly in the CPU scheduler coll…

   ...

   3aa8d76
6. erikdarlingdata commented on Oct 11, 2026

   

   erikdarlingdata

   on Oct 11, 2026

   Owner

   More actions

   Fixed on dev by #5657 (3aa8d76). Claude posting for Erik Darling.
7. erikdarlingdata

   closed this as completedon Oct 11, 2026
8. erikdarlingdata

   removed

   in-progressActively being worked by a local session or its agents (PR open or in flight)Actively being worked by a local session or its agents (PR open or in flight)

   on Oct 11, 2026