Skip to content

Severe-error rows read the database id from the event's action and drop what sp_HealthParser drops (#4452) - #4455

Merged
erikdarlingdata merged 2 commits into
devfrom
fix/4452-system-health-parity-severe-errors
Sep 27, 2026
Merged

erikdarlingdata merged 2 commits into
devfrom
fix/4452-system-health-parity-severe-errors

Conversation

@erikdarlingdata

@erikdarlingdata erikdarlingdata commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner

Refs #4452.

Why

The system_health parser's severe-error shred read database_id from the event's data axis
(data[@name="database_id"]/value). Real error_reported events never carry that value on the
data axis — it rides on the event's ACTION axis instead — so the field was always null in
practice. sp_HealthParser hit the same bug and fixed it by switching to the action axis; this
change mirrors that fix.

Separately, the parser applied none of sp_HealthParser's severe-error base filter. The proc always
drops severity below 16 and the two benign connection-reset error numbers (17830, 18056), in both
its filtered and unfiltered modes; only the raise from 16 to 19 is specific to the "significant"
(warnings_only) view. The app's shared significance helper only implemented the 19 floor, so an
unfiltered read (any surface that lists all rows rather than only the significant ones) could show
a severity-15 row or a 17830/18056 the proc's own base population never contains.

What changes

  • SystemHealthParser.ParseSevereError now reads database_id from action[@name="database_id"]/value
    instead of the data axis.
  • ParseSevereError now applies sp_HealthParser's always-on base filter itself: a severity below 16, or
    an error number of 17830/18056, drops the row before a SevereErrorRecord is ever created. This is
    applied once, in the shared parser, so every consumer (unfiltered listings and the significant-only
    gate alike) already agrees with the proc's base population. SystemHealthSignificance.IsSignificant
    is unchanged in behavior — it still raises the floor to 19 for the significant set — but its doc
    comment now says where the 16-floor half of the job is done.
  • Doc comments on SevereErrorRecord and the SystemHealthParser class summary are updated to match:
    they no longer claim the base filter is left entirely to a later stage.
  • No change to ParseSchedulerIssue/SchedulerIssueRecord (out of scope; ported on its own branch).

Real-event coverage

Every fixture under Darling/Darling.Tests/Fixtures/SystemHealth/real/ is a real captured event,
scrubbed (no host, database, login, client address, or object/tenant name — callstack_rva/call_stack
stripped) but otherwise byte-identical in shape to the engine's own XE output. Each pin asserts values
hand-computed from the fixture's XML using sp_HealthParser's own xpath, in a comment beside the assertion.

section pinned against real XML notes
error_reported (severe errors) yes confirms database_id is an action, not a data element, on this real event
memory_broker_ring_buffer_recorded yes field-for-field match
wait_info (significant waits) yes real event's sql_text is a BACKUP statement, so it's shredded but not significant — the statement text itself is now a synthetic BACKUP DATABASE [SyntheticDb] TO DISK = N'NUL' value, keeping the real shape (still starts with BACKUP) without carrying the captured statement text
sp_server_diagnostics_component_result / SYSTEM yes field-for-field match
sp_server_diagnostics_component_result / RESOURCE yes *_gb integer-division checked against the raw byte/KB attributes
sp_server_diagnostics_component_result / QUERY_PROCESSING yes real event has one pending task; below the significance threshold
sp_server_diagnostics_component_result / IO_SUBSYSTEM yes real event has no pending request, so the parser correctly yields an empty list
memory_node_oom_ring_buffer_recorded no no real event was captured in the query window; not fabricated — left unpinned
scheduler_monitor_system_health_ring_buffer_recorded n/a out of scope; ported on its own branch

Test plan

  • SystemHealthParserTests (existing, macOS in-process run): 29/29 pass. The pre-existing
    error_reported.xml synthetic fixture was updated to carry database_id as an action (matching the
    real XE shape) instead of a data element, since the old shape no longer round-trips through the fix.
  • SystemHealthParserRealEventTests (new, macOS in-process run): 7/7 pass — the real-event pins above.
  • ViewerSystemEventsTests: 67/67 pass (unaffected; confirms the significance predicate still matches).
  • DarlingMcpHealthParserToolsSurfaceAndSqlTests (macOS in-process run): 21/21 pass.
  • DarlingMcpHealthParserToolsLivePostgresTests (macOS in-process run, own Postgres/TimescaleDB
    container, darling role/db provisioned): 2/2 pass, including the new
    GetSevereErrors_BaseFilter_DropsBelow16AndIgnoredNumbers_KeepsOnly19PlusSignificant, which plants a
    severity-15, a 16, a 20, and an 18056 row and reads them back through get_health_parser_severe_errors
    (the product's own MCP read path) plus the parser directly: the tool returns exactly the severity-20 row
    as significant, and the parser's own base-filtered population is exactly the 16 and the 20.
  • DocCommentHygieneTests (macOS in-process run): 77/77 pass.
  • RED on the pre-fix commit (detached worktree at the branch's base): copying the new live test onto
    the old parser and running it against the same container produced
    GetSevereErrors_BaseFilter_DropsBelow16AndIgnoredNumbers_KeepsOnly19PlusSignificant [FAIL] — a
    Collections differ assertion failure, confirming the base-filter population differed before the fix
    (the old parser let every severity through unfiltered, including the 15 and the 18056).
  • Mutation pins, run directly against this branch's code (temporary edits, reverted after each):
    • Reverting ParseSevereError to read database_id off the data axis again turned
      SevereError_RealEvent_DatabaseIdReadsActionAxis RED: Assert.Equal() Failure: Values differ / Expected: 0 / Actual: null.
      Restoring the action-axis read turned it GREEN again.
    • Removing the severity/error-number base filter from ParseSevereError turned
      GetSevereErrors_BaseFilter_DropsBelow16AndIgnoredNumbers_KeepsOnly19PlusSignificant RED:
      Assert.Equal() Failure: Collections differ / Expected: [50002, 50003] / Actual: [18056, 50001, 50002, 50003].
      Restoring the filter turned it GREEN again (182/182 across the full pinned set).
  • Darling.Tests.csproj and Lite.Tests.csproj both build 0 errors / 0 warnings with
    -p:EnableWindowsTargeting=true. Lite.Tests cannot run on macOS (WindowsBase discovery); CI decides it.

Final head: cc77f1bb3.

CHANGELOG entry

SECTION: Fixed
ENTRY:

… what sp_HealthParser drops (#4452)

error_reported carries database_id as an XE action, not a data element; the data-axis read this
parser used returned NULL for every event. Switch to the action axis, matching sp_HealthParser's own
fix for the same bug.

Also apply sp_HealthParser's always-on base filter (severity < 16 dropped, error numbers 17830/18056
dropped) once, in the shared parser, so every consumer already agrees with the proc's unfiltered
population; the significant-only gate only adds the warnings_only raise to severity 19 on top.

Adds real (scrubbed) system_health event fixtures with hand-computed parity pins for every section
that has one: error_reported, memory_broker_ring_buffer_recorded, wait_info, and the SYSTEM,
RESOURCE, QUERY_PROCESSING and IO_SUBSYSTEM sp_server_diagnostics_component_result components.
memory_node_oom has no real captured event to pin against, and scheduler_monitor is out of scope
(ported separately).

Refs #4452.
@erikdarlingdata
erikdarlingdata marked this pull request as ready for review September 27, 2026 00:44
@erikdarlingdata
erikdarlingdata merged commit 5336662 into dev Sep 27, 2026
16 of 18 checks passed
@erikdarlingdata
erikdarlingdata deleted the fix/4452-system-health-parity-severe-errors branch September 27, 2026 00:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant