Skip to content

Move the test suite to Microsoft.Testing.Platform for xunit.v3 4.0.0 (#434) - #442

Merged
erikdarlingdata merged 1 commit into
devfrom
chore/xunit-v3-4-mtp
Aug 21, 2026
Merged

erikdarlingdata merged 1 commit into
devfrom
chore/xunit-v3-4-mtp

Conversation

@erikdarlingdata

Copy link
Copy Markdown
Owner

Fixes the build failure on Dependabot #434, and supersedes both #434 and #433.

Why #434 fails

Microsoft.Testing.Platform.MSBuild.targets(320,5): error : Testing with VSTest target is
no longer supported by Microsoft.Testing.Platform on .NET 10 SDK and later.

xunit.v3 4.0.0 drops VSTest on the .NET 10 SDK. There is no version of that bump that keeps the old runner — it is migrate or stay on 3.2.2. #433 is the same family and passes only because bumping the VSTest adapter on its own changes nothing.

What moved

  • The opt-in is repo-level, not per-project. The target names it outright (_SupportsGlobalJsonTestRunner), so global.json now selects the runner. TestingPlatformDotnetTestSupport in the csproj alone does not do it — that was my first attempt and it reproduced the same error.
  • xunit.runner.visualstudio is dropped, because it is the VSTest adapter and has no role any more. That moots deps: Bump xunit.runner.visualstudio from 3.1.5 to 4.0.0 #433.
  • --filter still works in the form the workflows and local runs already use.

The watchdog port — the part worth reviewing

Both layers of 5f84116 were VSTest-only. RunSettingsFilePath/TestSessionTimeout covered bare local runs; --blame-hang covered the three workflows.

before after
bare local dotnet test RunSettingsFilePath → TestSessionTimeout 15m TestingPlatformCommandLineArguments → --timeout 15m
ci / nightly / release --blame-hang --blame-hang-timeout 5m -- --hangdump --hangdump-timeout 5m --hangdump-type none

The local layer is verified the way the original was, since a silently ignored property looks exactly like a working one: set to 1ms temporarily, a bare dotnet test cancelled the run with 0 tests succeeded, then restored.

What the port does NOT recover

Stating this plainly because it would be easy to leave implied. Neither MTP mechanism stops the GC-suspension livelock the original watchdog was written for. Measured against a genuinely wedged host on macOS ARM64:

  • 2m55s at 107% CPU against --hangdump-timeout 90s — never fired.
  • 8 minutes against --timeout 60s — never fired.

The reason is the one 5f84116 already gives for xUnit's own timeouts: the diagnostics the platform needs are served by the execution engine that is suspended. MTP does put the timeout in a separate controller process — I confirmed the process tree, and that was the property that mattered — and it still cannot win.

An ordinary hang, where the runtime is responsive, is killed fine.

I am not claiming this is a regression against VSTest. I did not prove the old TestSessionTimeout could kill a livelocked host either — it was verified at 1ms on a healthy run, never against a real wedge. Both runners may be equally powerless here. Filed separately as #441.

Tests

279 passed, 0 failed, 2 skipped under the new runner — the whole suite except the Mcp/Repl classes, which wedge on macOS for reasons that predate this PR and are the subject of #441. CI runs on ubuntu-latest and is unaffected. The CI flag form was run verbatim to confirm the options parse.

dotnet build clean; the 9 warnings are the pre-existing MCP9005 obsolete-API uses in McpSmokeTests.cs.

Not merging this myself — it changes how tests run everywhere, including your laptop.

🤖 Generated with Claude Code

@claude

claude Bot commented Aug 21, 2026

Copy link
Copy Markdown

Reviewed. This is a self-contained test-infra/CI change (xunit.v3 4.0.0 + Microsoft.Testing.Platform migration) — no application code, execution-plan XML handling, T-SQL generation, or PlanViewer.Ssms/PlanViewer.Web files touched, so most of the usual correctness/security/versioning checks don't apply here.

Went through the mechanics specifically:

  • global.json test.runner + TestingPlatformDotnetTestSupport is the documented combination needed to get dotnet test onto MTP (the property alone isn't sufficient, matches what the PR description says).
  • The -- separator in the three workflow files correctly forwards --hangdump/--hangdump-timeout/--hangdump-type to the MTP host instead of being parsed by dotnet test itself.
  • TestingPlatformCommandLineArguments (15m) and CI's --hangdump-timeout 5m both apply on CI runs simultaneously — that's not new, the old RunSettingsFilePath (15m TestSessionTimeout) + --blame-hang-timeout 5m were layered the same way pre-PR, so behavior here is preserved, not changed.
  • No dangling references to the deleted hang-watchdog.runsettings or removed xunit.runner.visualstudio package anywhere else in the repo.
  • Only one test project in the solution, so no risk of a project left on the old runner conflicting with the new global.json-selected runner.

Description is unusually candid about what this does not fix (the GC-suspension livelock, deferred to #441) — appreciate that being called out rather than implied. No findings to flag.

…434)

Dependabot #434 fails to build, and not because of anything it did wrong:

  Microsoft.Testing.Platform.MSBuild.targets(320,5): error : Testing with VSTest
  target is no longer supported by Microsoft.Testing.Platform on .NET 10 SDK and
  later.

xunit.v3 4.0.0 drops VSTest on the .NET 10 SDK. There is no version of this bump
that keeps the old runner, so it is migrate or stay on 3.2.2. #433 is the same
family and passes only because bumping the VSTest adapter alone changes nothing.

The opt-in is repo-level, not per-project, which the target names outright
(_SupportsGlobalJsonTestRunner): global.json now selects the runner. Setting
TestingPlatformDotnetTestSupport in the csproj alone does not do it, and I tried
that first.

xunit.runner.visualstudio is dropped because it IS the VSTest adapter and has no
role here any more. That moots #433.

The watchdog from 5f84116 needed porting, and this is the part worth reading.
Both of its layers were VSTest-only: RunSettingsFilePath/TestSessionTimeout for
bare local runs, and the blame-hang options in the three workflows. The local
layer is now TestingPlatformCommandLineArguments carrying a 15m session timeout,
same value and same no-flags-to-remember property; the workflows pass the hangdump
options, which produce the same name-the-wedged-test artifact.

I verified the local layer the way the original was verified, because a silently
ignored property looks exactly like a working one: set it to 1ms temporarily and a
bare `dotnet test` cancelled with 0 tests succeeded, then restored it.

What the port does NOT recover, stated plainly because it would be easy to leave
implied: neither MTP mechanism stops the GC-suspension livelock the original
watchdog was written for. Measured against a genuinely wedged host on macOS ARM64,
not assumed - it ran 2m55s at 107% CPU against a 90s hangdump timeout, and 8
minutes against a 60s session timeout, ignoring both. The reason is the same one
5f84116 gives for xUnit timeouts: the diagnostics the platform needs are served by
the execution engine that is suspended. MTP does put the timeout in a separate
controller process, which was the property that mattered, and it still cannot win.
An ordinary hang, where the runtime is responsive, is killed fine.

So this is not a regression against VSTest so much as both runners being equally
powerless there, and I want to be careful not to claim more: I did not prove the
OLD TestSessionTimeout could kill a livelocked host either. It was verified at 1ms
on a healthy run, never against a real wedge. Filed separately.

Tested: 279 passed, 0 failed, 2 skipped under the new runner (the whole suite
except the Mcp/Repl classes, which wedge on macOS for reasons that predate this and
are the subject of that separate issue). The CI flag form was run verbatim to
confirm the options parse. dotnet build clean; the 9 warnings are the pre-existing
MCP9005 obsolete-API uses in McpSmokeTests.cs.

Two XML-comment '--' errors on the way in, exactly as 5f84116 warned. Noted in the
comment so the next person does not rediscover it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@claude

claude Bot commented Aug 21, 2026

Copy link
Copy Markdown

Reviewed. This PR is entirely CI/test-infrastructure (VSTest → Microsoft.Testing.Platform migration for xunit.v3 4.0.0) — none of it touches execution-plan XML parsing, generated T-SQL, or the PlanViewer.Ssms/PlanViewer.Web linked-file/version-sync conventions, so those review dimensions don't apply here.

Static read of the diff:

  • global.json (test.runner: Microsoft.Testing.Platform) + TestingPlatformDotnetTestSupport=true in the csproj are complementary, not redundant — repo-level opt-in plus per-project participation — and match the documented MTP dotnet test integration.
  • TestingPlatformCommandLineArguments=--timeout 15m correctly replaces the old RunSettingsFilePath/TestSessionTimeout for bare local runs; the three workflow files consistently switch to -- --hangdump --hangdump-timeout 5m --hangdump-type none, mirroring the old --blame-hang* flags 1:1 (including none as a valid dump-type, same as the VSTest blame collector).
  • hang-watchdog.runsettings is fully removed and I don't see any other reference to it, RunSettingsFilePath, blame-hang, or xunit.runner.visualstudio left anywhere in the repo (workflows, docs, CONTRIBUTING.md) — no stale pointers.
  • The XML-comment -- gotcha the PR body mentions is correctly worked around (flags spelled without leading dashes inside the comment).
  • Scope of the new watchdog vs. the GC-suspension livelock is stated honestly in the comment and PR body, with measured (not assumed) numbers — good, since a security/correctness reviewer's first question would be "does this actually solve the problem," and the answer is explicitly "no, and here's what it does solve."

One caveat I can't independently close out: this sandbox has no network/dotnet-execution access, so I could not run dotnet restore/dotnet test myself to confirm Microsoft.Testing.Extensions.HangDump 2.3.3 exists on NuGet and that --hangdump-type none is accepted by that exact version — I'm relying on the PR description's own local verification (279 passed, watchdog behavior measured on a wedged host) for that. Worth letting CI actually go green before merging, which the author already flagged by not self-merging.

No correctness, security, or convention issues found.

@erikdarlingdata
erikdarlingdata merged commit 42045a1 into dev Aug 21, 2026
3 checks passed
erikdarlingdata added a commit that referenced this pull request Aug 21, 2026
Minor rather than patch. 1.19.x would understate it: #439 adds a "source" field
to every warning in the JSON and MCP output and a new badge in the app and CLI, and
#437 changes what an existing analysis rule concludes about a plan. Both are things
a consumer can notice, and one of them is output-shape.

What ships:

- #437 Rule 12 no longer calls a conversion non-SARGable when it converts the
  parameter rather than the column. Plans carrying a parameter-side conversion on a
  scan lose that warning and report the scan's residual predicate instead. Verified
  against all 38 committed plans: no other plan's verdict moves.
- #439 SQL Server's own warnings are now told apart from ours, tagged [SQL Server]
  in the app and CLI and carried as "source" in JSON/MCP. Additive, but it changes
  the bytes of analyze --compact.
- #431 Robot Advice no longer takes the app down on a deep plan.
- #438 querystore gets the same depth ceiling analyze got; it had been failing
  quietly on deep plans, one ERROR row per plan.
- #443 the macOS handle resolver no longer corrupts memory on Apple silicon.
- #425 Entra MFA works again (WAM parent window handle).

Not user-facing but worth knowing for anyone building from this tag: the suite runs
on Microsoft.Testing.Platform now (#442), and `dotnet test` finishes on macOS for
the first time (#443) - 307 tests, 305 passing, 13 seconds.

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@erikdarlingdata
erikdarlingdata deleted the chore/xunit-v3-4-mtp branch August 21, 2026 10:33
@erikdarlingdata erikdarlingdata mentioned this pull request Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant