You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Android][CoreCLR] Runtime-test merged runners skip all tests because the bundle path is passed as the filter #135092
Android CoreCLR runtime-test lanes can exit successfully while executing zero test bodies. The Android CoreCLR host inserts the application bundle path as managed args[0]; generated merged runtime-test runners interpret that argument as an inclusive method-name filter. The path matches no method, so every test is skipped.
Run the existing android-x64 Release AllSubsets_CoreCLR_RuntimeTests lane on release/11.0-rc2 at 474ed963699fc29809015392738a15231aaae2de. The test-only validation merge is 3fe28a8c04c9198dbc5833af4d1f2b02ae2f9141; the PR changes only pipeline YAML.
Inspect the API 29 CoreCLR Helix job 9e801237-be10-41c5-9ef0-176701f8a52f, including:
The APKs install on verified x86_64/API 29 emulators and invoke the CoreCLR entry point. Their retained testResults.xml files contain only skips. Samples of JIT.CodeGenBringUpTests_r on APIs 24-28 and 31-36 independently show the same nine-skipped/zero-executed result.
Expected behavior
With no requested method filter, the generated runtime-test runner executes applicable test bodies. A lane with zero executed tests must not be mistaken for functional runtime validation.
Actual behavior
The complete 76 API 29 runtime XML result files contain 4,272 test rows, 0 passed, 0 failed, 4,272 skipped, including 4,088 with No Known Skip Reason. The remaining skip reasons reference existing ActiveIssues. Six workitems contain no tests.
Despite this, instrumentation, XHarness and the Helix workitem commands return zero. Azure run 44904232 records 4,093 NotExecuted results and zero passes; that collapsed Azure count differs from the raw XML-row count, but both establish no executed tests.
Regression?
The introduction/last passing Android revision has not been established. A matching argument/filter defect was reported for Apple CoreCLR in closed issue #134766; that is supporting evidence for the cause, not proof that Android or RC2 has been fixed.
Known Workarounds
No workaround was applied in this validation run. The host must forward only managed arguments to coreclr_execute_assembly, rather than including the program/bundle path. A follow-up fix should also validate that an unfiltered merged runner actually executes tests. Existing legitimate ActiveIssue skips should be preserved.
Configuration
.NET 11 RC2 branch, Android x64 CoreCLR runtime inner-loop tests, Release. Exact build 1620862 / 20261002.1; head 2e00770eefed18941fec179509e106d329926ba9, base 474ed963699fc29809015392738a15231aaae2de. Observed across the API 24-36 emulator campaign; API 29 was exhaustively reconciled against all 76 XML files, and the named test was sampled on the other eleven zero-exit API levels.
Other information
Verified source flow in the tested worktree:
src/tasks/AndroidAppBuilder/Templates/monodroid-coreclr.c:280-295 sets managed_argc = args_len + 1 and managed_argv[0] = g_bundle_path, then forwards the complete array to CoreCLR.
src/coreclr/dlls/mscoree/exports.cpp:541-569 forwards the supplied array to managed assembly execution; it does not discard element zero as a program name.
src/tests/Common/XUnitWrapperGenerator/XUnitWrapperGenerator.cs:494-496 passes managed args[0] as the runtime-test filter.
RunnerEntryPoint.cs constructs an inclusive method filter, so the bundle-directory path excludes all methods. The generated runner returns zero when no tests failed, even if none executed.
Related validation tracking: #122021. No product source changes or CI reruns were made while investigating this finding.
Note
This issue was prepared with GitHub Copilot from retained CI artifacts and source inspection.
Description
Android CoreCLR runtime-test lanes can exit successfully while executing zero test bodies. The Android CoreCLR host inserts the application bundle path as managed
args[0]; generated merged runtime-test runners interpret that argument as an inclusive method-name filter. The path matches no method, so every test is skipped.Observed while validating .NET 11 RC2 in #135084, https://dev.azure.com/dnceng-public/public/_build/results?buildId=1620862. This is a false-green test-harness/coverage defect, not evidence that the runtime tests passed.
Reproduction Steps
Run the existing
android-x64 Release AllSubsets_CoreCLR_RuntimeTestslane onrelease/11.0-rc2at474ed963699fc29809015392738a15231aaae2de. The test-only validation merge is3fe28a8c04c9198dbc5833af4d1f2b02ae2f9141; the PR changes only pipeline YAML.Inspect the API 29 CoreCLR Helix job
9e801237-be10-41c5-9ef0-176701f8a52f, including:The APKs install on verified x86_64/API 29 emulators and invoke the CoreCLR entry point. Their retained
testResults.xmlfiles contain only skips. Samples ofJIT.CodeGenBringUpTests_ron APIs 24-28 and 31-36 independently show the same nine-skipped/zero-executed result.Expected behavior
With no requested method filter, the generated runtime-test runner executes applicable test bodies. A lane with zero executed tests must not be mistaken for functional runtime validation.
Actual behavior
The complete 76 API 29 runtime XML result files contain 4,272 test rows, 0 passed, 0 failed, 4,272 skipped, including 4,088 with
No Known Skip Reason. The remaining skip reasons reference existing ActiveIssues. Six workitems contain no tests.Representative console output:
Despite this, instrumentation, XHarness and the Helix workitem commands return zero. Azure run 44904232 records 4,093
NotExecutedresults and zero passes; that collapsed Azure count differs from the raw XML-row count, but both establish no executed tests.Regression?
The introduction/last passing Android revision has not been established. A matching argument/filter defect was reported for Apple CoreCLR in closed issue #134766; that is supporting evidence for the cause, not proof that Android or RC2 has been fixed.
Known Workarounds
No workaround was applied in this validation run. The host must forward only managed arguments to
coreclr_execute_assembly, rather than including the program/bundle path. A follow-up fix should also validate that an unfiltered merged runner actually executes tests. Existing legitimate ActiveIssue skips should be preserved.Configuration
.NET 11 RC2 branch, Android x64 CoreCLR runtime inner-loop tests, Release. Exact build 1620862 / 20261002.1; head
2e00770eefed18941fec179509e106d329926ba9, base474ed963699fc29809015392738a15231aaae2de. Observed across the API 24-36 emulator campaign; API 29 was exhaustively reconciled against all 76 XML files, and the named test was sampled on the other eleven zero-exit API levels.Other information
Verified source flow in the tested worktree:
src/tasks/AndroidAppBuilder/Templates/monodroid-coreclr.c:280-295setsmanaged_argc = args_len + 1andmanaged_argv[0] = g_bundle_path, then forwards the complete array to CoreCLR.src/coreclr/dlls/mscoree/exports.cpp:541-569forwards the supplied array to managed assembly execution; it does not discard element zero as a program name.src/tests/Common/XUnitWrapperGenerator/XUnitWrapperGenerator.cs:494-496passes managedargs[0]as the runtime-test filter.RunnerEntryPoint.csconstructs an inclusive method filter, so the bundle-directory path excludes all methods. The generated runner returns zero when no tests failed, even if none executed.Related validation tracking: #122021. No product source changes or CI reruns were made while investigating this finding.
Note
This issue was prepared with GitHub Copilot from retained CI artifacts and source inspection.