MutexTests failing on Unix, browser and WASI after #135778 (default named-mutex scope change) #135887

cc @jkoritzinsky

#135778 ("Fix shared mutex privliges/permissions/defaults to be secure") merged on 2026-10-15 with System.Threading.Tests.MutexTests failures across Unix-like platforms. The failures show in that PR's own runtime build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1635689

Failing tests

From build 1635689 (Linux CoreCLR legs, for example net11.0-linux-Debug-x64-coreclr_release-Ubuntu.2604.Amd64.Open):

• NameOptionsApiCompatibilityTest
• NamePrefixAndOptionsCompatibilityTest
• NameNamespaceTest
• AbandonExisting
• CrossProcess_NamedMutex_ProtectedFileAccessAtomic
• NamedMutex_DefaultScope_UsesExpectedSharedMemoryPath (added in Fix shared mutex privliges/permissions/defaults to be secure #135778)
• NamedMutex_InvalidSharedMemoryHeaderKind
• NamedMutex_InvalidSharedMemoryHeaderVersion
• NamedMutex_TooSmallSharedMemoryFile

NameOptionsApiCompatibilityTest also fails on macOS (net11.0-osx-Debug-arm64-coreclr_release), on browser (CoreCLR and Mono, Chrome/Firefox/V8), and on WASI (CoreCLR_WASI_Release-WasmTestOnWasmtime). Android CoreCLR legs also show Mutex failures.

Example error on Linux:

NamedMutex_InvalidSharedMemoryHeaderVersion:
Expected one of: (WaitHandleCannotBeOpenedException, InvalidDataException) -> Actual: (System.IO.IOException):
The directory '/tmp/.dotnet/shm/global' does not have the expected owner permissions: 40755.

Analysis of NameOptionsApiCompatibilityTest

#135778 changed the Unix default for named mutexes created without NamedWaitHandleOptions from global to current-user-only (Mutex.Unix.cs, CalculateUserScope, unless System.Threading.Mutex.UnsafeDefaultMutexScopeGlobal is set). The test still assumes the default matches CurrentUserOnly = false:

1. new Mutex(false, prefixedName, out createdNew) with no options. This is now user-scoped.
2. new Mutex(false, name, new() { CurrentUserOnly = false, ... }, out createdNew). This is global scope.
3. Assert.False(createdNew) fails because the two scopes don't share storage. On platforms with cross-process mutexes they use different directories (/tmp vs XDG_RUNTIME_DIR / TMPDIR). Without FEATURE_CROSS_PROCESS_MUTEX (browser, WASI), user scope adds a User\ prefix to the in-process name.

Reproduced locally on CoreCLR WASI (Release) on main 6b1cd4b85c4: all three NamePrefixes_MemberData cases ("", Local\, Global\) fail with Assert.False() Failure. The same failures occur on a branch with and without unrelated wait changes, so they come from #135778 itself.

It's not clear to me whether the test expectations or the new default should change. The remaining Linux failures (for example the /tmp/.dotnet/shm/global owner-permission IOException) look like a separate symptom of the new directory validation on Helix machines.

Known Issue Error Message

DO NOT USE JSON BELOW IF THIS IS A BUILD BREAK otherwise build analysis will allow pull requests to merge that break the build worse. For a build break, do not use this issue form. Make a regular new issue.

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Report

Summary