AF: op.Next is null == op.Previous is null in ChannelUtilities.Remove (RendezvousChannel cancellation)

[pavelsavara]

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1478944
Leg Name: System.Diagnostics.Process.Tests — Mono libraries test run, azurelinux.3.arm64.open (queue azurelinux.3.arm64.open)

Observed on PR #129713 merge run (Helix work item System.Diagnostics.Process.Tests, job 9c28ca4d-566f-4099-9e96-a37d4e9ab27a). The failure is not related to that PR's changes (a native Mono exception-handling change) — it is a managed concurrency assert in System.Threading.Channels.

Error Details

A managed Debug.Assert fires in ChannelUtilities.Remove from the RendezvousChannel<ProcessOutputLine> cancellation callback running on a thread-pool worker, then FailFast aborts the process (SIGABRT, exit code 134). The RendezvousChannel waiter list (WaitingReadAsyncOperation) is left with an inconsistent Next/Previous state — a race between the cancellation callback's Remove and the read/complete path.

Process terminated. Assertion failed.
op.Next is null == op.Previous is null
   at System.Diagnostics.DebugProvider.Fail(String message, String detailMessage) in /_/src/libraries/System.Private.CoreLib/src/System/Diagnostics/DebugProvider.cs:line 22
   at System.Diagnostics.Debug.Fail(String message, String detailMessage) in /_/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Debug.cs:line 135
   at System.Diagnostics.Debug.Assert(Boolean condition, String message, String detailMessage) in /_/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Debug.cs:line 99
   at System.Diagnostics.Debug.Assert(Boolean condition, String message) in /_/src/libraries/System.Private.CoreLib/src/System/Diagnostics/Debug.cs:line 88
   at System.Threading.Channels.ChannelUtilities.Remove[WaitingReadAsyncOperation](WaitingReadAsyncOperation& head, WaitingReadAsyncOperation op) in /_/src/libraries/System.Threading.Channels/src/System/Threading/Channels/ChannelUtilities.cs:line 152
   at System.Threading.Channels.RendezvousChannel`1.<>c[[System.Diagnostics.ProcessOutputLine, System.Diagnostics.Process, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].<get_CancellationCallbackDelegate>b__16_1(KeyValuePair`2 state) in /_/src/libraries/System.Threading.Channels/src/System/Threading/Channels/RendezvousChannel.cs:line 470
   at System.Threading.QueueUserWorkItemCallbackDefaultContext`1[...].Execute() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 1361
   at System.Threading.ThreadPoolWorkQueue.Dispatch() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/ThreadPoolWorkQueue.cs:line 960
   at System.Threading.PortableThreadPool.WorkerThread.WorkerDoWork(PortableThreadPool threadPoolInstance) in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 137
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart() in /_/src/libraries/System.Private.CoreLib/src/System/Threading/PortableThreadPool.WorkerThread.cs:line 110
   at System.Threading.Thread.StartCallback() in /_/src/mono/System.Private.CoreLib/src/System/Threading/Thread.Mono.cs:line 218

Likely a sibling of the closed Known Build Error #127234 ("AF: The continuation was the completion sentinel.") in the same RendezvousChannel / System.Diagnostics.Process.Tests area, but a distinct assert and code path (ChannelUtilities.Remove via the cancellation callback, vs FailOperations/TryComplete in #127234). The RendezvousChannel<ProcessOutputLine> code is recent (#116097, #126987, #127106).

Error Message

{
    "ErrorMessage": "op.Next is null == op.Previous is null",
    "BuildRetry": false,
    "ExcludeConsoleLog": false
}

Note

This issue was created with the assistance of GitHub Copilot.

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1478944
Error message validated: [op.Next is null == op.Previous is null System.Threading.Channels.ChannelUtilities.Remove[WaitingReadAsyncOperation] get_CancellationCallbackDelegate]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/24/2026 12:05:20 PM UTC

Report

Build	Repository	Test	Pull Request
1485582	dotnet/runtime	System.Diagnostics.Process.Tests.WorkItemExecution	#128862
1478944	dotnet/runtime	System.Diagnostics.Process.Tests.WorkItemExecution	#129713

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	2	2

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1478944
Error message validated: [op.Next is null == op.Previous is null]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 6/29/2026 8:07:19 PM UTC

Report

Build	Repository	Test	Pull Request
1485582	dotnet/runtime	System.Diagnostics.Process.Tests.WorkItemExecution	#128862
1478944	dotnet/runtime	System.Diagnostics.Process.Tests.WorkItemExecution	#129713

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	2	2

Report

Build	Repository	Test	Pull Request
1551179	dotnet/runtime	System.Diagnostics.Process.Tests.WorkItemExecution	#131062
1551088	dotnet/runtime	System.Diagnostics.Process.Tests.WorkItemExecution	#132250
1537158	dotnet/runtime	System.Diagnostics.Process.Tests.WorkItemExecution	#131732
1522952	dotnet/runtime	System.Diagnostics.Process.Tests.WorkItemExecution	#129734

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	4

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-threading-channels
See info in area-owners.md if you want to be subscribed.

[pavelsavara]

cc @stephentoub @adamsitnik

[adamsitnik]

@PranavSenthilnathan @stephentoub does it look like a product bug introduced by #116097 but exposed by #126987 by taking dependency on the related API?

[PranavSenthilnathan]

I suspect #127234 and this are due to the same issue, and I was able to repro another AF with a similar-ish call stack. Basically, I think the root cause is that it's possible for the channel to get into a bad state when we reach the following line with a canceled token:

runtime/src/libraries/System.Threading.Channels/src/System/Threading/Channels/AsyncOperation.cs

Line 117 in 21b722d

_cancellationRegistration = cancellationToken.UnsafeRegister(cancellationCallback, this);

If the token is a canceled token, then the registered callback gets executed immediately. This callback is defined here:

runtime/src/libraries/System.Threading.Channels/src/System/Threading/Channels/RendezvousChannel.cs

Line 449 in 8b16f83

private Action<object?, CancellationToken> CancellationCallbackDelegate =>

Specifically, the TrySetCanceled in the callback will call TryReserveCompletionIfCancelable and then SignalCompletion. In a normal execution, the cancellation token registration would have been set correctly and !CancellationToken.CanBeCanceled would be false. However, in our scenario, the callback is being executed before the registration is set, so the CancellationToken property is actually default(CancellationToken) so !CancellationToken.CanBeCanceled is true. So this causes a short ciruit and the _completionReserved property is ignored incorrectly and the operation will not be marked as completed even though cancellation of the cancellation token has completed it.

Then when we consume the operation on the other side (e.g. we cancel a write and then do a read), we get this assertion failure because we're in a bad state.

This was tricky to repro because it requires the cancellation token to pass the following check:

runtime/src/libraries/System.Threading.Channels/src/System/Threading/Channels/RendezvousChannel.cs

Lines 376 to 379 in 8b16f83

	if (cancellationToken.IsCancellationRequested)
	{
	return new ValueTask(Task.FromCanceled<T>(cancellationToken));
	}

but it also needs to be canceled before

runtime/src/libraries/System.Threading.Channels/src/System/Threading/Channels/RendezvousChannel.cs

Line 402 in 8b16f83

new(parent._runContinuationsAsynchronously, cancellationToken, cancellationCallback: _parent.CancellationCallbackDelegate);

So for my repro, I commented out that if (cancellationToken.IsCancellationRequested) check, and ran the following code:

using System.Threading.Channels;

var cts = new CancellationTokenSource();
var c = Channel.CreateBounded<int>(0);

cts.Cancel();
c.Writer.WriteAsync(42, cts.Token);

while (!c.Reader.TryPeek(out _))
    ;

await c.Reader.ReadAsync();

The error:

Process terminated.
Assertion failed.
The continuation was the completion sentinel.
   at System.Threading.Channels.AsyncOperation.SignalCompletion() in D:\git\runtime\src\libraries\System.Threading.Channels\src\System\Threading\Channels\AsyncOperation.cs:line 212
   at System.Threading.Channels.AsyncOperation`2.DangerousSetResult(TResult result) in D:\git\runtime\src\libraries\System.Threading.Channels\src\System\Threading\Channels\AsyncOperation.cs:line 587
   at System.Threading.Channels.RendezvousChannel`1.RendezvousChannelReader.ReadAsync(CancellationToken cancellationToken) in D:\git\runtime\src\libraries\System.Threading.Channels\src\System\Threading\Channels\RendezvousChannel.cs:line 175
   at Program.<Main>$(String[] args)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine) in D:\git\runtime\src\libraries\System.Private.CoreLib\src\System\Runtime\CompilerServices\AsyncMethodBuilderCore.cs:line 38
   at Program.<Main>$(String[] args)
   at Program.<Main>(String[] args)

Having the if (cancellationToken.IsCancellationRequested) just masks the issue, but if the cancellation token is canceled at just the right time this error could show up even with that guard.