Repository navigation
Unhandled exception from AcceptCompletionCallback #108026
Description
2024-09-18T17:42:06.9787720Z Unhandled exception. System.ArgumentException: The supplied System.Net.SocketAddress is an invalid size for the System.Net.IPEndPoint end point. (Parameter 'socketAddress')
2024-09-18T17:42:06.9788630Z at System.Net.IPEndPoint.Create(SocketAddress socketAddress)
2024-09-18T17:42:06.9789430Z at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAccept(SocketAddress remoteSocketAddress)
2024-09-18T17:42:06.9790370Z at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationSyncSuccess(Int32 bytesTransferred, SocketFlags flags)
2024-09-18T17:42:06.9791310Z at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationAsyncSuccess(Int32 bytesTransferred, SocketFlags flags)
2024-09-18T17:42:06.9793120Z at System.Net.Sockets.SocketAsyncEventArgs.AcceptCompletionCallback(IntPtr acceptedFileDescriptor, Memory`1 socketAddress, SocketError socketError)
2024-09-18T17:42:06.9794100Z at System.Net.Sockets.SocketAsyncEngine.System.Threading.IThreadPoolWorkItem.Execute()
2024-09-18T17:42:06.9795710Z at System.Threading.ThreadPoolWorkQueue.Dispatch()
2024-09-18T17:42:06.9796250Z at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
2024-09-18T17:42:06.9798780Z at System.Threading.Thread.StartCallback()
The exception is similar to the one from #81437, but this might indicate a different problem, exceptions from FinishOperationAccept are not handled.
Activity
rzikm commentedon Sep 20, 2024
This report is not actionable for us without knowing more about the scenario when the exception occurred. Does this exception happen frequently? Are you able to reproduce it with some small code example you could share with us?
pepone commentedon Sep 20, 2024
This is happening on macOS, when accepting IPv6 TCP connections. I thought there is a bug here because .NET runtime not handling the exception and as a result the callback passed to
BeginAcceptis not called.The underlying cause is probably the same as
https://github.com/dotnet/runtime/issues/81437unreliable OS call, and it is not easy to reproduce.antonfirsov commentedon Sep 20, 2024
We haven't understood the root cause of #81437. In fact, it has likely stopped occurring in our CI or the frequency is so low, it's below of our radar.
You haven't answered the questions from #108026 (comment). Does the error happen sporadically or predictably? Any code you can possibly share? Any chance for creating a self-contained repro?
Are you using DualMode listener sockets (created by
new Socket(SocketType.Stream, ProtocolType.Tcp)? Does the issue disappear if you change it to be IPv6-only (created bynew Socket(AddressFamily.InterNetworkV6, SocketType.Stream, ProtocolType.Tcp)?wfurt commentedon Sep 23, 2024
I think the underlying issue is same as #102663. Would you be able to run custom code @pepone ? Perhaps we can come up with more instrumentation and/or improved error handling. If that is feasible you can grab binaries from PR or you could rebuild the code changes yourself.
pepone commentedon Sep 23, 2024
Sure, I can try with some custom code.
wfurt commentedon Sep 27, 2024
can you rebuild #108334 and give it shot @pepone ? The PR should also create binaries if that is easier....
pepone commentedon Sep 30, 2024
@wfurt I hit #104105 when trying to build your branch with Xcode 16, Where can I download the binaries from your PR for macOS ARM64?
rzikm commentedon Oct 1, 2024
@pepone you should be able to download the OSX ARM64 build artifacts (Debug) from https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_apis/build/builds/820467/artifacts?artifactName=libraries_bin_osx_x64_Debug&api-version=7.1&%24format=zip. you need only the System.Net.Sockets.dll from the testhost/shared/... directory.
In case you have problems accessing the above url, I attached the relevant files here directly
macAccept.zip
Publish app as self-contained and overwrite the System.Net.Sockets.dll in the publish output to test the change.
pepone commentedon Oct 1, 2024
With this PR I hit an assert instead of getting the unhandled exception:
To make this easy to reproduce I run my test while I also stress the CPU with
stress -c 10.wfurt commentedon Oct 1, 2024
I guess that make sense
runtime/src/libraries/System.Net.Sockets/src/System/Net/Sockets/SocketAsyncEventArgs.Unix.cs
Line 38 in 008ee9f
That one is not in release builds and blows up later. I'll update the PR to pass through that.
wfurt commentedon Oct 2, 2024
https://dev.azure.com/dnceng-public/public/_build/results?buildId=823654&view=artifacts&pathAsName=false&type=publishedArtifacts
pepone commentedon Oct 2, 2024
No more asserts with the latest build.
I think we still need a workaround as proposed in #102663, otherwise when this happens the socket wont be able to accept the connection.
wfurt commentedon Oct 2, 2024
I don't have repro in my hands ... but I don't think we need to. From what I saw the
Acceptonly assigns the address to_remoteEndPointe.g. it does not need it to create new socket. And if somebody accessRemoteEndPointand_remoteEndPointisnullwe would query underlying socket viagetsockoptto get it at that time.If you observer something else please let me know
pepone commentedon Oct 2, 2024
I will try to get more details on this part a comment on the other issue or open a new one.
The unhandled exception part seems to be fixed with your PR.
wfurt commentedon Oct 2, 2024
I'll get it to
main10.0 branch so more people have chance to test it. If that looks good we can start working on servicing permission.antonfirsov commentedon Oct 2, 2024
After reading #102663 (comment), I'm wondering if
getsocknamewill actually return the address succesfully in this weird case #108334 is addressing. @pepone can you check the what happens if you logacceptSocket.RemoteEndpoint?antonfirsov commentedon Mar 19, 2025
This should be fixed by the PR-s linked above.