Skip to content

Access violation (crashes the process) while writing EventSource events #99603

New issue
New issue
Closed
#99606
Closed
Access violation (crashes the process) while writing EventSource events#99603
#99606
Labels
area-System.Net.Httpbugtenet-reliabilityReliability/stability related issue (stress, load problems, etc.)
Milestone
8.0.x

Description

@minkom-ms
minkom-ms
opened on Mar 12, 2024

Describe the bug

There is a path in the logging of EventSource events in the HTTP stack which causes an Access Violation.

To Reproduce

Issue appears entirely in .NET call stacks so there are no specific steps to reproduce.

Exceptions (if any)

Application: Microsoft.Office.AugLoop.ServerCore.YarpGateway.exe
CoreCLR Version: 8.0.224.6711
.NET Version: 8.0.2
Description: The process was terminated due to an unhandled exception.
Exception Info: System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
Stack:
   at System.Buffer.Memmove(Byte ByRef, Byte ByRef, UIntPtr)
   at System.String.Ctor(Char*, Int32, Int32)
   at System.Diagnostics.Tracing.EventSource.DecodeObjects(System.Object[], System.Type[], EventData*)
   at System.Diagnostics.Tracing.EventSource.WriteToAllListeners(System.Diagnostics.Tracing.EventWrittenEventArgs, Int32, EventData*)
   at System.Diagnostics.Tracing.EventSource.WriteEventWithRelatedActivityIdCore(Int32, System.Guid*, Int32, EventData*)
   at System.Net.Http.HttpTelemetry.WriteEvent(Int32, Byte, Byte, Int64, System.String, System.String, Int32, System.String)
   at System.Net.Http.HttpConnectionBase..ctor(System.Net.Http.HttpConnectionPool, System.Net.IPEndPoint)
   at System.Net.Http.HttpConnectionPool+<ConstructHttp11ConnectionAsync>d__108.MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[[System.Net.Http.HttpConnectionPool+<ConstructHttp11ConnectionAsync>d__108, System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]](<ConstructHttp11ConnectionAsync>d__108 ByRef)
   at System.Net.Http.HttpConnectionPool+<CreateHttp11ConnectionAsync>d__105.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Http.HttpConnectionPool+<CreateHttp11ConnectionAsync>d__105, System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Net.Http.HttpConnectionPool+<ConnectAsync>d__103.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.ValueTuple`3[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Http.HttpConnectionPool+<ConnectAsync>d__103, System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Net.Http.ConnectHelper+<EstablishSslConnectionAsync>d__2.MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.__Canon, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Http.ConnectHelper+<EstablishSslConnectionAsync>d__2, System.Net.Http, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].SetExistingTaskResult(System.Threading.Tasks.Task`1<System.Threading.Tasks.VoidTaskResult>, System.Threading.Tasks.VoidTaskResult)
   at System.Net.Security.SslStream+<ForceAuthenticationAsync>d__150`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Threading.Tasks.VoidTaskResult, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<ForceAuthenticationAsync>d__150`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Runtime.CompilerServices.IAsyncStateMachineBox, Boolean)
   at System.Threading.Tasks.Task.RunContinuations(System.Object)
   at System.Net.Security.SslStream+<ReceiveHandshakeFrameAsync>d__151`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Net.Security.SslStream+<ReceiveHandshakeFrameAsync>d__151`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]], System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext(System.Threading.Thread)
   at System.Net.Security.SslStream+<EnsureFullTlsFrameAsync>d__161`1[[System.Net.Security.AsyncReadWriteAdapter, System.Net.Security, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].MoveNext()
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Net.Sockets.SocketAsyncEventArgs+<>c.<.cctor>b__173_0(UInt32, UInt32, System.Threading.NativeOverlapped*)
   at System.Threading.ThreadPoolTypedWorkItemQueue`2[[System.Threading.PortableThreadPool+IOCompletionPoller+Event, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Threading.PortableThreadPool+IOCompletionPoller+Callback, System.Private.CoreLib, Version=8.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]].System.Threading.IThreadPoolWorkItem.Execute()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool+WorkerThread.WorkerThreadStart()

Further technical details

CLR and .Net versions are in the call stack above.

Activity

  1. ghost added
    untriagedNew issue has not been triaged by the area owner
    on Mar 12, 2024

  3. KalleOlaviNiemitalo commented on Mar 12, 2024

    @KalleOlaviNiemitalo
    KalleOlaviNiemitalo
    on Mar 12, 2024

    There is a bug here:

    runtime/src/libraries/System.Net.Http/src/System/Net/Http/HttpTelemetry.cs

    Lines 342 to 386 in 1381d5e

    const int NumEventDatas = 7;
    EventData* descrs = stackalloc EventData[NumEventDatas];
    fixed (char* arg4Ptr = arg4)
    fixed (char* arg5Ptr = arg5)
    fixed (char* arg7Ptr = arg7)
    {
    descrs[0] = new EventData
    {
    DataPointer = (IntPtr)(&arg1),
    Size = sizeof(byte)
    };
    descrs[1] = new EventData
    {
    DataPointer = (IntPtr)(&arg2),
    Size = sizeof(byte)
    };
    descrs[2] = new EventData
    {
    DataPointer = (IntPtr)(&arg3),
    Size = sizeof(long)
    };
    descrs[3] = new EventData
    {
    DataPointer = (IntPtr)arg4Ptr,
    Size = (arg4.Length + 1) * sizeof(char)
    };
    descrs[4] = new EventData
    {
    DataPointer = (IntPtr)arg5Ptr,
    Size = (arg5.Length + 1) * sizeof(char)
    };
    descrs[5] = new EventData
    {
    DataPointer = (IntPtr)(&arg6),
    Size = sizeof(int)
    };
    descrs[6] = new EventData
    {
    DataPointer = (IntPtr)arg7Ptr,
    Size = (arg7.Length + 1) * sizeof(char)
    };
    }
    WriteEventCore(eventId, NumEventDatas, descrs);

    The WriteEventCore(eventId, NumEventDatas, descrs) call is not within the fixed block. This allows the garbage collector to move the strings and invalidate the pointers that were stored in descrs[3].DataPointer, descrs[4].DataPointer, and descrs[6].DataPointer. I don't know whether this is what actually causes the crash, but it could very well be.

  4. KalleOlaviNiemitalo commented on Mar 12, 2024

    @KalleOlaviNiemitalo
    KalleOlaviNiemitalo
    on Mar 12, 2024

    Repro by internal reflection

    using System;
using System.Diagnostics.Tracing;
using System.Net;
using System.Net.Http;
using System.Reflection;
using System.Threading;
using System.Threading.Tasks;

class Program
{
    static void Main()
    {
        Type httpTelemetryType = Type.GetType("System.Net.Http.HttpTelemetry, System.Net.Http", throwOnError: true)!;
        FieldInfo logField = httpTelemetryType.GetField("Log") ?? throw new Exception("HttpTelemetry.Log field does not exist");
        var httpTelemetry = (EventSource)logField.GetValue(null) ?? throw new Exception("HttpTelemetry.Log is null");

        // public void Http11ConnectionEstablished(long connectionId, string scheme, string host, int port, IPEndPoint? remoteEndPoint)
        Action<long, string, string, int, IPEndPoint?> http11ConnectionEstablished
            = httpTelemetryType.GetMethod("Http11ConnectionEstablished")
            .CreateDelegate<Action<long, string, string, int, IPEndPoint?>>(httpTelemetry);

        // public void Http11ConnectionClosed(long connectionId)
        Action<long> http11ConnectionClosed
            = httpTelemetryType.GetMethod("Http11ConnectionClosed")
            .CreateDelegate<Action<long>>(httpTelemetry);

        var listener = new DummyListener();
        listener.EnableEvents(httpTelemetry, EventLevel.Verbose);

        var gcInducerThread = new Thread(() =>
        {
            while (true)
            {
                Thread.Sleep(TimeSpan.FromSeconds(1));
                Console.Write("GC...");
                GC.Collect();
                Console.WriteLine("finished.");
            }
        });
        gcInducerThread.IsBackground = true;
        gcInducerThread.Start();

        Console.WriteLine("Running...");
        Parallel.For(1, 1_000_000_000, (long connectionId) =>
        {
            var remoteEndPoint = new IPEndPoint(address: connectionId, port: 80);
            http11ConnectionEstablished(connectionId, "http", $"host{connectionId}", remoteEndPoint.Port, remoteEndPoint);
            http11ConnectionClosed(connectionId);
        });
    }
}

class DummyListener : EventListener
{
}
  5. baronfel
    transferred this issue fromdotnet/sdkon Mar 12, 2024
  6. ghost added
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on Mar 12, 2024

  7. KalleOlaviNiemitalo commented on Mar 12, 2024

    @KalleOlaviNiemitalo
    KalleOlaviNiemitalo
    on Mar 12, 2024

    I guess area-System.Net.Http.

  8. vcsjones
    added
    area-System.Net.Http
    and removed
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on Mar 12, 2024
  9. MihaZupan
    added
    bug
    on Mar 12, 2024
  10. MihaZupan
    added this to the 9.0.0 milestone on Mar 12, 2024
  11. MihaZupan
    added
    tenet-reliabilityReliability/stability related issue (stress, load problems, etc.)
    and removed
    untriagedNew issue has not been triaged by the area owner
    on Mar 12, 2024

  12. MihaZupan commented on Mar 12, 2024

    @MihaZupan
    MihaZupan
    on Mar 12, 2024
    Member

    Looks like we introduced this one in 8.0: #88853
    Thanks for calling out the fixed scope issue @KalleOlaviNiemitalo

  13. MihaZupan
    closed this as completedin #99606on Mar 12, 2024
  14. MihaZupan
    reopened this on Mar 12, 2024

  15. MihaZupan commented on Mar 12, 2024

    @MihaZupan
    MihaZupan
    on Mar 12, 2024
    Member

    Reopening to track backport consideration

  16. MihaZupan commented on Mar 21, 2024

    @MihaZupan
    MihaZupan
    on Mar 21, 2024
    Member

    Fixed in main (9.0) in PR #99606, and in 8.0.5 in PR #99607.

  17. MihaZupan
    closed this as completedon Mar 21, 2024
  18. github-actions
    locked and limited conversation to collaborators on Apr 21, 2024
  19. karelz
    modified the milestones: 9.0.0, 8.0.x on Jun 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area-System.Net.Httpbugtenet-reliabilityReliability/stability related issue (stress, load problems, etc.)

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions