Repository navigation
Windows Service memory grows indefinitely with Tracing and Profiling enabled. #3375
Description
From jforward5 on Discord.
I've added Sentry to my .Net core project and see the project using an infinite amount of memory. My question is, how do i make it stop using all the memory?
If I comment out this code my app runs normally, and only uses about 24MB of memory, but uncommenting causes the memory to grow endlessly by 2-3MB per second:
builder.ConfigureLogging(loggingBuilder =>
{
loggingBuilder.AddSentry(o =>
{
o.Dsn = "https://my_dsn/";
o.Debug = true;
o.TracesSampleRate = 1.0;
o.ProfilesSampleRate = 1.0;
o.AddIntegration(new ProfilingIntegration(
TimeSpan.FromMicroseconds(500)
));
});
});I should also mention this is a Windows Service using .Net Core 8
Demo project to reproduce
i've attached a sample app that reproduces the issue. The app is a plain, "empty" windows service. It is using .Net 8 and Sentry. Run the app from VS and watch your task manager, you will see the process memory grow as long as the service is running. If you turn off the profiling and tracing it no longer consumes memory, and stabilizes. You'll need to add your own "DSN" since i removed mine, for security reasons.
Metadata
Metadata
Assignees
Labels
Projects
- Status
Activity
jamescrosswell commentedon May 21, 2024
So there does appear to be an issue here. This is what I'm seeing when I run the service in a memory profiler:
Although the LOH and POH do get cleaned up occassionally (somewhat), over time these just keep growing and growing. Heap Gen 0, 1 and 2 all appears stable.
The only thing that I can think might be using the LOH or POH is profiling, which might need to pin objects when passing pointers to unsafe code. I see a number of instances where the PerView module pins objects (e.g. here).
That's consistent with the call tree, which indicates 99% of the memory is being used by instances of
EventMarker, which is an internal class used bymodules/perfview/src/TraceEvent/EventPipe/EventCache.cs:The
EventMarkerconstructor takes aPinnedBufferas an argument.It looks like the ProcessEventBlock method creates multiple
EventMarkerinstances and never explicitly cleans any of these up.I'll play around to see if I can explicitly free these up.
jamescrosswell commentedon May 21, 2024
When I look at the generations I can see:
Pretty much all of that is accounted for by
Microsoft.Diagnostics.Tracing.Etlx.TraceLog+EventsToStackIndex[]I see events get added to that stack here but the only place I see them being removed is in the internal FlushRealtimeEvents method, which in turn gets called by TraceLogProcess, which we call here when starting the profiling:
sentry-dotnet/src/Sentry.Profiling/SampleProfilerSession.cs
Line 88 in ca3744b
As far as I can tell then, that GrowableArray just gets bigger and bigger, and there are no public APIs that we can use to flush out old item.
@vaind any ideas?
vaind commentedon May 23, 2024
@jamescrosswell any chance you could verify the fix in #3382 ?
jamescrosswell commentedon May 23, 2024
It works - thanks @vaind!🕺🏻
vaind commentedon May 23, 2024
thank you!
for reference: this was the actual fix: microsoft/perfview@36d2e2c#diff-31ffbfdad96fc84f3aa7d25e1980cdca5a4d7870529c669ac9cc975cf24969c8R342