Repository navigation
Moving to dotnet 8 causes a sudden increase in the LOH size. #101906
Description
Description
We have an application that reads messages from Azure Service Bus and process them. The processing pipeline is based on Microsoft TPL Dataflow library and it works well in dotnet 6 and 7. The application processes around 2-3K messages per second.
Unfortunately when moving to dotnet 8 we have been experimenting some memory problems. In this issue, I am focusing in a sudden growth of the Large Object Heap.
We have another ongoing issue regarding to dotnet 8 and a memory leak.
We created a new build of the application only changing the target framework and ran them side by side. We can see how the LOH gets very big from the get-go:
Violet: dotnet 7
Orange: dotnet 8
Configuration
Docker Image: mcr.microsoft.com/dotnet/aspnet:8.0.3
Server: Standard_F16s_v2
Host:
Version: 8.0.3
Architecture: x64
Commit: 9f4b1f5d66
RID: linux-x64
.NET SDKs installed:
No SDKs were found.
.NET runtimes installed:
Microsoft.AspNetCore.App 8.0.3 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 8.0.3 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
Regression?
Yes. We ran the two versions side by side ensuring the only difference was the dotnet version used and it was easily reproducible.
Data
Using dotnet-dump in both servers, we can easily see the problem is that System.Char[] at the bottom, which does not appear in the application running with dotnet 7.
> dumpgen loh
Statistics:
MT Count TotalSize Class Name
00007f8e651e89e0 1 98352 System.Int32[]
00007f8e6f212518 1 131096 StumbleBackend.Core.Code.InactiveUsers.InactiveUser[]
00007f8e68dc21a0 2 404160 System.Collections.Generic.Dictionary<System.Int32, System.IO.FileSystemWatcher+RunningInstance+WatchedDirectory>+Entry[]
00007f8e6a927b08 2 524336 System.Collections.Concurrent.ConcurrentQueueSegment<StatsdClient.Statistic.Stats>+Slot[]
00007f8e6be74c20 32 7340800 StackExchange.Redis.RawResult[]
00007f8e657c7d70 224 67304490 System.Byte[]
0000561da6889480 1085 138226408 Free
00007f8e651ed7c8 126 154194278 System.String
00007f8e652b63b8 697 1631207768 System.Char[]
Total 2170 objects
Picking some of them we saw that those big arrays belongs to a SharedArrayPool:
> gcroot 7f4f1f052380
HandleTable:
00007f8ee2375828 (strong handle)
-> 7f4eb159cd20 System.Object[]
-> 7f4eb159cf58 System.Buffers.SharedArrayPool<System.Char>+ThreadLocalArray[]
-> 7f4f1f052380 System.Char[]
However nothing in our code uses an array pool of chars. We use ArrayPool<byte> in many places, but not for chars.
Any idea what could be the problem? or how to find out who is using System.Buffers.SharedArrayPool<System.Char>?
Cheers.
Activity
huoyaoyuan commentedon May 6, 2024
Pretty a lot of internal
stringmanipulating methods are usingArrayPool<char>when appropriate.dotnet-policy-service commentedon May 6, 2024
Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.
mangod9 commentedon May 6, 2024
@vtortola, this could be related to an issue fixed by: #100502. The issue has a workaround and should be fixed in 8.0.5. Can you try the workaround and check if that helps your scenario?
vtortola commentedon May 6, 2024
@mangod9 the workaround is just switching to alpine, correct?
BTW when is 8.0.0 released more or less?
mangod9 commentedon May 6, 2024
yeah workaround is to try Alpine or set
DOTNET_GCgen0Size=0x1E00000. 8.0.5 would be released in the next couple of weeks when the workarounds wouldn't be necessary assuming you are hitting the same issue.vtortola commentedon May 10, 2024
For the moment we are going to wait for 8.0.5. We will try to schedule some time to test that workaround with the environment variable, but I cannot tell when, if ever. Cheers.
debracey commentedon May 19, 2024
I tried the
DOTNET_GCgen0Sizeenvironment variable and outlined my results here.tl;dr This does not fix the issue, at least not completely
jbroumels commentedon Jun 5, 2024
shouldn't
DOTNET_GCgen0SizebeDOTNET_GCgen0size? (lowercase s)vtortola commentedon Jun 9, 2024
We decided to wait for 8.0.5 and indeed it fixed the problem. Many thanks.