Repository navigation
Huge memory usage regression in dotnet 7 on linux #79633
Description
Description
A dotnet 7 application consumes much more memory on linux than a dotnet 6 or dotnet 5 application.
After upgrading our application to dotnet 7 we noticed a huge increase in memory consumption in linux, services that used 100 MB were now using twice that amount.
The base issue can be easily reproduced
- create new console application
- make sure it does not exit instantly, e.g. by using a while loop like:
while (true)
{
Console.WriteLine("Hello, World!");
Thread.Sleep(TimeSpan.FromSeconds(1));
}- Verify that in your .csproj file you are targeting dotnet 7:
<TargetFramework>net7.0</TargetFramework>- Switch to a
Releaseconfiguration - Start it under linux, e.g. by select
WSLin the dropdown to launch it. - Verify the used memory, e.g. by running
ps auxin your wsl shell and taking a look at theRSScolumn:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 5335 2.5 0.4 273549912 103944 pts/6 Sl+ 17:17 0:00 /usr/bin/dotnet /mnt/c/projects/sources/DotNet7Test/DotNet7Test/bin/Release/net7.0/DotNet7Test.dll
- Realize in horror that the console process consumes 100+ MB
- Stop the debugging
- Target dotnet 6 by updating your csproj file to:
<TargetFramework>net6.0</TargetFramework>- Start it under linux, e.g. by select
WSLin the dropdown to launch it. - Verify the used memory, e.g. by running
ps auxin your wsl shell and taking a look at theRSScolumn:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 5383 1.0 0.0 2961956 25504 pts/6 Sl+ 17:21 0:00 /usr/bin/dotnet /mnt/c/projects/sources/DotNet7Test/DotNet7Test/bin/Release/net6.0/DotNet7Test.dll
- Wonder why the same process consumes only 25MB using dotnet 6.
Configuration
Dotnet 7 on Linux x64, see above
Regression?
Yes, this is a major performance regression in dotnet 7 compared to dotnet 5 or 6.
Other information
Notes:
The memory consumption on windows on dotnet 7 is reasonable - it's just a little higher, but not by crazy amounts.
The difference between dotnet 5 and dotnet 6 on linux is negliable.
Take care,
Martin
Reacted by Damian Pozimski, Bartłomiej Gaweł, shaosu, dependabotkoro, Wojciech Rak, Piotr Jakóbczyk, Damian, l-baumgartner, Piotr Oginski, Soheil Alizadeh and 3 more
Reacted by Symbai
Reacted by Wojciech Rak
Activity
buyaa-n commentedon Dec 14, 2022
CC @dotnet/linux-team
mangod9 commentedon Dec 14, 2022
Hi @8,
yeah this is something we are aware of for unconstrained memory environments. We are working on resolving it in a upcoming servicing release, a workaround is to set
DOTNET_GCHeapHardLimitto something reasonable (based on expected app usage or physical memory). Its an absolute increase of64mbhence its exaggerated for hello world style apps.It also shouldnt be an issue for memory constrainted containers
enclave-alistair commentedon Dec 14, 2022
Just to add a note that we deploy a single-file self-contained linux application to end-users machines, so cannot realistically set
DOTNET_GCHeapHardLimit, and an unrequired increase of64mbwill no doubt raise significant complaints.mangod9 commentedon Dec 14, 2022
Ok, will update this issue once the fix is merged. Thanks
EricStG commentedon Dec 16, 2022
We have a similar issue.
We upgraded 2 services to .NET 7 / ASPNET Code 7, both at least doubled memory usage (one went from a max of 15% memory usage to 80%)
Both are running in container with a 512mb limit (in Cloud Run if that matters)
mangod9 commentedon Dec 16, 2022
this issue shouldn't be affecting constrained memory containers (in fact, our TechEmpower measurements show a reduction in working set). Would you be able to share more details and/or a dump to investigate the increase further?
EricStG commentedon Dec 16, 2022
@mangod9 Absolutely, let me know what you need
6 remaining items
ilia-cy commentedon Feb 5, 2023
Hello @mangod9, are there any updates regarding the fix?
mangod9 commentedon Feb 6, 2023
Hi @ilia-cy, this should be fixed as part of #80173, so should be included in 7.0.3 which should be released next week. Thx.
idan-cyber commentedon Feb 14, 2023
@mangod9 - Any updates on version 7.0.3? Thanks!
richlander commentedon Feb 14, 2023
It should ship one month from today, 3/14.
Would you test with a pre-release build to validate?
I never realized that Pi day is one month after Valentines.
8 commentedon Feb 15, 2023
Hi Rich,
super!
yes, I would - if you have a download link that would be great!
Take care,
Martin
idan-cyber commentedon Mar 12, 2023
https://versionsof.net/core/7.0/7.0.3/ - Thanks @richlander!
declard commentedon Mar 14, 2023
The issue doesn't seem to be fixed in either 7.0.3 or 7.0.4
The following is a screenshot of htop with the dotnet6 version of our app running on a machine:
Compare it to the screenshot of htop with the dotnet7 version of the same app running on the same machine and the same data set:
The executable is a generic host console app, it runs
in both versions after finishing loading the data.
Heap analysis didn't show any object size/count difference.
Such threefold increase in memory is unacceptable when an app consumes a hundred gigs in prod environments.
mangod9 commentedon Mar 15, 2023
Hi @declard, your issue seems to be different than the original posted here. That was regarding a vanilla app consuming more memory. I suggest you create a new issue and provide some traces to compare 6 with 7. Would also be good to check if setting
DOTNET_GCName=libclrgc.sohas any effect on your scenario. Thx!declard commentedon Mar 15, 2023
@mangod9 thank you for the advice, it actually worked! The memory dropped back to dotnet6 size, so I guess the issue is in the new GC indeed.
I'll dig further to file a new issue when time allows
8 commentedon Mar 15, 2023
After the update I get:
After rebuilding and running the executable I can see the memory usage dropped from 100MB to 30MB:
Thank you for the fix!
We can now complete the upgrade of our services to dotnet 7, thank you!
Take care,
Martin
mangod9 commentedon Mar 15, 2023
Thanks for validating @8 . @declard, yeah please open a separate issue so we could investigate your specific scenario.
mihaimyh commentedon Apr 6, 2023
Wow, this actually worked also for my app. The memory dropped back to .NET 6 levels after settings this variable. What does actually do?
mangod9 commentedon Apr 7, 2023
@mihaimyh , the config moves the GC mechanism back to .NET 6. I would suggest creating a separate issue since your issue would most likely be different that the original one here. If you need notice working set differences with this setting would be good to collect some traces so we could investigate further. Thx!