dotnet-dump makes process to double its used memory and fails #71472

Activity

1. ghost added

   area-Diagnostics-coreclr

   on Jun 30, 2022
2. ghost added

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Jun 30, 2022
3. afilatov-st

   changed the title [-]dotnet-dump makes process to double its used memory and fail[/-] [+]dotnet-dump makes process to double its used memory and fails[/+] on Jun 30, 2022
4. tommcdon commented on Jun 30, 2022

   

   tommcdon

   on Jun 30, 2022

   Member

   More actions

   @afilatov-st Thanks for the bug report! I do not believe that memory doubling is expected in this scenario (though some memory usage is expected). Dotnet-Dump sends an IPC command using a domain socket on Linux to the target process to collect a dump. The target process will then launch createdump as a child process to collect a dump of the parent process. When the memory is doubled - is it the target process's memory that increases, createdump, or dotnet-dump itself that uses the extra memory?
   @mikem8361 @hoyosjs
5. afilatov-st commented on Jun 30, 2022

   

   afilatov-st

   on Jun 30, 2022

   Last edited by afilatov-st

   Author

   More actions

   @tommcdon thanks for the prompt response!
   the crash happens pretty quickly, so I had to run the following script in a parallel session

   while true                      
   do 
     ps aux;
     sleep 0.5;
   done
   

   it shows that the RSS of the target dotnet process is in its initial value of 3.6 Gb for around 5 seconds, then it quickly grows to 6.2 Gb before Kubernetes kills it.
6. afilatov-st commented on Jun 30, 2022

   

   afilatov-st

   on Jun 30, 2022

   Author

   More actions

   For the context, the Docker image is based on mcr.microsoft.com/dotnet/aspnet:6.0.5-bullseye-slim-amd64
7. agocke

   added this to Runtime Infra and removed this from Runtime Infraon Jul 1, 2022
8. mikem8361

   self-assigned this

   on Jul 6, 2022
9. tommcdon

   removed

   untriagedNew issue has not been triaged by the area ownerNew issue has not been triaged by the area owner

   on Jul 7, 2022
10. tommcdon

    added this to the 7.0.0 milestone on Jul 7, 2022
11. mikem8361 commented on Jul 12, 2022

    

    mikem8361

    on Jul 12, 2022

    Contributor

    More actions

    Can you try a heap dump by add --type Heap to the dotnet-dump collect command line?

    We think maybe when createdump is reading memory to write pages to the dump file that causes them to be "swapped" back in or read from the module files into the target process memory. A heap dump doesn't touch/read most of the module pages.
12. mikem8361

    added

    needs-author-actionAn issue or pull request that requires more info or actions from the author.An issue or pull request that requires more info or actions from the author.

    on Jul 12, 2022
14. afilatov-st commented on Jul 12, 2022

    

    afilatov-st

    on Jul 12, 2022

    Author

    More actions

    --type Heap behaves in the same way
15. 59 remaining items

    Load more
16. tommcdon

    added this to the 9.0.0 milestone on Jul 17, 2023
17. ezgambac commented on Oct 31, 2023

    

    ezgambac

    on Oct 31, 2023

    Contributor

    More actions

    @tommcdon @hoyosjs Is there an eta for having heap analyzers fixed?
    This doubling memory issue makes dotnet dump unusable in the required scenarios, like debugging why is there high memory, as k8s will kill the pod.
18. hoyosjs commented on Oct 31, 2023

    

    hoyosjs

    on Oct 31, 2023

    Member

    More actions

    @ezgambac Even in the case this got improved (the change wasn't backed out; it's just flagged off since it will make commands like verifyheap in SOS fail), it will still force memory swapping and some growth since the dumper itself runs in the cgroup of the container. For OOM, there's other options that could work since they are started in the init process's context if you have access to the host.
19. ezgambac commented on Oct 31, 2023

    

    ezgambac

    on Oct 31, 2023

    Contributor

    More actions

    @hoyosjs What do you suggest doing then for the scenario where the pod is running at 80% memory then?
    We currently have dotnet monitor 6, which uses an older version of dotnet dump, but from what you are saying, even if we moved to latest, the dumper would generate enough extra memory that k8s will kill the process?
    From following this thread, it seemed like the change @ezsilmar had significantly reduced the memory consumption by dotnet dump while getting a dump. Would this dump be analyzable by perfview/visual studio?
20. hoyosjs commented on Oct 31, 2023

    

    hoyosjs

    on Oct 31, 2023

    Member

    More actions

    It's analyzable, but you might get tooling telling you that the heap is inconsistent. You need to deploy the app with DOTNET_DbgDisablePagemapUse=0 and this is only there in .NET 8. Do you have access to the host (node)?
21. tommcdon

    modified the milestones: 9.0.0, 10.0.0 on Jul 23, 2024
22. tommcdon

    unassigned

    hoyosjs

    on Aug 20, 2024
23. noahfalk

    added 2 commits that reference this issue on Apr 17, 2025

    Warn about dotnet-dump memory usage

    ...

    eb347dc

    Warn about dotnet-dump memory usage (#45841)

    ...

    0e92e7f
24. noahfalk

    added

    enhancementProduct code improvement that does NOT require public API changes/additionsProduct code improvement that does NOT require public API changes/additions

    on Apr 17, 2025
25. steveisok

    modified the milestones: 10.0.0, 11.0.0 on Jul 30, 2025
26. noahfalk

    modified the milestones: 11.0.0, 12.0.0 on Jul 28, 2026