Response strings and closures objects leak with http.Agent enabled (Memory Leak)

[snow01]

• Version: v6.9.1
• Platform: Linux ip-172-31-46-151 4.4.11-23.53.amzn1.x86_64 deps: update openssl to 1.0.1j #1 SMP Wed Jun 1 22:22:50 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
• Subsystem: Http(s) Agent

http.Agent based sockets pool is leading to memory leaks. With http.Agent enabled, both response strings and response handlers (closures) are not getting released so quickly -- all these objects and closures are getting promoted to old generation. Even on a normal load of 2-3 requests per second, garbage collection is not able to free these promoted objects an closures, application's memory usage reaches to 1GB in just 10-12 hours of run -- eventually application crashes due to out of memory.

Pool settings are normal --

new http.Agent({
        keepAlive: true,
        maxSockets: 10,
        maxFreeSockets: 5,
        keepAliveMsecs: 5000
    })

I have done comparison of heap snapshots at various time points to reach to this conclusion. I have even verified that with default agent settings (that is http.globalAgent), I do not see any such behaviour - application memory remains consistent at ~90-95Mb

Below are screenshots from chrome developer tools for various scenarios --

1. With http.Agent enabled, strings delta is 10 Mb. From Retainers tab it is clear that these objects are from http requests.

2. With http.Agent enabled, closures delta is 0.5 Mb. From Retainers tab it is clear that these closures are from http requests.

3. Without http.Agent, there is no positive delta for strings and closures

• I can see similar issue was discussed and reported for 0.x version of NodeJS too - Memory leak using http's request/get/clientRequest and closures node-v0.x-archive#3179
• One of the discussion also suggests that keeping agent disabled worked - Filirom1/node@6730cdc

Please guide how can I help quick resolution of this issue.

[snow01]

Any update on this issue ?

[bnoordhuis]

You should post a standalone (no third-party modules) test case if you want people to take a look.

[snow01]

@bnoordhuis - is http.Agent a third party module for NodeJS ? Are there any existing http performance tests I can adapt ?

[bnoordhuis]

http.Agent is built-in. You can find performance tests in benchmark/http.

[andrasq]

I recently came across this same issue (memory leak in 6.9.1 that's not there in 6.2.2) that I too
traced to _http_agent. On the above screenshots, notice how the 'timeout' event handlers array
in Agent.freesockets localhost:9200 is 173 entries deep.

I then came across PR#9440 (#9440) which fixes it.
Turns out _http_client is re-adding a timeout listener on reused sockets on every request.
The PR is a small fix that removes the timeout listener on response end.

Agent is not at fault, but it is the one that's retaining the sockets. The sockets accrue callbacks,
and the callbacks are bound to the (annotated by the app, possibly very large) request objects.

[Trott]

I infer from the fact that #9440 landed that this can be closed. Please let me know if that's incorrect. Closing now. Thanks!