pummel/test-heapdump-shadow-realm.js is flaky due to OOM

[joyeecheung]

Test

pummel/test-heapdump-shadow-realm.js

Platform

Linux x64

Console output

11:41:28 not ok 3651 pummel/test-heapdump-shadow-realm
11:41:28   ---
11:41:28   duration_ms: 72141.13100
11:41:28   severity: crashed
11:41:28   exitcode: -6
11:41:28   stack: |-
11:41:28     
11:41:28     <--- Last few GCs --->
11:41:28     0.[3846001:0x61f1c60]    46446 ms: Mark-Compact (reduce) 977.0 (993.2) -> 976.4 (993.7) MB, 1058.93 / 0.00 ms  (+ 78.7 ms in 13 steps since start of marking, biggest step 13.7 ms, walltime since start of marking 1165 ms) (average mu = 0.260, current mu = 0.[3846001:0x61f1c60]    47642 ms: Mark-Compact (reduce) 977.1 (993.7) -> 976.5 (993.9) MB, 892.02 / 0.00 ms  (+ 65.2 ms in 9 steps since start of marking, biggest step 16.2 ms, walltime since start of marking 978 ms) (average mu = 0.234, current mu = 0.200
11:41:28     
11:41:28     <--- JS stacktrace --->
11:41:28     
11:41:28     FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
11:41:28      1: 0xca2d40 node::Abort() [out/Release/node]
11:41:28      2: 0xb7f1ab  [out/Release/node]
11:41:28      3: 0xee4ff0 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [out/Release/node]
11:41:28      4: 0xee53ac v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [out/Release/node]
11:41:28      5: 0x10fa1e5  [out/Release/node]
11:41:28      6: 0x111095d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [out/Release/node]
11:41:28      7: 0x10e875a v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [out/Release/node]
11:41:28      8: 0x10e92f5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [out/Release/node]
11:41:28      9: 0x10c72fe v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [out/Release/node]
11:41:28     10: 0x1516e42 v8::internal::Runtime_AllocateInOldGeneration(int, unsigned long*, v8::internal::Isolate*) [out/Release/node]
11:41:28     11: 0x1951ef6  [out/Release/node]
11:41:28   ...

Build links

nodejs/reliability#660

First CI: https://ci.nodejs.org/job/node-test-pull-request/53749/
Last CI: https://ci.nodejs.org/job/node-test-pull-request/53778/

Additional information

cc @legendecas At first glance I think we could lower the number of shadow realms created in the test. But then the realms are supposed to be GC-able anyway...perhaps we should switch to setImmediate() instead to give GC some time to kick in?

[legendecas]

I'm not aware that weak callbacks for ShadowRealm are deferred to prevent it from being collected. Since this is related to the heapdump specifically (there is no reliability report on test/parallel/test-shadow-realm-gc.js), I'm wondering if this might be related to insufficient memory for the heapdump operation.

[joyeecheung]

Normally heapdump should not have more JS heap overhead than the size of a semi-space (because of promotion) + a little bit of margin for cached source positions (can't be that much for ShadowRealms that only have one tiny script). It could be though that because the weak callbacks are deferred, they are still recognized as reachable when the heap snapshot is taken and thus still in the heap snapshot which means the heap snapshot can get bloated a lot (and a huge heap snapshot in itself can eat a lot of memory because we serialize it into JS on the same thread for test verification). We can see if the setImmediate() trick deflakes it and if it doesn't, consider adding some logging to the test to see what's causing the flake.

[joyeecheung]

Looks like this disappeared from the CI. Closing for now. We can reopen if it reappears.

[targos]

I think it's back: https://ci.nodejs.org/job/node-test-pull-request/54396/

[targos]

Another one: https://ci.nodejs.org/job/node-test-pull-request/54404/

[joyeecheung]

Oh, my, this failed18 PRs across the last 100 CI runs. nodejs/reliability#681 looks like it started from 09-29

[joyeecheung]

I am guessing something landed around that time introduced a memory leak in the realms?

[joyeecheung]

Actually it seems this time around it's a different bug. It's STATUS_STACK_BUFFER_OVERRUN on Windows (or it could be other assertion failures, https://devblogs.microsoft.com/oldnewthing/20190108-00/?p=100655)