You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Memory management cgroup diagram does not show which config value sizes each region #6028
The cgroup diagram in memory_management, added in #6014, groups everything the container counts by who accounts for it. It does not say which configuration value sizes each region.
That leaves the two facts the section goes on to state as prose only:
spark.memory.offHeap.size is part of the pod limit, not extra headroom on top of it.
spark.executor.memoryOverhead is the only slack in the container.
Both are properties of how the configuration maps onto the regions, which is exactly the kind of
thing a diagram carries better than a sentence. The mapping is also uneven in a way that is easy to
get wrong: spark.memory.offHeap.size alone sizes three of the regions, including Comet's native
reservations, which are neither off-heap in Spark's sense nor allocated by the JVM, while spark.executor.memoryOverhead sizes none of them.
Adding the three configuration values as inputs to the existing diagram would make that visible
without introducing a third diagram to the page.
Superseded by #6237. Its diagram sits near the top of both the memory management page and the memory tuning guide, and it labels each part of the executor with the setting that sizes it: spark.executor.memory, spark.memory.offHeap.size and spark.executor.memoryOverhead. The cgroup diagram further down the page is unchanged. If that one should carry the labels too, reopen this.
Activity
andygrove commentedon Sep 26, 2026
Superseded by #6237. Its diagram sits near the top of both the memory management page and the memory tuning guide, and it labels each part of the executor with the setting that sizes it:
spark.executor.memory,spark.memory.offHeap.sizeandspark.executor.memoryOverhead. The cgroup diagram further down the page is unchanged. If that one should carry the labels too, reopen this.