Skip to content

Memory management cgroup diagram does not show which config value sizes each region #6028

Closed

Description

@andygrove

The cgroup diagram in memory_management, added in #6014, groups everything the container counts by who accounts for it. It does not say which configuration value sizes each region.

That leaves the two facts the section goes on to state as prose only:

  1. spark.memory.offHeap.size is part of the pod limit, not extra headroom on top of it.
  2. spark.executor.memoryOverhead is the only slack in the container.

Both are properties of how the configuration maps onto the regions, which is exactly the kind of
thing a diagram carries better than a sentence. The mapping is also uneven in a way that is easy to
get wrong: spark.memory.offHeap.size alone sizes three of the regions, including Comet's native
reservations, which are neither off-heap in Spark's sense nor allocated by the JVM, while
spark.executor.memoryOverhead sizes none of them.

Adding the three configuration values as inputs to the existing diagram would make that visible
without introducing a third diagram to the page.

Activity

  1. andygrove commented on Sep 26, 2026

    @andygrove
    MemberAuthor

    Superseded by #6237. Its diagram sits near the top of both the memory management page and the memory tuning guide, and it labels each part of the executor with the setting that sizes it: spark.executor.memory, spark.memory.offHeap.size and spark.executor.memoryOverhead. The cgroup diagram further down the page is unchanged. If that one should carry the labels too, reopen this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions