Describe the bug
The executor's native memory usage log (#6162) warns when untracked native memory plus Spark's off-heap usage exceeds spark.memory.offHeap.size plus the memory overhead. CometExecIterator.executorMemoryOverhead takes the overhead from spark.executor.memoryOverhead. If that is unset, it uses spark.executor.memoryOverheadFactor (default 0.1) of the executor memory, with a minimum of spark.executor.minMemoryOverhead (CometExecIterator.scala#L465-L483).
On Kubernetes, Spark does not size the executor pod this way when spark.executor.memoryOverheadFactor is unset. BasicExecutorFeatureStep falls back to spark.kubernetes.memoryOverheadFactor (BasicExecutorFeatureStep.scala#L66-L70). For PySpark and SparkR applications in cluster mode, BasicDriverFeatureStep sets that factor to 0.4 and passes it on to the executors (BasicDriverFeatureStep.scala#L50-L63). For those applications Comet's limit is too low, and the executor warns that the cluster manager may kill it while it is still inside its pod. A user-set spark.kubernetes.memoryOverheadFactor is ignored in the same way.
The tuning guide documents the 0.4 default (tuning.md#L163-L165), yet also says the warning sizes the overhead "as Spark sizes the default container" (tuning.md#L234-L237).
A standalone cluster has no container limit, but it gets the same warning text. Only local mode is excluded.
Steps to reproduce
Submit a PySpark application to Kubernetes in cluster mode with spark.executor.memory=8g, spark.memory.offHeap.enabled=true, spark.memory.offHeap.size=4g and no overhead settings. The pod gets 3276 MiB of overhead (0.4 x 8 GiB), but the log compares against 819 MiB (0.1 x 8 GiB).
Expected behavior
The overhead is resolved the same way the cluster manager resolves it. Standalone clusters either get no container warning or a differently worded one.
Additional context
Python applications also add spark.executor.pyspark.memory to the pod. The log leaves it out, which only makes the warning fire earlier.
Describe the bug
The executor's native memory usage log (#6162) warns when untracked native memory plus Spark's off-heap usage exceeds
spark.memory.offHeap.sizeplus the memory overhead.CometExecIterator.executorMemoryOverheadtakes the overhead fromspark.executor.memoryOverhead. If that is unset, it usesspark.executor.memoryOverheadFactor(default 0.1) of the executor memory, with a minimum ofspark.executor.minMemoryOverhead(CometExecIterator.scala#L465-L483).On Kubernetes, Spark does not size the executor pod this way when
spark.executor.memoryOverheadFactoris unset.BasicExecutorFeatureStepfalls back tospark.kubernetes.memoryOverheadFactor(BasicExecutorFeatureStep.scala#L66-L70). For PySpark and SparkR applications in cluster mode,BasicDriverFeatureStepsets that factor to 0.4 and passes it on to the executors (BasicDriverFeatureStep.scala#L50-L63). For those applications Comet's limit is too low, and the executor warns that the cluster manager may kill it while it is still inside its pod. A user-setspark.kubernetes.memoryOverheadFactoris ignored in the same way.The tuning guide documents the 0.4 default (tuning.md#L163-L165), yet also says the warning sizes the overhead "as Spark sizes the default container" (tuning.md#L234-L237).
A standalone cluster has no container limit, but it gets the same warning text. Only local mode is excluded.
Steps to reproduce
Submit a PySpark application to Kubernetes in cluster mode with
spark.executor.memory=8g,spark.memory.offHeap.enabled=true,spark.memory.offHeap.size=4gand no overhead settings. The pod gets 3276 MiB of overhead (0.4 x 8 GiB), but the log compares against 819 MiB (0.1 x 8 GiB).Expected behavior
The overhead is resolved the same way the cluster manager resolves it. Standalone clusters either get no container warning or a differently worded one.
Additional context
Python applications also add
spark.executor.pyspark.memoryto the pod. The log leaves it out, which only makes the warning fire earlier.