Kubernetes guide example never enables Comet because it sets no off-heap memory

[andygrove]

Describe the bug

The Spark Operator example in the Kubernetes guide (kubernetes.md#L61-L100) enables Comet through spark.plugins, but it does not set spark.memory.offHeap.enabled or spark.memory.offHeap.size. CometDriverPlugin therefore logs "Comet plugin is disabled because Spark is not running in off-heap mode", and the application runs without Comet. The example also sets no spark.executor.memoryOverhead, which the tuning guide says Comet needs. Kubernetes is where that matters most, because the pod limit is enforced.

There are two more problems with the same example. It mixes versions: the image and the extra class path are Spark 3.5 with Scala 2.12, but mainApplicationFile is spark-examples_2.13-4.1.3.jar and sparkVersion is 4.1.3. And SparkPi only uses RDDs, so it would not exercise Comet even with the plugin loaded.

Expected behavior

The example sets off-heap memory and the memory overhead, uses consistent versions, and runs a workload that Comet accelerates.

[andygrove]

#6168 fixed the off-heap settings, the memory overhead and the version mix. What's left is the workload. SparkPi only uses RDDs, so it never runs through Comet, and a small DataFrame or SQL job would show Comet working.