A bare byte count for `spark.memory.offHeap.size` is read as MiB when sizing the Comet memory pool

[andygrove]

Describe the bug

Spark reads spark.memory.offHeap.size as bytes unless a unit is given. CometExecIterator.getMemoryConfig reads it with SparkConf.getSizeAsMb, which treats a bare number as MiB (CometExecIterator.scala#L617). The native memory usage log in the same file reads it correctly with getSizeAsBytes (CometExecIterator.scala#L498).

With spark.memory.offHeap.size=4294967296, Spark's off-heap pool is 4 GiB, but Comet computes memory_limit as 4294967296 MiB, about 4 PiB. The fair_unified pool caps each task at memory_limit / num_consumers, so the cap never binds: fair_unified behaves like greedy_unified, and spark.comet.exec.memoryPool.fraction has no effect. Spark still enforces its own pool size, so Comet cannot acquire more than 4 GiB.

Steps to reproduce

Call CometExecIterator.getMemoryConfig with spark.memory.offHeap.enabled=true and spark.memory.offHeap.size=4294967296. memoryLimit comes back as 4294967296 x 2^20 bytes instead of 4294967296.

Expected behavior

memoryLimit is 4 GiB, matching Spark's pool.

Additional context

Fix: read the value with getSizeAsBytes, and add a test that uses a bare byte count.