Repository navigation
Config naming audit: rename inconsistent spark.comet.* keys before 1.0 #4978
Description
Activity
- added a commit that references this issue
on Jul 28, 2026 Closing this out ahead of 1.0. Most of the audit landed; the remainder is deliberately will not do.
Done
- Category 1 —
spark.comet.exec.replaceSortMergeJoin→spark.comet.exec.forceShuffledHashJoin(the pilot rename). - Category 2 — the dot-separated keys are gone: no
spark.comet.columnar.shuffle.async.*orspark.comet.parquet.read.parallel.io.thread-pool.sizeremain. The UDF casing split is fixed in refactor: rename pyarrowUdf config to pyarrowUDF and fix stale config docs #5197 (pyarrowUdf→pyarrowUDF, no alias needed — the key never shipped). - Category 3 —
spark.comet.explain.*grouped in refactor: group orphanspark.comet.explain.*configs under one prefix #5026. - Category 4 — shuffle prefixes unified under
spark.comet.shuffle.*in refactor: unify shuffle configs underspark.comet.shuffle.*prefix #4986.
Also folded into #5197:
config_conventions.mdhad drifted from the code (it still listedcolumnaras a category segment, and its symbol-naming table named a symbol that does not exist).Will not do
Category 5 —
spark.comet.sparkToColumnar.*. The two candidate targets both have real problems. Folding intospark.comet.convert.*overloads a prefix that currently means source-scan conversion (convert.parquet,convert.json,convert.csv) with a different concept, converting arbitrary Spark operators to Arrow. Renaming the segment tosparkColumnaris churn for pure aesthetics. Not a call worth making on release week.Category 6 — top-level scalars. These are not inconsistencies:
spark.comet.memoryOverheaddeliberately mirrors Spark's ownspark.executor.memoryOverhead.spark.comet.maxTempDirectorySizemirrors DataFusion'smax_temp_directory_size.spark.comet.batchSizeis probably the most widely cited Comet config in tuning guides, blog posts, and benchmark scripts.
spark.comet.exceptionOnDatetimeRebase. Restructuring a shipped, compatibility-sensitive key is not worth it. Appending.enabledalone is cosmetic.spark.comet.scan.allowDisabledParquetVectorizedReader. Already compliant — the conventions guide exempts action-formallow…/force…flags from the.enabledsuffix.The reasoning on timing
Under the versioning policy, config keys are Comet's primary public API. An alias added now and one added in 1.3 are both removable only in 2.0, so alias lifetime is not the deciding factor — user churn is. Renames at the 0.x→1.0 boundary are one migration, documented once. The same renames mid-1.x hit users who already migrated with fresh deprecation warnings, against names by then baked into 1.0-era docs and blog posts. That makes the cheap renames worth doing now and the expensive ones not worth doing at all, rather than leaving them open as "post-1.0".
Two smaller
.enabledsuffix gaps remain unaddressed (spark.comet.nativeLoadRequired,spark.comet.exec.strictFloatingPoint). They are legitimate but low-value, and both would now need aliases. Happy to reopen as a focused issue if anyone feels strongly.- Category 1 —
- added 5 commits that reference this issue
on Aug 26, 2026
Tracker for #4082 "Review all configuration options and consider renaming some for consistency."
An audit of the 54 keys in
spark/src/main/scala/org/apache/comet/CometConf.scalasurfaced six categories of inconsistency. This issue is the umbrella; individual rename PRs should each cover one cluster and use the.withAlternative(...)mechanism to keep old keys working as deprecated aliases (see the Configuration Conventions contributor-guide page).Category 1 — Boolean flags missing the
.enabledsuffixCurrent convention: boolean flags end in
.enabled. Exceptions to rename:spark.comet.nativeLoadRequired→spark.comet.nativeLoadRequired.enabledspark.comet.exceptionOnDatetimeRebase→spark.comet.datetimeRebase.exceptionOnRead.enabled(or similar; the current name is also very long)spark.comet.exec.replaceSortMergeJoin→spark.comet.exec.forceShuffledHashJoin(done as the pilot rename)Category 2 — Word separator inside a segment
Most segments use
camelCase; a handful use dots or dashes to separate words. Fix:spark.comet.columnar.shuffle.async.max.thread.num→spark.comet.columnar.shuffle.async.maxThreadNumspark.comet.columnar.shuffle.async.thread.num→spark.comet.columnar.shuffle.async.threadNumspark.comet.parquet.read.parallel.io.thread-pool.size→spark.comet.parquet.read.parallel.io.threadPoolSizespark.comet.exec.pyarrowUdf.enabledvsspark.comet.exec.scalaUDF.codegen.enabled— pick one casing for UDF and apply consistently.Category 3 —
.explain.*group has orphaned siblingsspark.comet.explain.*already exists (format,native.enabled,rules). These related keys live at the top level:spark.comet.explainCodegen.enabled→spark.comet.explain.codegen.enabledspark.comet.explainFallback.enabled→spark.comet.explain.fallback.enabledspark.comet.logFallbackReasons.enabled→ related concept, current name is orphaned; discuss placement (spark.comet.explain.fallback.log.enabled?spark.comet.fallback.log.enabled?).Category 4 — Two shuffle prefixes
spark.comet.columnar.shuffle.*(async/batch/memory/spill)spark.comet.native.shuffle.*(partitioning)Users likely expect
spark.comet.shuffle.columnar.*andspark.comet.shuffle.native.*. This is the largest rename cluster and is worth its own thread.Category 5 —
sparkToColumnaruses camelCase as a top-level categoryspark.comet.sparkToColumnar.*is the only categorical prefix using camelCase (others are lowercase words:exec,scan,parquet,shuffle). Options:spark.comet.convert.*family:spark.comet.convert.spark.enabled.spark.comet.sparkColumnar.*.Category 6 — Top-level scalars that could join a category
spark.comet.batchSizespark.comet.memoryOverhead(candidate:spark.comet.memory.overhead)spark.comet.maxTempDirectorySize(candidate:spark.comet.tempDirectory.maxSize)Lower priority — these are top-level scalars that pre-date the current category convention.
Process
Each rename PR:
.withAlternative("oldKey")on theconf(...)call for the new key.val(grepCometConf.for callsites).docs/source/for the old key string).CometConfSuiteif the rename exercises a new alias pattern.Alias removals happen no earlier than the next Comet major after the rename first ships (see the versioning policy).