perf: [branch-1.1] speed up string, map and array conversion in JVM columnar shuffle (#6604)

[andygrove]

Which issue does this PR close?

Closes #6596 on branch-1.1. #6604 already closed it on main.

Rationale for this change

This backports #6604 to branch-1.1. It is a performance change, not a bug fix, so as the backporting guide asks, here is why it is worth the risk on a branch that has shipped a release.

When Comet can't use native shuffle, for example because the child is a Spark operator that produces rows, the default auto shuffle mode falls back to JVM columnar shuffle, which converts each UnsafeRow to Arrow in native code. #6596 found that three per-value costs took most of that conversion's time for strings, maps and arrays. #6604 measured these per-row costs before and after:

Column	Before	After
map<string,string>	150 ns	47.3 ns
array<string>	35.5 ns	24.3 ns
string, default preferDictionary.ratio	21.5 ns	16.9 ns
array<int>, 8 / 16 / 50 elements, no nulls	12.5 / 18.8 / 51.6 ns	8.5 / 7.7 / 9.6 ns

Fixed-width types, decimals and structs are unchanged.

The risk is small:

• The change is confined to per-value conversion code and produces the same output as before. It adds no config and changes no plan, protobuf or shuffle format.
• The cherry-pick's diff is byte-identical to perf: speed up string, map and array conversion in JVM columnar shuffle #6604's, blob hashes included, so the four files on this branch were exactly the ones perf: speed up string, map and array conversion in JVM columnar shuffle #6604 changed on main.
• perf: speed up string, map and array conversion in JVM columnar shuffle #6604 has been on main since 2026-10-05, and nothing there has had to fix it since. The two later commits that touch the same files, perf: bulk-append long nullable primitive arrays in SparkUnsafeArray conversion #6824 and perf: presize collections and reuse buffers in native loops #6861, are further optimizations and are not part of this backport.

Not proposed for branch-1.0, which takes only selected bug fixes.

What changes are included in this PR?

A cherry-pick (-x) of #6604's squash commit, with no adaptations. See #6604 for the details. In short:

• decode_utf8_spark_lossy checks a value of up to 32 bytes with an inline is_ascii() before calling str::from_utf8, and checks a longer value 32 bytes at a time, validating only from the first block that isn't ASCII. Besides JVM columnar shuffle, the default CAST(binary AS string) decodes every value through this function, and a string import falls back to it when a buffer fails UTF-8 validation.
• append_map_elements reads a map's key and value fields through map_key_value_fields, instead of building a MapFieldNames, which allocated and freed six strings, for every map value.
• A nullable primitive array with no nulls and at least eight elements is appended with one append_slice. The timestamp and date appenders, which were copies of the macro, now use it.

How are these changes tested?

Run locally on this branch with JDK 17:

• cargo test -p datafusion-comet-shuffle -p datafusion-comet-common passes (172 and 67 tests), including perf: speed up string, map and array conversion in JVM columnar shuffle #6604's nullable_primitive_arrays_round_trip_on_every_append_path and decode_utf8_spark_lossy_ascii_is_borrowed_zero_copy. The 105 conversion_funcs tests in datafusion-comet-spark-expr pass too, including the binary-to-string casts that call the decoder.
• On the default Spark 4.1 profile, CometShuffleSuite (46 tests) and DisableAQECometShuffleSuite (45) pass. They run JVM columnar shuffle over strings, maps, arrays, structs and the other supported types. So do the three binary-to-string tests in CometNativeCastSuite.
• cargo fmt --check and workspace clippy with --all-targets -D warnings are clean, both with my local Rust 1.97 and with a 1.99 nightly, since CI uses stable.

#6604's tests check that each new append path gives the same output as the old one, so they pass without the change as well. There is no bug to show failing here. I didn't rerun the benchmarks, because the conversion code on this branch was identical to main's before #6604.

By CI's path routing, which I checked with dev/ci/compute-changes.py, this PR runs the all-profiles Linux build, macOS, the PyArrow UDF tests, the Spark SQL jobs for 3.5, 4.0 and 4.1, and the four Iceberg versions, so no run-* label is needed.

[sunchao]

No introduced P1/P2 issues found within this review.

Reviewed the entire four-file diff from 3b16d9f21ce3a46dd3737e1250fd1c86d4a17771 to f3f7ce211ddff1b245303c5af5b179a1b01647e9. The PR remains non-draft. The snapshot and refreshed GitHub discussion contain no existing reviews or comments. Routed skills: review-comet-pr, review-comet-shuffle-pr, review-comet-expression-pr, and review-comet-ffi-pr.

Summary

• Prior state and problem: JVM columnar shuffle repeatedly validated strings, allocated unused map-field names, and appended nullable primitive arrays element by element even when they contained no nulls. This backports #6604 to branch-1.1.
• Design approach: Add ASCII and null-free-array fast paths while retaining the existing decoding and per-element fallbacks. Separate borrowed map-field lookup from builder-name construction.
• Correctness: The ASCII prefix ends at a valid character boundary, and the remaining bytes still undergo UTF-8 validation. Array copying requires a clear null bitmap and suitable alignment. Local checks preserved malformed-byte replacement, null placement, and floating-point bit patterns.
• Compatibility analysis: Compared UnsafeArrayData, UnsafeArrayWriter, UnsafeMapData, UTF8String, and cast implementations across Spark 3.4.3, 3.5.9, 4.0.4, 4.1.3, and 4.2.0. Their relevant layouts and rendered-string semantics agree with these paths. Comet’s existing lossy-decoding difference from Spark’s raw-byte identity remains unchanged. No new shim, configuration, or shuffle-format change is needed.
• Key design decisions: The 32-byte ASCII block and eight-element bulk-copy threshold keep the optimization bounded. Null-containing arrays retain element-wise handling, misaligned values retain unaligned reads, and map builders still receive their original field names.
• Implementation sketch: utils.rs adds ASCII detection, list.rs adds bitmap scanning and consolidates date/timestamp appenders into the existing macro, and map.rs/row.rs use allocation-free field lookup. All four resulting files are byte-identical to the merged #6604 versions.
• Performance: The map path demonstrably removes six temporary string allocations per general-path map conversion. Bulk appending avoids repeated builder operations for null-free arrays. #6604 reports improvements including 150 to 47.3 ns for map<string,string> conversion. I did not rerun those benchmarks or establish an end-to-end speedup here.
• Design: The change stays within conversion routines and preserves partitioning, spilling, ownership, and wire-format behavior. The localized fast paths are straightforward to reason about.
• Abstraction & complexity: The borrowed map-field helper separates schema inspection from allocation. Reusing the primitive-appender macro removes duplicated date/timestamp implementations without adding a new framework. No complexity issue meeting the P1/P2 bar was identified.
• Behavioral changes worth calling out: The shared string optimization also affects binary-to-string casts and string-import fallback decoding. Expected values and error behavior remain unchanged.
• Suggested improvements: No code change meeting the P1/P2 reporting bar is requested. Integration CI remains outstanding validation.

Local validation passed: cargo test --locked --offline -p datafusion-comet-common -p datafusion-comet-shuffle --lib (67 and 172 tests), plus the datafusion-comet-spark-expr conversion_funcs tests (105). A disposable decoder harness matched the base revision and JDK 21 on 90,576 cases. Another harness using the exact-head array implementation and Arrow builders passed 6,912 layouts covering thresholds, alignment, null patterns, repeated appends, and float bit patterns. Project sources remain unchanged.

Exact-head CI: title, labeling, preflight, change detection, Linux lint, and Scala syntactic lint passed. Linux/macOS builds, Rust tests, Spark SQL 3.5/4.0/4.1, Iceberg, and PyArrow checks remain queued or running. No failures were reported at the final check. Spark SQL 3.4 and benchmark CI were skipped. I did not run JVM/Spark integration suites locally, and those pending CI outcomes are not established by the local Rust and standalone checks.

[comphead]

Thanks @andygrove