fix: [branch-1.0] zero sliced boolean offsets at every level before exporting to the JVM (#6339)

[andygrove]

Which issue does this PR close?

Backport of #6339 to branch-1.0. The original PR closes #6288.

Rationale for this change

Arrow Java's C Data import ignores ArrowArray.offset at every level. arrow-rs folds a slice into the buffers for every Spark type except boolean, which keeps its bit offset, so the JVM can read sliced boolean values and nulls from bit 0. The existing fix only checks the top-level output column and therefore misses booleans nested in structs, lists, or maps, as well as inputs to the JVM UDF bridge.

What changes are included in this PR?

• Add zero_offsets to normalize offsets recursively before arrays cross the native-to-JVM boundary.
• Apply normalization in move_to_spark and the JVM UDF bridge, replacing the top-level take workaround.
• Add native unit coverage and six Scala regressions for sliced booleans in aggregate, offset, UDF, dispatched-expression, and collect_list paths.
• Document the export requirement in the FFI contributor guide.
• Adapt the implementation to branch-1.0's Arrow API and existing two-argument move_to_spark API.

How are these changes tested?

• cargo test --manifest-path native/Cargo.toml -p datafusion-comet-common ffi_offsets --lib — 7 passed.
• cargo test --manifest-path native/Cargo.toml -p datafusion-comet test_move_to_spark_zeroes_nested_boolean_offsets --lib — 1 passed.
• ./mvnw test -Dtest=none -Dsuites="org.apache.comet.CometCodegenSuite,org.apache.comet.exec.CometExecSuite" -Dscalastyle.skip=true — 236 passed across 2 suites.
• cargo fmt --manifest-path native/Cargo.toml --all -- --check.

[sunchao]

Summary

• Prior state and problem: The top-level offset workaround missed sliced boolean children and JVM UDF inputs, allowing Arrow Java to read incorrect values and nulls.
• Design approach: Normalize offsets recursively at both native-to-JVM export boundaries through the shared zero_offsets helper.
• Correctness / compatibility analysis: Checked Arrow Rust 58.4, Arrow Java 18.3 and relevant Spark sources across supported 3.4–4.2 versions. Normalization preserves values, validity, nested structure and buffer ownership. No introduced P1/P2 issues found within this review.
• Key design decisions: Already normalized inputs are borrowed. Byte-aligned boolean slices share storage, while unaligned bitmaps are copied. This removes the previous index-vector allocation and take workaround without adding another export abstraction.
• Implementation sketch: Add the common helper, call it from move_to_spark and JvmScalarUdfExpr, and add native tests, six Scala regressions and FFI documentation.
• Behavioral changes worth calling out: Sliced booleans now retain their values and nulls through nested outputs, aggregates, OFFSET, collect_list and dispatched JVM expressions.
• Suggested improvements: No P1/P2 changes requested.

Reviewed the complete eight-file diff at d9fe07d48c10331d97a279b901f5852869d93515 against 8084fb138b343f7d60607bbc2feea830797a35b8. The PR is not a draft. Snapshot and live discussion checks contained no reviews, comments or threads.

Routed skills: review-comet-pr, review-comet-ffi-pr and review-comet-expression-pr. This backport lacks .ai/skills, so I read its local .claude/skills/review-comet-pr and the requested .ai review skills from sibling checkout 6547-74057e426d60, alongside AGENTS.md and contributor guidance.

Exact-head CI: 13 checks succeeded, including the macOS native build, eight remain pending/running and seven were skipped. The Rust job failed before tests on Clippy redundant_field_names diagnostics in unchanged native/jni-bridge/src/errors.rs.

Validation: All seven focused common-crate tests passed. An independent probe using this head's helper and Arrow 58.4 passed 5,460 combinations of value offsets, validity offsets and lengths. The targeted core export test, with HDFS disabled because JNI headers are unavailable, reached its 180-second limit during compilation without running. Scala end-to-end suites and performance benchmarks were not rerun locally. Project source remains unchanged.