Repository navigation

feat: Add Spark-compatible `encode` function to datafusion-spark - #21331

Merged

Jefffrey merged 22 commits into

apache:mainfrom

JeelRajodiya:feat/spark-encode-function

Sep 7, 2026

Commits on Apr 15, 2026
- feat: Add Spark-compatible `encode` function to datafusion-spark
  JeelRajodiya
  committed
  2709f2b
- test: Add LargeBinary, BinaryView, and emoji surrogate pair tests
  JeelRajodiya
  committed
  fbf8a07
- refactor: Make extract_charset private
  JeelRajodiya
  committed
  91222a3
- fix: Inline format args to satisfy clippy::uninlined_format_args
  JeelRajodiya
  committed
  0a434fe
- fix remaining clippy error
  JeelRajodiya
  committed
  295416f
- fix the fmt error
  JeelRajodiya
  committed
  db519be
- feat: Add UTF-32 charset support
  JeelRajodiya
  committed
  93f1ec9
- feat: Add ANSI mode support for unmappable characters
  JeelRajodiya
  committed
  6cb99a7
Commits on Apr 16, 2026
- move tests to slt file
  JeelRajodiya
  committed
  b6dcd45
Commits on Apr 19, 2026
- fix: UTF-32 to match Spark output (no BOM) and document target version
  JeelRajodiya
  committed
  701850d
Commits on Jun 17, 2026
- Merge branch 'main' into feat/spark-encode-function
  Jefffrey
  authored
  9900e96
Commits on Jun 29, 2026
Commits on Aug 30, 2026
- Merge branch 'main' into feat/spark-encode-function
  JeelRajodiya
  authored
  3b3ac49
- perf(spark): return scalar null for encode with null charset
  
  JeelRajodiya
  and
  Jefffrey
  authored
  a88725e
- refactor: remove todo from the doc comment
  
  JeelRajodiya
  and
  Jefffrey
  authored
  d92dbe4
- refactor: get rid of null_binary_array function, build null array directy
  
  JeelRajodiya
  and
  Jefffrey
  authored
  42be89d
- docs(spark): explain encode binary coercion
  JeelRajodiya
  committed
  e0764e5
- fix(spark): import binary array for encode
  JeelRajodiya
  committed
  7f693d7
- Merge branch 'main' into feat/spark-encode-function
  JeelRajodiya
  authored
  9b3faea
- fix(spark): remove unused encode null helper
  JeelRajodiya
  committed
  2afe5cd

Navigation Menu

Repository navigation

feat: Add Spark-compatible `encode` function to datafusion-spark - #21331

feat: Add Spark-compatible `encode` function to datafusion-spark#21331
Jefffrey merged 22 commits into
apache:mainfrom
JeelRajodiya:feat/spark-encode-function

Commits on Apr 15, 2026

feat: Add Spark-compatible `encode` function to datafusion-spark

test: Add LargeBinary, BinaryView, and emoji surrogate pair tests

refactor: Make extract_charset private

fix: Inline format args to satisfy clippy::uninlined_format_args

fix remaining clippy error

fix the fmt error

feat: Add UTF-32 charset support

feat: Add ANSI mode support for unmappable characters

Commits on Apr 16, 2026

move tests to slt file

Commits on Apr 19, 2026

fix: UTF-32 to match Spark output (no BOM) and document target version

Commits on Jun 17, 2026

Merge branch 'main' into feat/spark-encode-function

Commits on Jun 29, 2026

Merge branch 'main' into feat/spark-encode-function

move the tests to slt file, added coerce args so we preserve the binary types and use the binary arm in encode_dispatch

add support for charset per rows

Commits on Aug 30, 2026

Merge branch 'main' into feat/spark-encode-function

perf(spark): return scalar null for encode with null charset

refactor: remove todo from the doc comment

refactor: get rid of null_binary_array function, build null array directy

docs(spark): explain encode binary coercion

fix(spark): import binary array for encode

Merge branch 'main' into feat/spark-encode-function

fix(spark): remove unused encode null helper