feat: Add Spark-compatible `encode` function to datafusion-spark by JeelRajodiya · Pull Request #21331 · apache/datafusion

JeelRajodiya · 2026-04-03T06:46:27Z

Rationale

The datafusion-spark crate is missing the encode function. Spark's encode(expr, charset) converts a string or binary value into binary using a specified character encoding — commonly used in Spark SQL workloads and needed by engines built on DataFusion that target Spark compatibility.

What changes are included in this PR?

Adds SparkEncode to datafusion-spark's string functions, emulating Spark 3.5 semantics. It supports US-ASCII, ISO-8859-1, UTF-8, UTF-16, UTF-16BE, UTF-16LE, UTF-32, UTF-32BE, and UTF-32LE, including common aliases (UTF8, LATIN1, etc.) and case-insensitive matching. The charset can be a constant or a per-row column. Binary input is decoded as lossy UTF-8 (invalid bytes → U+FFFD) before re-encoding, and unmappable characters are silently replaced with ?, matching Spark.

Are these changes tested?

Yes. Coverage lives in encode.slt (sqllogictest) and exercises all charsets and aliases, case-insensitive matching, null value/charset handling, per-row charsets, binary input (Binary/LargeBinary/BinaryView) with lossy UTF-8, Utf8View input, and the unsupported-charset error. A Rust unit test covers return-field nullability.

Are there any user-facing changes?

New encode scalar function available when using datafusion-spark.

Zeel-e6x · 2026-04-03T10:49:13Z

run benchmarks

adriangbot · 2026-04-03T10:49:15Z

Hi @Zeel-e6x, thanks for the request (#21331 (comment)). Only whitelisted users can trigger benchmarks. Allowed users: Dandandan, Fokko, Jefffrey, Omega359, adriangb, alamb, asubiotto, brunal, buraksenn, cetra3, codephage2020, comphead, erenavsarogullari, etseidl, friendlymatthew, gabotechs, geoffreyclaude, grtlr, haohuaijin, jonathanc-n, kevinjqliu, klion26, kosiew, kumarUjjawal, kunalsinghdadhwal, liamzwbao, mbutrovich, mzabaluev, neilconway, rluvaton, sdf-jkl, timsaucer, xudong963, zhuqi-lucas.

File an issue against this benchmark runner

xanderbailey

Looks good to me but you’ll need a committer to Approve also! Thanks for the PR!

JeelRajodiya · 2026-04-07T15:48:14Z

Hey @xanderbailey, Do I need to mention the maintainers for review? if yes please suggest whom incase you know.
I'm planning to open more PRs for implementing other functions but I'm waiting for this PR to get merged.

xanderbailey · 2026-04-07T16:19:42Z

They will normally pick it up within a week or so. If not we can ping them here.

alamb · 2026-04-07T18:13:13Z

Thanks @xanderbailey and @JeelRajodiya -- the PR load is pretty intense! I started the CI for this PR

JeelRajodiya · 2026-04-07T18:55:56Z

I pushed the fixes for clippy errors. @alamb Can you rerun the checks please?

JeelRajodiya · 2026-04-08T18:20:25Z

Please rerun the checks

andygrove · 2026-04-12T15:10:15Z

+            }
+            Ok(bytes)
+        }
+        _ => exec_err!(


Spark also supports UTF-32. It would be worth adding a comment here explaining why this isn't or can't be supported.

arguments = """ Arguments: * str - a string expression * charset - one of the charsets 'US-ASCII', 'ISO-8859-1', 'UTF-8', 'UTF-16BE', 'UTF-16LE', 'UTF-16', 'UTF-32' to encode `str` into a BINARY. It is case insensitive. """,

I missed adding support for UTF-32, I've added it now with respective tests.

Implements `encode(string_or_binary, charset)` that converts a string or binary value into binary using the specified character encoding, matching Apache Spark's behavior.

In ANSI mode (default), encoding a character that cannot be represented in the target charset (e.g. non-ASCII char in US-ASCII) returns an error. In legacy mode, unmappable characters are silently replaced with '?'.

andygrove · 2026-04-17T14:33:55Z

Thanks for iterating on this @JeelRajodiya. One issue I noticed:

spark-sql> SELECT hex(encode('A', 'UTF-32'));                                                                                                                                                                        
00000041

This PR returns 0000FEFF00000041, with a BOM. Both Spark 3.5 and Spark 4.1 return the four-byte form.

Once challenge for this PR is that there is different behavior across Spark versions for the encode expression. Which Spark version is this PR targeting? It would be good to document that.

…ry types and use the binary arm in encode_dispatch

Jefffrey · 2026-06-30T14:26:12Z

+            DataType::Utf8
+            | DataType::LargeUtf8
+            | DataType::Utf8View
+            | DataType::Binary


does Spark also operate on binary types for this function? source code here seems to suggest only strings

https://github.com/apache/spark/blob/dd4b74184e36625aff54d3695a134839a9721bd3/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala#L3276-L3280

Not directly, but Spark Catalyst’s Analyzer inserts the cast specifically through the ImplicitTypeCasts coercion rule.

Thus:

encode(binary_value, 'UTF-8')

is analyzed as:

encode(CAST(binary_value AS STRING), 'UTF-8')

We preserve binary input in coerce_types instead of letting DataFusion cast it to a string because:

Spark’s binary-to-string cast preserves malformed bytes, and encode later converts them to U+FFFD.

DataFusion’s normal Arrow binary-to-UTF8 cast rejects malformed UTF-8 before encode runs.

Example: X'FF' is malformed UTF-8. Spark’s encode path converts it to U+FFFD, while DataFusion’s normal binary-to-UTF8 cast returns an invalid UTF-8 error.

this is worth leaving a comment here about (keep it succinct and to the point)

github-actions · 2026-08-30T01:40:39Z

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>

codecov-commenter · 2026-08-30T09:24:17Z

Codecov Report

❌ Patch coverage is 89.06250% with 21 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.53%. Comparing base (4b8ad88) to head (2afe5cd).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/spark/src/function/string/encode.rs	89.00%	17 Missing and 4 partials ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #21331    +/-   ##
========================================
  Coverage   81.53%   81.53%            
========================================
  Files        1123     1124     +1     
  Lines      406042   406294   +252     
  Branches   406042   406294   +252     
========================================
+ Hits       331059   331271   +212     
- Misses      55621    55652    +31     
- Partials    19362    19371     +9

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

JeelRajodiya · 2026-08-30T09:41:49Z

I'm still working on the PR

Jefffrey · 2026-08-30T12:56:26Z

+            DataType::Utf8
+            | DataType::LargeUtf8
+            | DataType::Utf8View
+            | DataType::Binary


this is worth leaving a comment here about (keep it succinct and to the point)

Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>

…ecty Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>

JeelRajodiya · 2026-09-06T17:24:33Z

if it looks good, Can we get this merged @Jefffrey ?

Jefffrey · 2026-09-07T00:46:03Z

thanks @JeelRajodiya & co

sorry it hung for so long

github-actions Bot added the spark label Apr 3, 2026

xanderbailey reviewed Apr 3, 2026

View reviewed changes

Comment thread datafusion/spark/src/function/string/encode.rs Outdated

Comment thread datafusion/spark/src/function/string/encode.rs

Comment thread datafusion/spark/src/function/string/encode.rs

xanderbailey approved these changes Apr 4, 2026

View reviewed changes

JeelRajodiya force-pushed the feat/spark-encode-function branch from 22a2705 to bf46433 Compare April 7, 2026 18:54

andygrove reviewed Apr 12, 2026

View reviewed changes

Comment thread datafusion/spark/src/function/string/encode.rs

github-actions Bot added the core Core DataFusion crate label Apr 15, 2026

JeelRajodiya force-pushed the feat/spark-encode-function branch from 55f4694 to 835ae8d Compare April 15, 2026 05:32

github-actions Bot removed the core Core DataFusion crate label Apr 15, 2026

JeelRajodiya added 8 commits April 15, 2026 11:10

feat: Add Spark-compatible encode function to datafusion-spark

2709f2b

Implements `encode(string_or_binary, charset)` that converts a string or binary value into binary using the specified character encoding, matching Apache Spark's behavior.

test: Add LargeBinary, BinaryView, and emoji surrogate pair tests

fbf8a07

refactor: Make extract_charset private

91222a3

fix: Inline format args to satisfy clippy::uninlined_format_args

0a434fe

fix remaining clippy error

295416f

fix the fmt error

db519be

feat: Add UTF-32 charset support

93f1ec9

feat: Add ANSI mode support for unmappable characters

6cb99a7

In ANSI mode (default), encoding a character that cannot be represented in the target charset (e.g. non-ASCII char in US-ASCII) returns an error. In legacy mode, unmappable characters are silently replaced with '?'.

JeelRajodiya force-pushed the feat/spark-encode-function branch from 835ae8d to 6cb99a7 Compare April 15, 2026 05:40

JeelRajodiya requested a review from andygrove April 15, 2026 05:43

move tests to slt file

b6dcd45

github-actions Bot added the sqllogictest SQL Logic Tests (.slt) label Apr 16, 2026

andygrove mentioned this pull request Apr 17, 2026

test: add SQL tests documenting Spark encode behavior apache/datafusion-comet#3975

Closed

JeelRajodiya added 3 commits June 29, 2026 11:21

Merge branch 'main' into feat/spark-encode-function

ee22456

move the tests to slt file, added coerce args so we preserve the bina…

ac43977

…ry types and use the binary arm in encode_dispatch

add support for charset per rows

7cb0731

JeelRajodiya requested a review from Jefffrey June 29, 2026 11:17

Jefffrey reviewed Jun 30, 2026

View reviewed changes

comphead mentioned this pull request Jul 2, 2026

feat: Support Spark Expression Encode apache/datafusion-comet#4315

Open

github-actions Bot added the Stale PR has not had any activity for some time label Aug 30, 2026

JeelRajodiya and others added 2 commits August 30, 2026 14:07

Merge branch 'main' into feat/spark-encode-function

3b3ac49

perf(spark): return scalar null for encode with null charset

a88725e

Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>

JeelRajodiya requested a review from Jefffrey August 30, 2026 09:39

Jefffrey removed the Stale PR has not had any activity for some time label Aug 30, 2026

Jefffrey reviewed Aug 30, 2026

View reviewed changes

JeelRajodiya and others added 2 commits August 30, 2026 18:31

refactor: remove todo from the doc comment

d92dbe4

Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>

refactor: get rid of null_binary_array function, build null array dir…

42be89d

…ecty Co-authored-by: Jeffrey Vo <jeffrey.vo.australia@gmail.com>

github-actions Bot added the auto detected api change Auto detected API change label Aug 30, 2026

JeelRajodiya added 3 commits August 30, 2026 18:39

docs(spark): explain encode binary coercion

e0764e5

fix(spark): import binary array for encode

7f693d7

Merge branch 'main' into feat/spark-encode-function

9b3faea

github-actions Bot removed the auto detected api change Auto detected API change label Aug 30, 2026

fix(spark): remove unused encode null helper

2afe5cd

Jefffrey approved these changes Aug 30, 2026

View reviewed changes

Jefffrey added this pull request to the merge queue Sep 7, 2026

Merged via the queue into apache:main with commit fdfb67d Sep 7, 2026
38 checks passed

nuno-faria added the v56.0.0 label Oct 3, 2026

Conversation

JeelRajodiya commented Apr 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Zeel-e6x commented Apr 3, 2026

Uh oh!

adriangbot commented Apr 3, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

xanderbailey left a comment

Choose a reason for hiding this comment

Uh oh!

JeelRajodiya commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

xanderbailey commented Apr 7, 2026

Uh oh!

alamb commented Apr 7, 2026

Uh oh!

JeelRajodiya commented Apr 7, 2026

Uh oh!

JeelRajodiya commented Apr 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andygrove Apr 12, 2026

Choose a reason for hiding this comment

Uh oh!

JeelRajodiya Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

andygrove commented Apr 17, 2026

Uh oh!

Uh oh!

Jefffrey Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

JeelRajodiya Aug 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Jefffrey Aug 30, 2026

Choose a reason for hiding this comment

Uh oh!

JeelRajodiya Aug 30, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 30, 2026

Uh oh!

codecov-commenter commented Aug 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

JeelRajodiya commented Aug 30, 2026

Uh oh!

Uh oh!

Uh oh!

Jefffrey Aug 30, 2026

Choose a reason for hiding this comment

Uh oh!

JeelRajodiya commented Sep 6, 2026

Uh oh!

Jefffrey commented Sep 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

JeelRajodiya commented Apr 3, 2026 •

edited

Loading

JeelRajodiya commented Apr 7, 2026 •

edited

Loading

JeelRajodiya commented Apr 8, 2026 •

edited

Loading

JeelRajodiya Apr 15, 2026 •

edited

Loading

JeelRajodiya Aug 30, 2026 •

edited

Loading

codecov-commenter commented Aug 30, 2026 •

edited

Loading