Comet 1.0.0 Blog Post - #201
Merged
Merged
Conversation
Incorporates PRs merged to datafusion-comet main between 2026-07-17 and 2026-07-27, updates PR/contributor counts, and bumps DataFusion version to 54.1. Adds a new Shuffle Improvements section.
Remove inline PR links throughout, compress long enumerations into grouped lists, and consolidate the Iceberg / correctness / config sections into shorter prose. The change log link at the top is now the pointer for per-PR detail.
Replace the full enumeration of ~30 correctness fixes with a stat, two representative examples (whitespace trimming, ANSI short-circuit), and a short note on conservative fallbacks. Full list stays in the change log.
Replace the generic "A 1.0 release is only as good as its results" opener with concrete framing: correctness fixes have been part of every release, and 1.0 was an extra push to clear the bulk of the known issues.
Sketch the arc of the project across five threads — query coverage, Spark support, ecosystem, correctness rigor, and performance — grounded in concrete facts from earlier releases (106 → 404 expressions, JVM codegen dispatcher in 0.17.0, Iceberg in 0.10.0, 9% TPC-DS gain in 0.17.0). Trim the "What 1.0 Means" intro since the arc is now covered above.
Corrected the number of major releases from nineteen to sixteen and updated the supported expressions count to more than 400 in version 1.0.0.
andygrove
commented
Aug 4, 2026
|
|
||
| [TOC] | ||
|
|
||
| The Apache DataFusion PMC is pleased to announce version 1.0.0 of the [Comet](https://datafusion.apache.org/comet/) subproject. |
Member
Author
There was a problem hiding this comment.
This link won't be active until apache/datafusion-comet#5263 is merged
comphead
reviewed
Aug 5, 2026
|
|
||
| ## New Expression and Aggregate Support | ||
|
|
||
| This release expands the set of Spark expressions and aggregates that run natively: |
Contributor
There was a problem hiding this comment.
we can add wider generator support, explode/explode_outer/posexplode/posexplode_outer
comphead
reviewed
Aug 5, 2026
Comment on lines
+270
to
+271
| - **JDK 11** is deprecated and scheduled for removal in Comet 1.1.0. | ||
| - **Apache Spark 3.4** is deprecated and scheduled for removal in Comet 1.1.0. |
Member
Author
There was a problem hiding this comment.
This matches the versioning policy that we published as part of the 1.0.0 release
hsiang-c
reviewed
Aug 5, 2026
| - **String and array kernels**: `lpad`, `unhex`, `size`, `arrays_overlap`, `escape_string`, and the `try_*` | ||
| arithmetic kernel. | ||
|
|
||
| To make this kind of work repeatable, the release also adds a scalar expression optimization guide |
mbutrovich
reviewed
Aug 6, 2026
- Add expression fusion as a fourth consequence of codegen dispatch - Link the scalar expression optimization guide - Note generator and wider operator coverage in the Road to 1.0 section - Explain why Spark 3.4 is deprecated now
mbutrovich
reviewed
Aug 7, 2026
mbutrovich
reviewed
Aug 7, 2026
mbutrovich
reviewed
Aug 7, 2026
Co-authored-by: Matt Butrovich <mbutrovich@users.noreply.github.com>
Co-authored-by: Matt Butrovich <mbutrovich@users.noreply.github.com>
andygrove
marked this pull request as ready for review
August 7, 2026 20:19
hsiang-c
reviewed
Aug 7, 2026
| - **Cast**: Cast expressions where the native implementation is marked as incompatible or unsupported | ||
| are now routed through codegen dispatch. | ||
| - **Grouping**: `grouping()` and `grouping_id()`. | ||
| - **Intervals**: interval types via `make_ym_interval` and `make_dt_interval`, `CalendarIntervalType`, |
There was a problem hiding this comment.
(nit) Maybe we can call out interval support b/c adding new types is infrequent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Comet 1.0.0 blog post: