Skip to content

docs: correct what the Iceberg split-operator plan moves into AQE - #6407

Merged
andygrove merged 1 commit into
apache:mainfrom
andygrove:docs/iceberg-writes-aqe
Sep 30, 2026
Merged

andygrove merged 1 commit into
apache:mainfrom
andygrove:docs/iceberg-writes-aqe

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

No issue; documentation only. Follow-up to review feedback from @mbutrovich on the Comet 1.1.0 blog post (apache/datafusion-site#205), which repeated this wording.

Rationale for this change

Both Iceberg writes guides say the query feeding an Iceberg write can't be re-planned by AQE, and that the split-operator plan makes the write's input visible to AQE. That isn't right. Spark's InsertAdaptiveSparkPlan has case c: V2CommandExec => c.withNewChildren(c.children.map(apply)), in both Spark 3.5 and 4.0, so the input query of a V2 write such as AppendDataExec is already wrapped in AQE. What sits outside AQE is the write operator itself.

As #4658 describes it, the split moves the data-file writing step inside AQE, so it can be re-planned in response to its upstream operators, and separates it from the metadata writing and commit, which Comet does not accelerate.

What changes are included in this PR?

  • user-guide/latest/iceberg-writes.md: the Overview now says AQE already re-plans the sub-query, that the write operator is what sits outside AQE, and that the split moves data-file writing inside AQE, apart from the commit.
  • contributor-guide/iceberg-writes.md: the Split-Operator Plan section says the same and names InsertAdaptiveSparkPlan. The plan diagram now marks which operators run inside and outside AQE instead of labeling the input query "now visible to AQE and Comet".

How are these changes tested?

Documentation only. npx prettier@latest --check passes on both files.

Spark's InsertAdaptiveSparkPlan already wraps the input query of a
V2CommandExec write in AQE; what sits outside AQE is the write operator
itself. The split moves data-file writing inside AQE, apart from the
commit, rather than making the write's input visible to AQE.
@github-actions github-actions Bot added documentation Improvements or additions to documentation area:Iceberg labels Sep 29, 2026

@mbutrovich mbutrovich left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @andygrove!

@andygrove
andygrove added this pull request to the merge queue Sep 29, 2026
Merged via the queue into apache:main with commit 26b9316 Sep 30, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Iceberg documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants