Repository navigation
docs: date the float value and null count divergence to Iceberg 1.9 - #6681
Merged
Merged
Conversation
ParquetMetrics, which takes a float field's value and null counts from the writer's FieldMetrics, arrived in Iceberg 1.9.0, so the accepted divergence applies from 1.9, not 1.10.
sunchao
approved these changes
Oct 5, 2026
sunchao
left a comment
Member
There was a problem hiding this comment.
Summary
- Prior state and problem: The guide dated the nested float/double count divergence to Iceberg
1.10+, although it starts in1.9.0. - Design approach: Correct that version boundary in the existing accepted-divergence entry.
- Correctness / compatibility analysis: Verified that Iceberg
1.8.1obtains value/null counts from Parquet footers. In1.9.0,ParquetUtil.footerMetricsdelegates toParquetMetrics, which prefers suppliedFieldMetricscounts. Confirmed this behavior in1.9.1,1.10.0, and1.11.0, and traced the Spark writer’s nullable-struct handling against Comet’s metric reconstruction. - Key design decisions: A documentation-only correction fits the problem and introduces no runtime overhead or abstraction complexity.
- Implementation sketch: The entire base-relative diff changes
1.10+to1.9+atdocs/source/user-guide/latest/iceberg-writes.md:369. There are no prerequisite commits in scope. - Behavioral changes worth calling out: Users receive the corrected affected-version range. Compared the page against latest release branch
branch-1.1; this PR introduces no runtime behavior change. - Suggested improvements: None meeting the P1/P2 bar. No introduced P1/P2 issues found within this review.
Reviewed full SHA 6a0371c1787d23b68f3aded33c34b70d9da03674 against base ba08acd815d5fd15f75ce9f314715778b9161cf1. Routed skills: review-comet-pr and review-comet-iceberg-write-pr. Confirmed non-draft status. Snapshot and live discussion contained no reviews, comments, or review threads. Also read linked issue #6659 and its closure discussion.
Exact-head CI: label passed. Preflight, check-pr-title, and Analyze Actions remained queued. No failures were reported, but CI was not complete.
Validation limits: Upstream source comparison and git diff --check completed. No documentation build or Spark/Iceberg runtime tests were run for this single version-text correction.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
No issue. #6659 reported this difference, which the user guide already lists as an accepted divergence, so it was closed. This corrects the Iceberg version the guide gives for it.
Rationale for this change
The user guide's accepted divergences say that manifest
value_counts/null_value_countsfor a float or double field under a nullable struct differ from iceberg-java's on Iceberg 1.10+. The difference starts in Iceberg 1.9.0. From that releaseParquetUtil.footerMetricsdelegates toParquetMetrics, whosemetricsFromFieldMetricstakes a field's value, null and NaN counts from the writer'sFieldMetricswhenever one exists. In 1.8,footerMetricstook value and null counts from the footer.What changes are included in this PR?
The version in that entry of
iceberg-writes.mdchanges from 1.10+ to 1.9+.How are these changes tested?
Docs only. Checked against the Iceberg sources at the release tags:
ParquetMetrics.javaexists fromapache-iceberg-1.9.0on,ParquetUtil.footerMetricsdelegates to it in 1.9.1 and 1.10.0, and in 1.8.1 it reads value and null counts from the footer.