Skip to content

docs: date the float value and null count divergence to Iceberg 1.9 - #6681

Merged
andygrove merged 1 commit into
apache:mainfrom
andygrove:docs-iceberg-metrics-1-9
Oct 5, 2026
Merged

andygrove merged 1 commit into
apache:mainfrom
andygrove:docs-iceberg-metrics-1-9

Conversation

@andygrove

Copy link
Copy Markdown
Member

Which issue does this PR close?

No issue. #6659 reported this difference, which the user guide already lists as an accepted divergence, so it was closed. This corrects the Iceberg version the guide gives for it.

Rationale for this change

The user guide's accepted divergences say that manifest value_counts / null_value_counts for a float or double field under a nullable struct differ from iceberg-java's on Iceberg 1.10+. The difference starts in Iceberg 1.9.0. From that release ParquetUtil.footerMetrics delegates to ParquetMetrics, whose metricsFromFieldMetrics takes a field's value, null and NaN counts from the writer's FieldMetrics whenever one exists. In 1.8, footerMetrics took value and null counts from the footer.

What changes are included in this PR?

The version in that entry of iceberg-writes.md changes from 1.10+ to 1.9+.

How are these changes tested?

Docs only. Checked against the Iceberg sources at the release tags: ParquetMetrics.java exists from apache-iceberg-1.9.0 on, ParquetUtil.footerMetrics delegates to it in 1.9.1 and 1.10.0, and in 1.8.1 it reads value and null counts from the footer.

ParquetMetrics, which takes a float field's value and null counts from
the writer's FieldMetrics, arrived in Iceberg 1.9.0, so the accepted
divergence applies from 1.9, not 1.10.

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

  • Prior state and problem: The guide dated the nested float/double count divergence to Iceberg 1.10+, although it starts in 1.9.0.
  • Design approach: Correct that version boundary in the existing accepted-divergence entry.
  • Correctness / compatibility analysis: Verified that Iceberg 1.8.1 obtains value/null counts from Parquet footers. In 1.9.0, ParquetUtil.footerMetrics delegates to ParquetMetrics, which prefers supplied FieldMetrics counts. Confirmed this behavior in 1.9.1, 1.10.0, and 1.11.0, and traced the Spark writer’s nullable-struct handling against Comet’s metric reconstruction.
  • Key design decisions: A documentation-only correction fits the problem and introduces no runtime overhead or abstraction complexity.
  • Implementation sketch: The entire base-relative diff changes 1.10+ to 1.9+ at docs/source/user-guide/latest/iceberg-writes.md:369. There are no prerequisite commits in scope.
  • Behavioral changes worth calling out: Users receive the corrected affected-version range. Compared the page against latest release branch branch-1.1; this PR introduces no runtime behavior change.
  • Suggested improvements: None meeting the P1/P2 bar. No introduced P1/P2 issues found within this review.

Reviewed full SHA 6a0371c1787d23b68f3aded33c34b70d9da03674 against base ba08acd815d5fd15f75ce9f314715778b9161cf1. Routed skills: review-comet-pr and review-comet-iceberg-write-pr. Confirmed non-draft status. Snapshot and live discussion contained no reviews, comments, or review threads. Also read linked issue #6659 and its closure discussion.

Exact-head CI: label passed. Preflight, check-pr-title, and Analyze Actions remained queued. No failures were reported, but CI was not complete.

Validation limits: Upstream source comparison and git diff --check completed. No documentation build or Spark/Iceberg runtime tests were run for this single version-text correction.

@andygrove
andygrove enabled auto-merge October 5, 2026 16:25
@andygrove
andygrove added this pull request to the merge queue Oct 5, 2026
Merged via the queue into apache:main with commit 57f27db Oct 5, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Iceberg documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants