Skip to content

HIVE-29829: Iceberg: Maintain and invalidate column statistics per partition - #6707

Open
deniskuzZ wants to merge 8 commits into
apache:masterfrom
deniskuzZ:col_stats_refactor
Open

HIVE-29829: Iceberg: Maintain and invalidate column statistics per partition#6707
deniskuzZ wants to merge 8 commits into
apache:masterfrom
deniskuzZ:col_stats_refactor

Conversation

@deniskuzZ

@deniskuzZ deniskuzZ commented Aug 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

depends on #6711

@deniskuzZ
deniskuzZ marked this pull request as draft August 18, 2026 17:14
@deniskuzZ
deniskuzZ force-pushed the col_stats_refactor branch 2 times, most recently from f73473a to 04270e5 Compare August 19, 2026 20:52
@deniskuzZ deniskuzZ changed the title WIP: Iceberg col stats refactor HIVE-29829: Iceberg: Partition-level column statistics refactor Aug 21, 2026
@deniskuzZ
HIVE-29781: Introduce PARTITION__NAME virtual column and refactor Ice…
dbf09be 
…berg virtual column plumbing

Adds VirtualColumn.PARTITION_NAME, materialized by the Iceberg readers: vectorized via
HiveBatchIterator, row mode via PartitionInfo -> IOContext -> MapOperator.

Refactors the read path to take everything the file already carries from the scan task
instead of the per-record constants map: spec id, partition hash, file path, partition
name, first row id and file sequence number are computed once per task in both
VirtualColumnAwareIterator and HiveBatchIterator. Drops the now unread _partition column
from the ACID read schema, along with the write-only VIRTUAL_COLS_TO_META_COLS map and the
per-record helpers it fed.

Fixes VectorizedParquetRecordReader taking a column type from the job column list while
indexing it with a batch column number: the two orders diverge past the data columns
because the batch drops virtual columns the readers cannot fill.
@deniskuzZ
HIVE-29781: Make PARTITION__PROJECTION a merge task record field inst…
2f642c5 
…ead of a virtual column

The partition key a merge task carries is read back only by HiveIcebergDeleteWriter under
isMergeTask: an ordinary delete derives the key from the row data, and a merge task cannot
because it reads delete files, whose records hold only a path and a position.

Declaring it as a virtual column put an always empty string into every DELETE, UPDATE and
MERGE record and into every Iceberg plan. It now lives in a merge specific serde layout that
IcebergMergeRecordReader and HiveIcebergSerDe select on the merge task flag, so the ordinary
delete record is one column shorter and the plans no longer estimate it.
@deniskuzZ
HIVE-29829: Iceberg partition-level column statistics refactor
51824c0
@deniskuzZ
HIVE-29829: Iceberg column statistics granularity and branch-scoped s…
df23971 
…tatistics

Adds hive.iceberg.stats.collect.partlevel (default false), scoping column
  statistics granularity for Iceberg tables independently of the generic
  hive.analyze.stmt.collect.partlevel.stats. Partition-level statistics are
  produced by ANALYZE; every other write maintains table-level ones.

Keys the column-statistics read path on the calling API rather than on session
  config, so a granularity mismatch cannot surface a wrongly-shaped blob.

Stores branch-scoped statistics on the branch's snapshot and stops branch
  writes from overwriting the table-scoped metastore row, including the footer
  scan path used by ANALYZE ... COMPUTE STATISTICS.

Lets a whole-table INSERT OVERWRITE of a partitioned table replace its column
  statistics, which previously skipped because whole-table and partition-scoped
  overwrites are indistinguishable in the snapshot summary.
@deniskuzZ
HIVE-29829: Iceberg write-path partition column statistics
2419328 
  Derive partition names through the Iceberg spec so a write keys statistics
  the way a read looks them up; substitute per partition for a scoped write
  or ANALYZE; gather partition-level statistics for CTAS.
@deniskuzZ
HIVE-29829: Judge Iceberg column statistics per partition
e6cedaf 
A statistics file records the snapshot it was written for, and every blob in
it was stamped with that same snapshot. An entry the merge carried forward
from an earlier file therefore looked measured by the write that stored it:
after a full ANALYZE, a write reaching two partitions, and an ANALYZE naming
one of them, the other's numbers still came from before that write yet
claimed the snapshot after it.

Stamp each blob with the snapshot its numbers were computed at. Puffin
carries the field per blob already, so nothing about the format changes and
existing files stay readable. checkAndMergeColStats tells writeColStats which
entries it carried, and those keep the anchor they arrived with, while what
this write measured is anchored at it.

A read then judges each partition on its own anchor. partitionsChangedSince
walks back from the current snapshot as far as the oldest anchor among the
partitions asked about, collecting the ones the writes in between reached and
skipping the ones that preserve rows, and getAggrColStatsFor leaves out the
partitions those writes left behind. The walk reads manifests, and every
column of a query asks the same of them, so its answer is cached for the
query. It reports every partition changed when it cannot trace them - an
expired snapshot breaks the ancestry, or a file names a partition of a spec
the statistics never described - which is the verdict the whole table had
before.

A compaction changes no rows, so its statistics stand only for what it read
whole: the whole table, or one named partition of a table that keeps them per
partition. The rewrite policy alone does not say which - clearing the
partitions an older spec left behind carries PARTITION too, while rewriting
into partitions that hold rows it never read - so the decision reads four
facts: whether it is a compaction, whether it skipped files by size, and
which of the two scopes it was pointed at. What it may not store it no longer
measures.
@deniskuzZ deniskuzZ changed the title HIVE-29829: Iceberg: Partition-level column statistics refactor HIVE-29829: Iceberg: Maintain and invalidate column statistics per partition Aug 23, 2026
@deniskuzZ
HIVE-29829: Serve no column statistics a write has left behind
b6d4239 
Statistics stored for the table as a whole were served whatever had happened
to it since, because the read walked to the nearest ancestor that had any. A
query planned on them was told they were complete, so after an UPDATE or a
DELETE the planner folded and estimated on the numbers of a table that no
longer existed.

The metastore withholds its own on the same terms: get_table_statistics_req
returns nothing for statistics that do not describe the write id asked about,
and get_partitions_statistics_req skips such a partition rather than passing
it off as current. An Iceberg table now answers the same way - what an
intervening write left behind is not served, and a read of an unpartitioned
table after a DML falls back to estimates, as it does for a transactional one.

An aggregate of the partitions that were asked about is a different case: the
ones it found still describe themselves, and only their number tells the
planner it covers a subset. Statistics assembled from fewer partitions than
the scan reads are marked PARTIAL, which is the state a partitioned table
already carries when only some of its partitions have been analyzed.
@deniskuzZ
HIVE-29829: Anchor carried column statistics at the write that keeps …
2f9bc21 
…them

An entry carried across writes kept the snapshot it was measured at, so a read
had to walk the manifests back to it to learn whether anything had reached its
partition since. After an ANALYZE of one partition the file sits on the current
snapshot while the entries beside it are anchored wherever they were computed,
which could be hundreds of commits back, and every query paid for that walk.

The merge is the one place that can settle it once: it already reads what it
carries, so it now asks whether each entry still holds and anchors the ones
that do at the write it is making. A read then walks only what happened after
the file it serves, which for a table analyzed at all recently is nothing.

What no longer holds keeps the snapshot it was measured at, which is what
marks it behind, and the walk is bounded for reads by
hive.iceberg.stats.max.snapshot.lookback: beyond it, entries measured further
back are treated as describing their partition no longer, while ones measured
within it are still placed. A write is not bounded - what it records stands for
every read after it, so giving up there would leave those entries unplaceable
for good.
@sonarqubecloud

sonarqubecloud Bot commented Aug 23, 2026

Copy link
Copy Markdown

Quality Gate Passed Quality Gate passed

Issues
28 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.7% Duplication on New Code

See analysis details on SonarQube Cloud

@deniskuzZ deniskuzZ mentioned this pull request Aug 23, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

tests unstable

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@deniskuzZ @asf-ci-hive