Skip to content

Native Iceberg write over-counts NaNs for nested float fields when the input batch is already sliced #6146

Description

@andygrove

Describe the bug

For float and double fields inside a list or map, the native Iceberg writer can over-count NaN values in the data file's metrics when the input batch is already a slice of a larger array.

The writer's RowSlicer handles slices it cuts itself. A batch that arrives already sliced (for example from GlobalLimitExec in INSERT ... SELECT ... LIMIT n) keeps its full child arrays through the schema cast (iceberg_write.rs:871-879), and iceberg-rust's NaN counter reads list.values(), so it also counts NaNs outside the batch's window.

Effect: wrong nan_value_counts for nested float/double fields. Spark cannot push predicates onto list elements, so this is mostly incorrect metadata rather than wrong query results. Found by reading the code; not reproduced.

Steps to reproduce

CREATE TABLE t (id INT, xs ARRAY<DOUBLE>) USING iceberg;
-- source where NaNs appear only in rows past the limit
INSERT INTO t SELECT id, xs FROM src ORDER BY id LIMIT 10;
SELECT readable_metrics FROM t.files;

Expected behavior

nan_value_counts counts only the elements of the rows written, matching iceberg-java. Compacting the list/map child arrays (or the whole batch) before the metrics are computed would fix it.

Additional context

Found in an audit of the native Iceberg write path. Part of #5649.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:Icebergarea:writerNative Parquet writerbugSomething isn't workingpriority:mediumFunctional bugs, performance regressions, broken features

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions