Add nested struct schema evolution support for Map types

[kosiew]

Which issue does this PR close?

Closes #20835
This is the last of a series of PR to close #20835

Rationale for this change

Map schema adaptation did not have explicit recursive evolution support when Map keys or values contain nested Structs.

This PR defines and implements Map evolution semantics so nested Struct fields can evolve consistently with existing Struct schema adaptation while preserving Map key identity. Both Map keys and values may evolve when they recursively contain Structs. Struct fields are matched by case-sensitive name, nullable target fields may be added as NULL, and source-only value fields may be omitted.

Because Map keys determine lookup identity, key evolution is more restrictive: source key Struct fields cannot be removed, primitive key casts are limited to known injective widenings, and sorted Maps require an unchanged key type. Source and target sorted flags must also match.

What changes are included in this PR?

• Adds Map handling to the central nested Struct compatibility and casting paths.

• Matches Map entry key/value children by semantic position rather than their technical entry field names, while continuing to match nested Struct fields by name.

• Supports recursive Struct evolution in both Map keys and Map values.

• Restricts Map key evolution to preserve key identity:

  • Source Struct key fields cannot be removed.
  • Newly added Struct key fields must be nullable.
  • Primitive key type changes are limited to explicitly supported injective widenings.
  • Sorted Maps require an unchanged key type.

• Requires source and target Map sorted flags to match.

• Validates Map entry invariants, including non-nullable entries and keys.

• Compacts sliced Maps and entries hidden by null parent rows before recursively casting their children, so unreachable entries do not affect casts.

• Uses StructArray::try_new when rebuilding Struct arrays so invalid nullability introduced by safe casts is reported as an error.

• Keeps Maps without semantic Struct children on Arrow's existing Map cast path.

• Adds upgrade documentation describing the Map nested Struct evolution semantics and migration considerations.

Are these changes tested?

Yes. The patch adds unit, integration, and SQL logic coverage for the new behavior, including:

• Safe casts producing nulls for non-nullable Struct fields and Map values.
• Primitive Map casts continuing to use Arrow's cast behavior.
• Int32 → Int64 and UInt32 → Int64 Map key casts.
• Supported and unsupported unsigned-to-signed injective key casts.
• Positional matching of Map key/value entry fields with nested Struct adaptation.
• Rejection of Map key Struct field removal and non-injective key casts.
• Rejection of Map key Struct evolution with no field-name overlap.
• Sorted Map key evolution rejection and sorted Map value evolution.
• Rejection of changes to the Map sorted flag.
• Null Map parents and sliced Maps with otherwise invalid unreachable values.
• Struct evolution in both Map keys and values.
• All-null Map columns with Struct values.
• Map entry invariant validation.
• Planner/runtime error parity for invalid Map evolution.
• Recursive requires_nested_struct_cast behavior for Maps.
• End-to-end Parquet Map value Struct schema evolution and incompatible evolution rejection.
• SQL logic coverage for primitive Map assignment casts, duplicate/null Map keys, and UInt32/Int32 key coercion to Int64.

Are there any user-facing changes?

Yes. Maps whose keys or values recursively contain Structs now support schema evolution through DataFusion's nested adaptation path.

Nested Struct fields are matched by case-sensitive name. Nullable target fields can be added and populated with NULL; incompatible changes and missing non-nullable target fields are rejected. Map key evolution has additional restrictions to preserve key identity, and sorted Maps require an unchanged key type and matching sorted flag.

Ordinary Maps without semantic Struct children continue to use Arrow's existing Map cast behavior.

The upgrade guide for 55.0.0 documents these semantics and the migration considerations.

LLM-generated code disclosure

This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 97.40420% with 21 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.31%. Comparing base (f1f0449) to head (e2c55a5).
⚠️ Report is 26 commits behind head on main.

Files with missing lines	Patch %	Lines
datafusion/common/src/nested_struct.rs	97.40%	15 Missing and 6 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #23914      +/-   ##
==========================================
+ Coverage   81.27%   82.31%   +1.03%     
==========================================
  Files        1116     1117       +1     
  Lines      395017   415524   +20507     
  Branches   395017   415524   +20507     
==========================================
+ Hits       321055   342032   +20977     
- Misses      55166    55208      +42     
+ Partials    18796    18284     -512     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[kosiew]

@TheBuilderJR
Can you test this PR?

[TheBuilderJR]

yep lgtm. thanks @kosiew

[adriangb]

Overall looks great!

I think this requires an upgrade guide entry since it's a change in behavior.

Could we add / move some tests to SLTs in https://github.com/apache/datafusion/blob/main/datafusion/sqllogictest/test_files/map.slt

[adriangb]

This may be a bug:

❯ datafusion-cli
DataFusion CLI v54.0.0
> SELECT map([arrow_cast(1,'UInt32')],['a']) UNION ALL SELECT map([arrow_cast(2,'Int32')],['b']);
+----------------------------------------------------------------------------+
| map(make_array(arrow_cast(Int64(1),Utf8("UInt32"))),make_array(Utf8("a"))) |
+----------------------------------------------------------------------------+
| {1: a}                                                                     |
| {2: b}                                                                     |
+----------------------------------------------------------------------------+
2 row(s) fetched.
Elapsed 0.062 seconds.

> \q

datafusion on  map-evolution-03-20835 [$?] is 📦 v54.1.0 via  v24.9.0 via 🐍 v3.12.8 via 🦀 v1.97.0 on ☁️  adrian@pydantic.dev(us-east4) took 2s
❯ cargo run -p datafusion-cli
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.51s
     Running `target/debug/datafusion-cli`
DataFusion CLI v54.1.0
> SELECT map([arrow_cast(1,'UInt32')],['a']) UNION ALL SELECT map([arrow_cast(2,'Int32')],['b']);
Optimizer rule 'simplify_expressions' failed
caused by
This feature is not implemented: Unsupported CAST from Map("entries": non-null Struct("key": non-null UInt32, "value": Utf8), unsorted) to Map("entries": non-null Struct("key": non-null Int64, "value": Utf8), unsorted)

[adriangb]

@kosiew ill let you decide when this is ready to merge

[kosiew]

🚀
@adriangb
Thanks for your review and feedback.