Skip to content

Audit multi-child Distribution::KeyPartitioned requirements before general Range satisfaction #23451

Description

@gene-bordegaray

Related:

Is your feature request related to a problem or challenge?

General Partitioning::Range satisfaction for Distribution::KeyPartitioned is tracked by #23266. Multi-child operators are harder than single-input operators because independently key-partitioned inputs are not always co-partitioned by partition index see #23184 for more contect on co-partitioning.

Some multi-child operators still declare per-child KeyPartitioned requirements directly. Before range satisfaction becomes general, these operators should either use a co-partitioning requirement to guard them to not be satisfied by independent children requirements alone.

Example

SortMergeJoinExec: join_type=Inner, on=[(range_key@0, range_key@0)]
  DataSourceExec: output_partitioning=Range([range_key@0 ASC], [(10), (20), (30)], 4)
  DataSourceExec: output_partitioning=Range([range_key@0 ASC], [(15), (20), (30)], 4)

Each input may independently satisfy KeyPartitioned(range_key), but the split points are not compatible. A partition-index aware operator needs co-partitioning proof before it can skip repartitioning.

Describe the solution you'd like

Audit multi-child physical operators that declare KeyPartitioned input requirements and convert partition-index aware operators to InputDistributionRequirements::co_partitioned(...).

Operators to audit (please do your own pass as well):

  • SortMergeJoinExec
  • SymmetricHashJoinExec

Acceptance criteria

  • Partition-index aware multi-child operators use explicit co-partitioning requirements.
  • Incompatible independently key-partitioned inputs fail the executable-plan invariant.
  • Tests cover compatible and incompatible cross-child key distributions.
  • Allow Partitioning::Range to satisfy Distribution::KeyPartitioned generally #23266 can later enable Range satisfaction without treating independent per-child satisfaction as co-partitioning.

Additional context

This should happen before or as part of #23266 so general range satisfaction does not accidentally treat per-child key colocation as cross-child co-partitioning.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions