DataFusion has an optimization where projections can be pushed down into hash joins. This implemented in the projection pushdown optimizer rule and was added in apache/datafusion#9236.
We do not currently apply this rule, so each HashJoin will output all columns and then we have a projection to discard the unwanted columns. Here is an example plan from TPC-DS q3 showing two projections after the outermost join. Pushing the projection into the join may show some small performance improvement.
AggregateExec: mode=Partial, gby=[col_0@0 as col_0, col_3@3 as col_1, col_2@2 as col_2], aggr=[sum]
ProjectionExec: expr=[col_0@0 as col_0, col_2@2 as col_1, col_1@4 as col_2, col_2@5 as col_3]
ProjectionExec: expr=[col_0@3 as col_0, col_1@4 as col_1, col_2@5 as col_2, col_0@0 as col_0, col_1@1 as col_1, col_2@2 as col_2]
HashJoinExec: mode=Partitioned, join_type=Inner, on=[(col_0@0, col_1@1)]
CopyExec [UnpackOrDeepCopy]
ScanExec: source=[BroadcastExchange (unknown)], schema=[col_0: Int32, col_1: Int32, col_2: Utf8]
CopyExec [UnpackOrClone]
ProjectionExec: expr=[col_1@1 as col_0, col_1@3 as col_1, col_2@4 as col_2]
HashJoinExec: mode=Partitioned, join_type=Inner, on=[(col_0@0, col_0@0)]
CopyExec [UnpackOrDeepCopy]
ScanExec: source=[BroadcastExchange (unknown)], schema=[col_0: Int32, col_1: Int32]
CopyExec [UnpackOrClone]
FilterExec: col_0@0 IS NOT NULL AND col_1@1 IS NOT NULL
ScanExec: source=[CometScan parquet (unknown)], schema=[col_0: Int32, col_1: Int32, col_2: Decimal128(7, 2)]
What is the problem the feature request solves?
DataFusion has an optimization where projections can be pushed down into hash joins. This implemented in the projection pushdown optimizer rule and was added in apache/datafusion#9236.
We do not currently apply this rule, so each HashJoin will output all columns and then we have a projection to discard the unwanted columns. Here is an example plan from TPC-DS q3 showing two projections after the outermost join. Pushing the projection into the join may show some small performance improvement.
Describe the potential solution
No response
Additional context
No response