[EPIC] Improve performance of TPC-DS queries

[andygrove]

What is the problem the feature request solves?

This issue is for tracking issues to perform Comet's performance when running benchmarks derived from TPC-DS.

Highest Priority

• Add InSubqueryExec support (to support DPP) #121 to support DPP
• Support sort merge join with a join condition #398
• Support RangePartitioning with native shuffle #458
• Avoid JVM shuffle when parent stage will just convert back to rows #4004
• Enable windowed aggregates and fix correctness issues #4007
• Avoid fallbacks for mixed partial/final aggregates (AQE may materialize a non-supported Final-mode HashAggregate #1389 and others)

Other

• [EPIC] Improve shuffle performance #1123
• Add option to replace SortMergeJoin with ShuffleHashJoin #1006
• Add support for bloom_filter_agg #846
• Add support for date_add - feat: date_add and date_sub functions #910
• stddev_sample disabled by default due to performance issue - Improve performance of standard deviation aggregate #824

Describe the potential solution

No response

Additional context

No response

[coderfender]

There has been a lot of work put in TPCDS / TPCDH . Keeping this issue open to track window aggregate support @comphead is working on

[coderfender]

#4007

[coderfender]

Assigning to @comphead Thank you

[andygrove]

Every item on this list is now closed, including windowed aggregates in #4007 and the Final-mode aggregate fallback in #1389. The remaining mixed Spark partial / Comet final gap for COUNT is tracked in #4242, and the broader TPC-DS goals are tracked in #2551 and #4936, so I am closing this EPIC.