Found while enabling spark.comet.exec.localTableScan.enabled by default in draft PR #6780 (Spark 4.1 SQL tests and [scans] job). Part of #4347.
With the default flipped (#6780, Spark 4.1), about 40 upstream tests fail only because the plan contains CometLocalTableScan / CometColumnarToRow where the test expects a Spark node. Examples: ReplaceNullWithFalseInPredicateEndToEndSuite SPARK-25860/33847, LocalScanSuite "full scan", JoinHintSuite "join strategy hint" (4), BroadcastJoinSuite (3), DataFrameSetOperationsSuite SPARK-52921 / SPARK-37371, CTEInlineSuite SPARK-36447 / "CTE Predicate push-down", DataFrameAggregateSuite SPARK-19471 / SPARK-31620, BatchEvalPythonExecSuite Python UDF push down (4), SQLAppStatusListenerSuite (SQLMetrics, codegen stage id), SPARK-46170, DatasetSuite SPARK-43124 / SPARK-47270, "Support ExplainMode". Decide per test: set localTableScan.enabled=false in the dev/diffs, or adapt the assertion.
Found while enabling
spark.comet.exec.localTableScan.enabledby default in draft PR #6780 (Spark 4.1 SQL tests and[scans]job). Part of #4347.With the default flipped (#6780, Spark 4.1), about 40 upstream tests fail only because the plan contains CometLocalTableScan / CometColumnarToRow where the test expects a Spark node. Examples: ReplaceNullWithFalseInPredicateEndToEndSuite SPARK-25860/33847, LocalScanSuite "full scan", JoinHintSuite "join strategy hint" (4), BroadcastJoinSuite (3), DataFrameSetOperationsSuite SPARK-52921 / SPARK-37371, CTEInlineSuite SPARK-36447 / "CTE Predicate push-down", DataFrameAggregateSuite SPARK-19471 / SPARK-31620, BatchEvalPythonExecSuite Python UDF push down (4), SQLAppStatusListenerSuite (SQLMetrics, codegen stage id), SPARK-46170, DatasetSuite SPARK-43124 / SPARK-47270, "Support ExplainMode". Decide per test: set localTableScan.enabled=false in the dev/diffs, or adapt the assertion.