What is the problem the feature request solves?
Many (most?) Spark SQL tests do not start with Parquet scan, but use RDDScanExec, LocalTableScanExec, etc. so Comet falls back to Spark very early. This means that we are missing out on test coverage.
Describe the potential solution
- Make
spark.comet.sparkToColumnar.enabled configurable via env var
- Set that env var in CI when running Spark SQL tests
- Fix failing tests (or ignore them until we can fix the bugs)
Additional context
No response
What is the problem the feature request solves?
Many (most?) Spark SQL tests do not start with Parquet scan, but use
RDDScanExec,LocalTableScanExec, etc. so Comet falls back to Spark very early. This means that we are missing out on test coverage.Describe the potential solution
spark.comet.sparkToColumnar.enabledconfigurable via env varAdditional context
No response