Enable spark.comet.exec.localTableScan.enabled when running Spark SQL tests #4347

Related to #2723.

A lot of Spark SQL tests don't put their source data in a format that Comet accelerates reading from. Comet generally wants data in Parquet or Iceberg. A number of Spark SQL suites (e.g., UDFSuite) don't write to Parquet so their scans are just LocalTableScanExec, and Comet is not exercising UDF compatibility because the scan underneath isn't native. We can either do what #2723 suggests and enable converting to columnar, or try enabling LocalTableScanExec now that we have #2735. @andygrove tried the former in #2714, but the logs are gone now so I'm not sure how ugly it was. Starting with LocalTableScanExec might be a smaller blast radius?

Tracking

Enabling spark.comet.exec.localTableScan.enabled by default is being tried in #4393 (draft) and #6780 (draft, run against Spark 4.1). Remaining work, by area:

Test expectations that assume Spark nodes (skip list or per-test config)

• test: Spark SQL tests that inspect the physical plan fail when LocalTableScanExec is native #6795 Spark SQL tests that inspect the physical plan fail once LocalTableScanExec is a Comet operator

Known Comet gaps newly exposed by the change

• collect_set / collect_list return elements in a different order than Spark #6796 collect_set / collect_list element order differs from Spark
• Math functions differ from Spark in the last digits #6797 Math function results differ from Spark in the last digits (asinh, cosh, tan, cot, atanh, cbrt, exp, pow, atan2)
• Native errors surface as SparkException where Spark raises a specific error class #6798 Native errors surface as SparkException where Spark raises a specific error class (to_time, to_binary, array_insert, map construction)
• percentile of -0.0 returns 0.0 #6799 percentile returns 0.0 instead of -0.0
• from_csv: wrong results in permissive modes and failure with a variant schema #6800 from_csv returns wrong results in permissive modes and fails with a variant schema
• GROUP BY on a struct containing a map fails with CometNativeException #6801 GROUP BY on a struct containing a map fails with CometNativeException
• Null-named attribute fails Arrow export with "field name cannot be null" #6802 Null-named attribute fails Arrow export (field name cannot be null)

Iceberg

• CometIcebergWriteDetectionSuite assumes a split Iceberg write plan #6803 CometIcebergWriteDetectionSuite expects the split Iceberg write plan but gets a fully native write over CometLocalTableScan (related to feat: enable native Iceberg writes by default #6664)

Done: #4789, #4786, #4787, #4788, plus the NullType and TimeType handling from #4393.