Introduce unified `DataSourceExec` for provided datasources, remove `ParquetExec`, `CsvExec`, etc - #14224

Merged

ozankabak merged 86 commits into

apache:mainfrom

synnada-ai:chore/single-source-exec

Feb 6, 2025

Commits on Jan 7, 2025

unify ParquetExec, AvroExec, ArrowExec, NDJsonExec, MemoryExec into one DataSourceExec plan
mertak-synnada
committed
5b1714c
Merge branch 'refs/heads/apache_main' into chore/single-source-exec
mertak-synnada
committed
d2fe022
fix license headers
mertak-synnada
committed
70505b3
fix compile errors on documents
mertak-synnada
committed
00db759
separate non-parquet code
mertak-synnada
committed
cea3ecd

Commits on Jan 8, 2025

format code
mertak-synnada
committed
2331577
fix typo
mertak-synnada
committed
15b812f
fix imports
mertak-synnada
committed
f020bd2
fix clippy
mertak-synnada
committed
a7a5bd8
Merge branch 'refs/heads/apache_main' into chore/single-source-exec
mertak-synnada
committed
94a306f
add comment to the example
mertak-synnada
committed
d69b012
fix cargo docs
mertak-synnada
committed
f14a6d8
change MemoryExec with MemorySourceConfig
mertak-synnada
committed
8040147

Commits on Jan 10, 2025

Merge branch 'refs/heads/apache_main' into chore/single-source-exec
mertak-synnada
committed
f540df0
merge fixes
mertak-synnada
committed
ddb221d
change MemoryExec to DataSourceExec
mertak-synnada
committed
069d28c

Commits on Jan 15, 2025

Merge branch 'refs/heads/apache_main' into chore/single-source-exec
mertak-synnada
committed
78dfce8
fix merge conflicts
mertak-synnada
committed
cb6a5ff
apply some syntactic sugars
mertak-synnada
committed
4acb2fc
fix imports and comment line
mertak-synnada
committed
ff68caa
simplify some lines
mertak-synnada
committed
7a62190
rename source_config as file_source
mertak-synnada
committed
fd37183
format code
mertak-synnada
committed
cf13028
format code
mertak-synnada
committed
201d8a0
make memory metrics default behavior
mertak-synnada
committed
7edc80f

Commits on Jan 16, 2025

remove unnecessary cfg check
mertak-synnada
committed
fe25de3
format code
mertak-synnada
committed
2452825
Merge branch 'refs/heads/apache_main' into chore/single-source-exec
mertak-synnada
committed
e084359
remove ParquetExec strings
mertak-synnada
committed
a4d5da3
fix documents and imports
mertak-synnada
committed
a6c018e
fix imports
mertak-synnada
committed
104c428

Commits on Jan 17, 2025

Merge branch 'refs/heads/apache_main' into chore/single-source-exec
mertak-synnada
committed
38134d3

Commits on Jan 20, 2025

add constraints and fix tests
mertak-synnada
committed
4af421e
delete redundant file
mertak-synnada
committed
91110cc
make metrics and statistics a part of File type specific configurations
mertak-synnada
committed
6c76b3f
format code
mertak-synnada
committed
12f0ac8
Merge branch 'refs/heads/apache_main' into chore/single-source-exec
mertak-synnada
committed
f528bf5
fix tests
mertak-synnada
committed
887922d
format code
mertak-synnada
committed
91fb10e

Commits on Jan 21, 2025

split repartitioning into DataSourceExec and FileSourceConfig parts
mertak-synnada
committed
cb8c2ae
move properties into DataSourceExec and split eq_properties and output_partitioning in DataSource trait
mertak-synnada
committed
64ccad7
clone source with Arc
mertak-synnada
committed
aa047d7
return file type as enum and do not downcast if not necessary
mertak-synnada
committed
93888a7
Merge branch 'refs/heads/apache_main' into chore/single-source-exec
mertak-synnada
committed
44f21a2
format code
mertak-synnada
committed
aabcd04
re-add deprecated plans in order to support backward compatibility
mertak-synnada
committed
18da494
reduce diff
mertak-synnada
committed
82b5257
fix doc
mertak-synnada
committed
6ec3aee

Commits on Jan 23, 2025

Merge branch 'refs/heads/apache_main' into chore/single-source-exec
mertak-synnada
committed
aeb1954
merge fixes
mertak-synnada
committed
f42c245
remove unnecessary files
mertak-synnada
committed
3b8b4e9

Commits on Jan 24, 2025

rename config structs to source
mertak-synnada
committed
de2c8e4
Merge branch 'refs/heads/apache_main' into chore/single-source-exec
mertak-synnada
committed
7112f2f
remove empty files
mertak-synnada
committed
23b272f
removed FileSourceConfig
mertak-synnada
committed
0fe7f00
fix base_config formatting
mertak-synnada
committed
1eb5343
format code
mertak-synnada
committed
b03ed3d
fix repartition logic
mertak-synnada
committed
7acb09a

Commits on Feb 3, 2025

Merge branch 'refs/heads/apache_main' into chore/single-source-exec
mertak-synnada
committed
72440dd
fix merge conflicts
mertak-synnada
committed
165f0ce
fix csv projection error
mertak-synnada
committed
184f293
clippy fix
mertak-synnada
committed
ae0e6a7
use new() on initialization
mertak-synnada
committed
e43a9fa

Commits on Feb 4, 2025

Merge branch 'refs/heads/apache_main' into chore/single-source-exec
mertak-synnada
committed
105c488
use DataSourceExec on deprecated file operators as well
mertak-synnada
committed
d9d3d64
move ParquetSource into source.rs
mertak-synnada
committed
7fe6a28
use ParquetSource only if parquet feature is enabled
mertak-synnada
committed
a6cbab1
Merge branch 'refs/heads/apache_main' into chore/single-source-exec
mertak-synnada
committed
dd4c7ac
fix slt tests
mertak-synnada
committed
f2d31a0

Commits on Feb 5, 2025

add with_fetch API to MemorySourceConfig and re-add deprecated MemoryExec
mertak-synnada
committed
4dd7b73
Merge branch 'refs/heads/apache_main' into chore/single-source-exec
mertak-synnada
committed
25badc4
fix merge conflicts
mertak-synnada
committed
e8b6232
format code
mertak-synnada
committed
962494a

Commits on Feb 6, 2025

change FileType enum into a dyn Trait so that it can be extensible
mertak-synnada
committed
da67917
remove metadata_size_hint from required ParquetSource parameters
mertak-synnada
committed
6957121
remove FileType trait and split with_predicate logic for ParquetSource
mertak-synnada
committed
d39e441
remove predicate from initialization of ParquetSource
mertak-synnada
committed
4c32ca2
remove unnecessary imports
mertak-synnada
committed
91071bd
deprecate ParquetExecBuilder and add doc hints
mertak-synnada
committed
e2d5a8d
Merge branch 'refs/heads/apache_main' into chore/single-source-exec
mertak-synnada
committed
81179ae
fix slt
mertak-synnada
committed
9a32a0a
fix clippy
mertak-synnada
committed
72059ee
fix fmt
mertak-synnada
committed
3e10283
return reference of the Arc in source()
mertak-synnada
committed
a952c41
re-add deprecated exec files
mertak-synnada
committed
9734ddf
fix doc error
mertak-synnada
committed
f797aaa