[EPIC] ListingTable object store usage improvements

[alamb]

This is a list of improvements we are working on in ListingTable in DataFusion

Background

DataFusion has a ListingTable that effectively reading tables stored in one or more files in a "hive partitioned" directory structure:

So for example, give files like this:

/path/to/my_table/file1.parquet
/path/to/my_table/file2.parquet
/path/to/my_table/file3.parquet

You can create a table with a command like

CREATE EXTERNAL TABLE my_table
LOCATION '/path/to/my_table'

And the ListingTable will handle figuring out schema, and running queries against those files as though they were a single table.

Team

• @BlakeOrth
• @alamb (maintainer)

Bugs

• Bug: ListingTableFactory fails to read data when the final path element contains a . #17212
• Auto detecting partitions with ListingTableFactory on Hive partitioned datasets #17049

Enhancements

• [Epic] Enable parquet metadata cache by default #17000
• Improved experience when remote object store URL does not end in / (retry as paths) #16302
• [datafusion-cli] Add a way to see what object store requests are made #17207
• Make datafusion-cli object store tracing mode work for local files #18119
• Reduce number of object store requests when reading parquet files by default (set metadata_size_hint) #18118
• Enable the ListFilesCache to be available for partitioned tables #17211
• Move the DefaultMetadataCache into its own module #19074
• Add a default ListFilesCache implementation for the ListingTable #18827
• Add a way to dynamically configure / update the ListFilesCache settings via RuntimeConfiguration #19056
• Partitioned object store lists all files on every query when using hive-partitioned parquet files #9654
• Bug: statistics not collected automatically upon creation of ListingTable #18952
• Add a way to show the contents of the FileStatisticsCache in datafusion-cli #18953
• Add a way to show the contents of the ListFilesCache in datafusion-cli #19055
• [datafusion-cli] Implement average LIST duration for object store profiling #18138
• Make DefaultListFilesCache work better with prefixed paths #19273
• Confusing behavior now required to to refresh the files of a listing table #19573
• Add a default FileStatisticsCache implementation for the ListingTable #19217
• Add limit to DefaultFileStatisticsCache #19052

[alamb]

I added myself and @BlakeOrth as people working on this epic

[alamb]

A brief update here is that I think the next steps for this project are:

1. Polish up [datafusion-cli] Add a way to see what object store requests are made #17207 so we can test / observe the improvements
2. Complete Enable the ListFilesCache to be available for partitioned tables #17211

@BlakeOrth has prototype PRs for both, and they need some help polishing / testing

[BlakeOrth]

@alamb I haven't pushed and opened a PR for #17211 yet, but I would be happy to do so if we want to start getting some feedback on the implementation. I actually think that code is more or less ready to go at this point, but I don't want to overload anyone on reviews when we still need #17207 to test/validate those changes.

I will comment in #17207 and cc you there so we can choose how to continue moving forward with that effort.

This all comes with a subtle caveat that I am currently focusing on some other efforts that unfortunately skyrocketed to the top of my priorities, so while I am more than happy to continue contributing to this effort I will be unlikely to do so for about the next week or so.

[alamb]

Thanks @BlakeOrth -- what I am secretly (well not so secretly anymore) is that we have this ready for DataFusion 51 which won't be until Nov, so we have some time

• Release DataFusion 51.0.0 (Nov 2025) #17558