Repository navigation
Add parquet support for importing and exporting data to/from DataFrame. #5972
Description
At the current moment, the only supported way to import/export data is to use CSV format; it would be very useful to add support for some more commonly used data formats or databases.
Reacted by Stoyan Dimov, bolcman, Galen Rice and Mark Walls
Activity
michaelgsharp commentedon Oct 14, 2021
What other data formats are you thinking of? If we can get specific formats people want it helps with prioritizing things.
andrei-faber commentedon Oct 14, 2021
@michaelcfanning I think that Parquet would be useful too, not sure about other formats.
michaelcfanning commentedon Oct 14, 2021
I think you meant @michaelgsharp but, since I'm here, I agree Parquet would be useful. :)
andrei-faber commentedon Oct 14, 2021
@michaelcfanning ah sorry my bad :)
michaelgsharp commentedon Oct 15, 2021
I'll go ahead and change this title to be adding parquet support then.
Glad you could join us, other Michael :)
[-]Add more ways to import and export data to/from DataFrame[/-][+]Add parquet support for importing and exporting data to/from DataFrame.[/+]adamreeve commentedon Jan 12, 2022
Hi, I'm one of the maintainers of ParquetSharp, and we've just released the first version of ParquetSharp.DataFrame which supports reading Parquet into DataFrames and exporting DataFrames to Parquet. The API isn't yet stable but functionality should be fairly complete so we'd love to get some feedback.
Microsoft may still want to provide built-in Parquet support in Microsoft.Data.Analysis so this probably doesn't fix this issue, but is hopefully useful to anyone following this issue, and if our work could be useful for providing that built-in Parquet support we'd be happy to help out where we can.
michaelgsharp commentedon Jan 18, 2022
@adamreeve thanks for pointing this out!
We are currently in the process of creating/prioritizing our dataprep plan so we will definitely keep this in mind.
pgovind commentedon Jan 28, 2022
I've been out of the loop here for a while, so this may not be as accurate anymore: FWIW, at one point the main thought here was to create a separate
DataFrame.IOlibrary that could incrementally add support for the different data formats in common use. Just something to consider.totalgit74 commentedon Nov 18, 2022
I like the idea of the separate IO library and believe that performant access to a format such as Parquet that will allow interop between .Net and Python as users see fit would be a good thing. CSVs aren't particularly good for maintaining types or large data sets.
x202611 commentedon Apr 17, 2023
DataFrame only seems to support primitive data types, unlike parquet which supports arrays, structs, maps, lists etc.
Therefore I can't see this working unless DataFrame adds support for complex types?
andrei-faber commentedon Apr 18, 2023
@aloneguid hm doesn't it support complex types like DataFrame in Spark does?
In any case, it's still possible to make it import/export files that contain primitive types only.
totalgit74 commentedon Apr 18, 2023
My concern currently is the limited amount of data interop between different languages. The defacto is currently Pandas which supports both parquet and feather. In order to gain better traction any libraries/languages positioning themselves in this space need to be able to interop large data sets with the defacto else they will find it very hard to improve their usage.
I am hitting this issue where, for a lot of things C# is clearly the better language choice but due to the inability to deal with a viable data interop format (CSV is not it) that Python or R can use it is a non-starter. This leads to the instance where yet another company will decrease its use of a language.
As @andrei-faber mentions - you don't need to support every type a format can accept first up, but supporting primitive types fully would be a start that could be built upon.
x202611 commentedon Apr 19, 2023
I couldn't find it. There is #6088 that requests to add a simple array support but that's all. Looking at pandas, even struct support is a bit awkward, but I'm not an expert, maybe someone can clarify why there is only basic types support, is this sufficient?
rosebyte commentedon Sep 30, 2026
Triage: it would make sense, however, we don't see high demand for the change, if you have a real world use case, please create a new issue so we can reconsider.
totalgit74 commentedon Oct 1, 2026
The real world use case would be persistence of dataframes and interop with other languages like R and Python. Without easy interop you will struggle to make headway against an established player like Python. With the ability to easily share data between them you can choose the best tool for the job. Open is always better.
matouskozak commentedon Oct 1, 2026
Thank you @totalgit74 , that's a definitely a valid use case and something that would bring parity with libraries like Pandas. However, we currently do not have capacity for adding new
DataFrame.IOformats and we will rather focus on supporting the existing one.Comments like this or 👍 on existing issues help us gauge demand and plan work for the future.
In the meantime, there is a community package https://github.com/G-Research/ParquetSharp.DataFrame that can read and write Parquet.Update: as mentioned by @aloneguid , the https://github.com/aloneguid/parquet-dotnet also provides a support for read/write of parquet files and its version is already part of ML.NET depedency chain.
x202611 commentedon Oct 1, 2026
Parquet library ML.NET is already using already supports DataFrame. I'd advise not to add G-Research dependency which is a wrapper around C++, unlike the already existing managed implementation.
matouskozak commentedon Oct 1, 2026
Thank you @aloneguid , I updated the original comment. If you would like to help with adding the support, it would be beneficial for us to see an API proposal and plan of changes needed so that we can evaluate the scope of this work. As mentioned above, unfortunutely, we currently do not have capacity to drive this forward.