Repository navigation
DataFrame LoadSql Method #5662
Description
I was playing around with the DataFrame library to start to build a SQL reconciliation tool, and I noticed that there is no easy way to go from SQL query to DataFrame.
I have some code for that can create the DataFrame from a SqlDataReader, but we may want an easier to use API than that.
One idea is to follow the convention used for the LoadCsv function and make something similar to the pandas function read_sql.
I am happy to put a PR together depending on the approach chosen.
Reacted by Prashanth Govindarajan and Marcus Adison Hudgins
Activity
pgovind commentedon Mar 20, 2020
I'm assuming this will take in a raw SQL query as an argument to
LoadSql? We should get input from @eerhardt and possible @tannergooding who most recently worked on some SQL readers. In particular, how .NETy is it to take in a SQL query as a parameter? What's the alternative otherwise?eerhardt commentedon Mar 20, 2020
A thing we need to be conscious about is that the dependencies of the core
DataFramelibrary don't grow too unwieldy. Whenever we add a new dependency to the coreDataFramelibrary, we have to consider if it is beneficial for the majority of use cases.I definitely see the value in an easy API to load a DataFrame from SQL, but I don't believe it is "core" to the library. For example,
DataFrameis used in https://github.com/dotnet/spark, which wouldn't want/need a dependency on SQL libraries.So I would see something like this as an extension to the
DataFramelibrary that ships in a separate NuGet package. We've designed theDataFrameAPI such that external libraries can "fill" or "load" one up from other data sources (ex. SQL).rspaulino commentedon Sep 2, 2020
Hi. I see that the code had been written for this, is there a design conflict or just the unit test that's is missing to merge this to main?
thanks
eerhardt commentedon Sep 3, 2020
It looks like it is blocked on adding tests.
rspaulino commentedon Sep 3, 2020
@eerhardt thanks.
How extensive the test have to be? I got it running on my computer and have use it with Sqlite, I try and help so it can be merge on the main.
eerhardt commentedon Sep 4, 2020
The tests should exercise the main functionality to ensure it works, and doesn't get broken in the future.
That would be great!
rspaulino commentedon Sep 4, 2020
👍🏾 will try and give a shot then