You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Aug 2, 2023. It is now read-only.
Repository navigation
This repository was archived by the owner on Aug 2, 2023. It is now read-only.
Why was indexing into a DataFrame removed in latest? Looking at the commit history too I can see the block was deleted. This makes very an awkward use experience now.
Just out of curiosity, are you using DataFrame in a notebook? Reason I ask is that we've worked on a cool extension for DataFrame in notebooks that'll let you write df.Int1 * 2 + df.Int2. To be specific, with the new extension you can now refer to a column as a field of a DataFrame object. With intellisense enabled in notebooks, this will be very discoverable.
Haven't tested the new notebook support yet but will do.
I would say that even though notebooks are very useful, I much prefer the experience to be the same when doing normal software and when doing notebooks.
Usually I prototype in notebooks and then structure and copy stuff to some kind of software that is more production-like. So I would avoid using any extensions in notebook except for ones that are interactive such as plotting.
For instance, I would expect code that runs in F# kernel notebook can be run in FSI under Visual Studio directly. I would also expect it to be compiled to be part of a bigger production system mixed with C# and F#. That's how I explore my problems in Ifsharp notebook and put them in production all the time.
However, if syntax involving dataframe relies on an extra notebook extension that only works in notebook, the beauty of production-ready scripts is no longer feasible.
Just tagging @eerhardt for visibility here. This is great feedback! We're busy helping out with .NET 5 stuff this week, but I'll revisit this next week. There's enough support here to consider bringing back the column name indexer on DataFrame.
Just out of curiosity, are you using DataFrame in a notebook? Reason I ask is that we've worked on a cool extension for DataFrame in notebooks that'll let you write df.Int1 * 2 + df.Int2. To be specific, with the new extension you can now refer to a column as a field of a DataFrame object. With intellisense enabled in notebooks, this will be very discoverable.
However, if syntax involving dataframe relies on an extra notebook extension that only works in notebook, the beauty of production-ready scripts is no longer feasible.
Yes, we need to be very careful about promoting non-standard extensions to the programming model for C# or F# which are only deployed only through select channels. Notebook programming should ideally not be using variations of these programming languages, though these things are subtle
This is a tricky area because there is a notable tendency to use the incremantal-dynamicity of notebook programming
@pgovind What APIs are you using to craft this language variation? Please discuss this with @MadsTorgersen, @jaredpar and myself. We can't have random variations on C# and F# floating around that fragment the overall programming experience.
What APIs are you using to craft this language variation?
It's not really a variation. It's a prototype right now (and not part of the type itself). Basically, given a DataFrame object, it looks at the types of the columns and spits out code to create a new SomeNameDataFrame type with the column names as properties. This code is then compiled on demand and the dotnet-interactive shell then exposes this type for use in the notebook.
@pgovind The problem is that this sort of "generating API from dynamic data" is a completely new thing in the .NET universe (the closest thing is F# type providers, and then source generators, though those are normally part of the static toolchain).
It doesn't really fit any existing part of the existing C#/F#/.NET programming model and can never really be incorporated into project-based programming, for example. It can only be done in notebook-like environments that assume a complete compiler toolchain at each stage of execution, even in production scripts.
It's a powerful thing to be sure but we have to be aware of the direction this is going. I understand why you're thinking of doing this but yes, fragmentation of the programming experience is an intrinsic part of this direction, as tempting as it is.
An approach that does fit within existing norms is to drive the code generation off some kind of static schema (declared or acquired).
Activity
pgovind commentedon Jun 15, 2020
Hello! Which API are you talking about? I still see this:
corefxlab/src/Microsoft.Data.Analysis/DataFrame.cs
Line 72 in f44a099
Address remaining concerns from the 2nd DataFrame API Review (#2861)
JamesAlexander42 commentedon Jun 15, 2020
I referred above the change I'm referring to
pgovind commentedon Jun 15, 2020
Ah I see. It still exists! We just moved that to the
DataFrameColumnCollectionclass. See this for an example:corefxlab/tests/Microsoft.Data.Analysis.Tests/DataFrameTests.cs
Line 648 in f44a099
rhysparry commentedon Jun 18, 2020
I think @hamaze was specifically asking why the API was changed that way. Maybe you can point to the API review?
I know that I'd rather write:
As opposed to:
JamesAlexander42 commentedon Jun 18, 2020
Yeah, the former is more pandas-esque and comfortable IMO.
MikaelUmaN commentedon Jul 4, 2020
Agreed.
I suspect the reason is for the row filter to work.
But you are more often interested in column selection than row selection so it's better to have the penalty there instead.
pgovind commentedon Jul 28, 2020
Just out of curiosity, are you using DataFrame in a notebook? Reason I ask is that we've worked on a cool extension for DataFrame in notebooks that'll let you write
df.Int1 * 2 + df.Int2. To be specific, with the new extension you can now refer to a column as a field of aDataFrameobject. With intellisense enabled in notebooks, this will be very discoverable.JamesAlexander42 commentedon Jul 30, 2020
I'm not using it in a notebook context for this exercise. Using it in an asp.net app.
MikaelUmaN commentedon Aug 20, 2020
Haven't tested the new notebook support yet but will do.
I would say that even though notebooks are very useful, I much prefer the experience to be the same when doing normal software and when doing notebooks.
Usually I prototype in notebooks and then structure and copy stuff to some kind of software that is more production-like. So I would avoid using any extensions in notebook except for ones that are interactive such as plotting.
zyzhu commentedon Aug 20, 2020
I concur with @MikaelUmaN.
For instance, I would expect code that runs in F# kernel notebook can be run in FSI under Visual Studio directly. I would also expect it to be compiled to be part of a bigger production system mixed with C# and F#. That's how I explore my problems in Ifsharp notebook and put them in production all the time.
However, if syntax involving dataframe relies on an extra notebook extension that only works in notebook, the beauty of production-ready scripts is no longer feasible.
cc @cartermp @dsyme to chime in.
pgovind commentedon Aug 20, 2020
Just tagging @eerhardt for visibility here. This is great feedback! We're busy helping out with .NET 5 stuff this week, but I'll revisit this next week. There's enough support here to consider bringing back the column name indexer on DataFrame.
eerhardt commentedon Aug 20, 2020
I agree. Personally I like the ease of use of
df["Int1"]as well, so I'm glad I'm not alone.It should be pretty easy to add the API back as a wrapper over the
.Columns[string]indexer, and a test or two. Anyone want to make a PR for that?dsyme commentedon Aug 24, 2020
Yes, we need to be very careful about promoting non-standard extensions to the programming model for C# or F# which are only deployed only through select channels. Notebook programming should ideally not be using variations of these programming languages, though these things are subtle
This is a tricky area because there is a notable tendency to use the incremantal-dynamicity of notebook programming
@pgovind What APIs are you using to craft this language variation? Please discuss this with @MadsTorgersen, @jaredpar and myself. We can't have random variations on C# and F# floating around that fragment the overall programming experience.
pgovind commentedon Aug 24, 2020
So, just to be clear, the extension I'm talking about here is only a prototype to explore the dotnet-interactive extensions APIs. There's no immediate plans to productize it right now, and we definitely don't want to create fragmentation. It lives here: https://github.com/dotnet/interactive/blob/main/src/Microsoft.DotNet.Interactive.ExtensionLab/DataFrameTypeGeneratorExtension.cs
dsyme commentedon Aug 25, 2020
@pgovind The problem is that this sort of "generating API from dynamic data" is a completely new thing in the .NET universe (the closest thing is F# type providers, and then source generators, though those are normally part of the static toolchain).
It doesn't really fit any existing part of the existing C#/F#/.NET programming model and can never really be incorporated into project-based programming, for example. It can only be done in notebook-like environments that assume a complete compiler toolchain at each stage of execution, even in production scripts.
It's a powerful thing to be sure but we have to be aware of the direction this is going. I understand why you're thinking of doing this but yes, fragmentation of the programming experience is an intrinsic part of this direction, as tempting as it is.
An approach that does fit within existing norms is to drive the code generation off some kind of static schema (declared or acquired).
ColumnName indexer on DataFrame
ColumnName indexer on DataFrame (#2959)
ColumnName indexer on DataFrame (dotnet#2959)
Move DataFrame to machinelearning (#5641)
Merge from main repository (#1)
Actually merge from main (#2)
Update editor config and format samples (#5893)