Repository navigation
List Loader #195
Description
I was looking through the ml.net api for a generalized data loader, and I could not find one. So here is my proposal:
Take the following example for the TextLoader:
var dataPath = "intents.txt";
pipeline.Add(new TextLoader<IntentData>(dataPath, separator: "tab"));
I would like to have something like this:
var intents = new List<IntentData();
intents.Add(new IntentData());
pipeline.Add(new ListLoader<IntentData>(intents));
With a generalized loader like this I would have a wider range of sources for my data.
Activity
danmoseley commentedon May 21, 2018
We are considering a broader set of loaders. What sources are you interested in using?
GalOshri commentedon May 21, 2018
Would something like the
CollectionDataSourcework for this? See example usage here.This is not in the docs yet as it was added after the v0.1 release, but it will be added there soon.
dotChris90 commentedon May 21, 2018
I just saw it - what are your ideas about this loaders? Will they all implement the same Interface? In my opinion thats quite important. Just like in EF Core the SQL connectors implement totally the same Interface.
Sorrien commentedon May 21, 2018
I would like to use JSON, SQL, and Entity Framework Core primarily. @danmosemsft
Sorrien commentedon May 21, 2018
@GalOshri That does look a lot like what I want, and I was not aware of that. I look forward to that being added.
veikkoeeva commentedon May 21, 2018
For relational I'd like to use just plain ADO.NET, possibly Dapper or something alike. In other words, do not make EF a required dependency. :)
Sorrien commentedon May 21, 2018
I'd prefer to be able to pass in data objects as a list to prevent exactly that. So you can use the database wrapper you want to use and then avoid any dependencies.
veikkoeeva commentedon May 22, 2018
@Sorrien It's a good point. Maybe we should consider
IEnumerable,IAsyncEnumerableIObservable(TPL, F# equivalents, what have you, just to make sure the design adapts) also and if it were possibly so the pipeline could drain the source if needed. I'm not sure aboutStream. I could imagine there'd be performance issues with larger data sets if it all would be needed in memory up-front.Sorrien commentedon May 22, 2018
I have now tested CollectionDataSource with the latest build of the project and it is exactly what I was looking for.
removed dummy command (dotnet#195)
code migration from machinelearning-automl to machinelearning repo fe…
removed dummy command (dotnet#195)
Added onnx export functionality for LpNormNormalizingTransformer (#4161)
Added Onnx export functionality to PCATransformer (#4188)