Better docs/samples for Data.SaveAsText & Data.SaveAsBinary

I do classification of MS Office document, in my case feature generation is relatively slow process, that why I want to do in once and save my data set to file like data.tsv and then experiment with different models and parameters.

Let's say we I defined model class

private class IrisInputAllFeatures
{
    [ColumnName("Label"), LoadColumn(4)]
    public string IgnoredLabel { get; set; }

    [LoadColumn(0, 3)]
    public float Features { get; set; }
}

then I wrote the code that generate / export data

var dataSet = new [] {
   new IrisInputAllFeatures() { ... },
   new IrisInputAllFeatures() { ... } 
}

the next thing that I expect is to be able to save all my data in file in format supported by LoadFromTextFile method (to be able reuse IrisInputAllFeatures class in both places), like

Data.SaveToTextFile("DataSet.tsv", dataSet);

Is there a way to save data to the file based on ColumnName, LoadColumn attributes?

I don't think ML.NET has an api to save the data.
The best way to go about it is to use C# csv libraries to save the data in csv format.
We can probably add such API. Adding the enhancement label.

@sfilipi

@sfilipi Don't we have MLContext.Data.SaveAsText extension in the TextLoaderSaverCatalog?

machinelearning/src/Microsoft.ML.Data/DataLoadSave/Text/TextLoaderSaverCatalog.cs

Line 189 in 205eb1d

public static void SaveAsText(this DataOperationsCatalog catalog,

@najeeb-kazmi

@najeeb-kazmi Thank you, it was not easy to find at all.

Looks like we can save & load IDataView to text or binary format

var mlContext = new MLContext();

IEnumerable<IrisInputAllFeatures> myData = GenerateData();
IDataView dataView = mlContext.Data.LoadFromEnumerable(myData);
 
// Save & Load from text file
using (var stream = File.Create(textFileName))
    mlContext.Data.SaveAsText(data, stream);
// Note: Here is important to use non-generic version of LoadFromTextFile
IDataView dataViewFromTextFile = mlContext.Data.LoadFromTextFile(textFileName);

// Save & Load from binary file
using (var stream = File.Create(binFileName))
    mlContext.Data.SaveAsBinary(data, stream);
IDataView dataViewFromBinFile = mlContext.Data.LoadFromBinary(binFileName);

@sfilipi Can we mark this issue as documentation improvement request?

Thanks @sfilipi & @najeeb-kazmi

Would #2811 help with the SaveAsBinary part?

Navigation Menu

Repository navigation

Better docs/samples for Data.SaveAsText & Data.SaveAsBinary #3661

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Better docs/samples for Data.SaveAsText & Data.SaveAsBinary #3661

Description

Activity

sfilipi commented on May 6, 2019

najeeb-kazmi commented on May 6, 2019

sergey-tihon commented on May 7, 2019

jwood803 commented on May 7, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions