Repository navigation
VectorType attribute with dynamic dimension while dealing with csv files #6151
Description
I am working on training and evaluation in ML.NET. The data comes from a .csv file. According to the requirements, there may be variable number of columns in the file. Until now, I have achieved this by providing a fixed vectorType dimension. But I am not aware of making this work dynamically.
Here is the a block of code working using the fixed dimension,
public class InputClass
{
public bool Lavel;
[VectorType(5)] // I want this to be dynamic
public string[] SpecVec;
public string PredictedTarget;
}
public static void Main(string[] args)
{
MLContext mlContext = new MLContext(seed:0);
TextLoader.Columns[] columns = new TextLoader.Columns[3];
columns[0] = new TextLoader.Column("Label", DataKind.Boolean, 0);
columns[2] = new TextLoader.Column("PredictedTarget", DataKind.String, 1);
columns[1] = new TextLoader.Column("SpecVec", DataKind.String, 2,6);
IDataView dataview = mlContext.Data.LoadFromTextFile(filepath, columns,
separatorChar:',', hasHeader:true, allowQuoting:false, trimWhitespace: true,
allowSparse:true);
//-- creating pipeline and training a model further
}
This code works without any issue. But I just need the VectorType to accept the dimension at runtime. Now it accepts only constant values. I am looking if there is any workaround to achieve this. I am a beginner into ML.Net, please do understand if I missed any steps or anything to be explained. I really appreciate any help to get this worked.
Activity
michaelgsharp commentedon Apr 11, 2022
@rishi-git when you say dynamically what exactly are you meaning? If you do this:
It should mark
SpecVecas a vector of unknown size. (I think you can even leave off theVectorTypeand it will just do a vector of unknown size)For the text loader, you can use the constructor that takes a
Rangeto set it to load from a certain start point until the end of the datafile.Does something like this work in your situation?
rishi-git commentedon Apr 12, 2022
@michaelgsharp Thank you very much for your reply.
When I say dynamically, I mean the dimensions for VectorType attribute must be initialized at the runtime.
I tried using your code within the program.
But it throws an error as "System.ArgumentOutOfRangeException : Schema mismatch for score column 'Score': expected Single, got Vector<Single, 5>".
This error occurs at the time of evaluation at the following line of code,
mlContext.BinaryClassification.Evaluate(predictions, labelColumnName : "Label");
I appreciate any further help, thanks.
michaelgsharp commentedon Apr 26, 2022
Hmm.. If thats the case I wonder if we have a bug because just doing
[VectorType()]with no parameters should declare it as a vector type, but its clearly not working. Let me take a closer look and see what I can find.mitsuomiyazato commentedon Mar 3, 2024
Hey, did anybody resolve that problem? I am also dealing with this...