Microsoft.ML.OnnxRuntime.OnnxRuntimeException: 1 : FAIL : Failed to find kernel for Neg #4795

Activity

1. antoniovs1029 commented on Feb 6, 2020

   

   antoniovs1029

   on Feb 6, 2020

   Last edited by antoniovs1029

   Contributor

   More actions

   Hi, @patmaasridharan , can you please share your code, the ONNX model you're using, and a sample dataset to reproduce your problem? Thanks!
2. mstfbl

   added

   need info

   on Feb 6, 2020
3. patmaasridharan commented on Feb 7, 2020

   

   patmaasridharan

   on Feb 7, 2020

   Last edited by patmaasridharan

   Author

   More actions

   Onnx model can be found here.
   https://www.dropbox.com/s/1wypohlqp9mj41f/Unet.onnx?dl=0

   Sample code

   using System;
   using System.IO;
   using System.Collections.Generic;
   using System.Linq;
   using Microsoft.ML;
   using Microsoft.ML.Data;
   
   
   namespace OnnxMlUnet
   {
       class Program
       {
   
   
   
           private const string imagesFolder = "H:/c#code/OnnxMlUnet/OnnxMlUnet/assets/images";
           private static readonly string modelpath = "H:/c#code/OnnxMlUnet/OnnxMlUnet/assets/model/Unet.onnx";
           private const string outputimage= "H:/c#code/OnnxMlUnet/OnnxMlUnet/assets/images/mlout.jpg";
   
           static void Main(string[] args)
           {
               MLContext mlContext = new MLContext();
   
               IEnumerable<ImageData> images = ImageData.LoadImagesFromDirectory(imagesFolder);
   
               var data = mlContext.Data.LoadFromEnumerable(images);
               var pipeline = mlContext.Transforms.LoadImages(outputColumnName: "image", imageFolder: "", inputColumnName: nameof(ImageData.ImagePath))
                    .Append(mlContext.Transforms.ExtractPixels(outputColumnName: "image"))
                   .Append(mlContext.Transforms.ApplyOnnxModel(modelFile: modelpath));
   
               var model = pipeline.Fit(data);
               PredictionEngine<ModelInput, ModelOutput> predictionEngine = mlContext.Model.CreatePredictionEngine<ModelInput, ModelOutput>(model);
   
               ModelInput image = mlContext.Data.CreateEnumerable<ModelInput>(data, reuseRowObject: true).First();
               ModelOutput prediction = predictionEngine.Predict(image);
               File.WriteAllBytes(outputimage, prediction.Image);
               //MemoryStream ms = new MemoryStream(prediction);
               //Image i = Image.FromStream(ms);
   
           }
   
       }
       class ImageData
       {
           public string ImagePath { get; set; }
           public byte[] image { get; set; }
   
           public string outputimage { get; set; }
   
   
           private static byte[] readbytefromImage(string file)
           {
   
               Image img = Image.FromFile(file);
               byte[] arr;
               using (MemoryStream ms = new MemoryStream())
               {
                   img.Save(ms, System.Drawing.Imaging.ImageFormat.Jpeg);
                   arr = ms.ToArray();
               }
               return arr;
           }
   
           public static IEnumerable<ImageData> LoadImagesFromDirectory(string folder)
           {
   
               List<ImageData> retCollection = new List<ImageData>();
               var files = Directory.GetFiles(folder, "*", searchOption: SearchOption.AllDirectories);
               foreach (var file in files)
               {
                   if ((Path.GetExtension(file) != ".jpg") && (Path.GetExtension(file) != ".png"))
                       continue;
   
                   retCollection.Add(new ImageData()
                   {
                       //read the image here as byte array
                       //ImagePath = file,
                       image = readbytefromImage(file)
                   });
               }
   
               return retCollection;
           }
   
       }
       class ModelInput
       {
           public byte[] Image { get; set; }
   
           public string ImagePath { get; set; }
       }
   
   
       class ModelOutput
       {
           public byte[] Image { get; set; }
   
           public string OutImagePath { get; set; }
   
         
       }
   
   
   }

   Images
   
   

   Output expected
   
   
4. antoniovs1029

   self-assigned this

   on Feb 7, 2020
5. antoniovs1029 commented on Feb 8, 2020

   

   antoniovs1029

   on Feb 8, 2020

   Last edited by antoniovs1029

   Contributor

   More actions

   So, I haven't tested this, but have you actually tried to build the nuget from OnnxRuntime master branch?
   https://github.com/microsoft/onnxruntime

   You mention that you used Microsoft.Ml.OnnxRuntime v1.1.1 (which is the latest official released build from master in there), and the problem is that version was created on January 23, whereas the changes that fixed this issue for you on microsoft/onnxruntime#2945 was merged into master on January 30. So I wouldn't expect v1.1.1 to include that fix, and so perhaps you might need to build the nuget from there.
6. patmaasridharan commented on Feb 10, 2020

   

   patmaasridharan

   on Feb 10, 2020

   Last edited by patmaasridharan

   Author

   More actions

   Yes i did build the master of onnxruntime and linked the C# dll to my project. Then now i get another run time exception at OnnxTransformer. I use opsetversion 11 when exporting.

   

   I have tried to load the test model squeezenet.onnx provided by you and Inference Session works for them perfectly with latest ONNXRuntime build.
7. antoniovs1029 commented on Apr 14, 2020

   

   antoniovs1029

   on Apr 14, 2020

   Contributor

   More actions

   Hi, @patmaasridharan . Are you still having this problem?

   Sorry for the delay.

   Since your last update ML.NET v 1.5.0-preview2 was released, and it had several changes on OnnxTransformer and also has an updated dependency on OnnxRuntime 1.2.0.

   I've now tried your sample with your model and I didn't get your exception. So my guess is that that problem is now gone. The Onnx model seems to be correctly loaded by the OnnxTransformer, so that's not a problem now.

   Now the problem is that when applying the model, it expects an input of shape "5 x 1 x 256 x 256", whereas the images that you provided are loaded as "3 x 256 x 256". So I wonder where the "5" dimension comes from. Can you let me know?

   Also, if you want to run your pipeline with ML.NET v 1.5.0-preview2, please change your pipeline to be:

               var pipeline = mlContext.Transforms.LoadImages(outputColumnName: "image", imageFolder: null, inputColumnName: nameof(ImageData.ImagePath))
                    .Append(mlContext.Transforms.ResizeImages("image", inputColumnName: "image", imageWidth: 256, imageHeight: 256))
                    .Append(mlContext.Transforms.ExtractPixels(outputColumnName: "image"))
                    .Append(mlContext.Transforms.CopyColumns("input.1", "image"))
                    .Append(mlContext.Transforms.ApplyOnnxModel(modelFile: modelpath));

   And also, notice that you'll now need to install the OnnxRuntime 1.2.0 nuget in your project.
8. antoniovs1029 commented on Apr 14, 2020

   

   antoniovs1029

   on Apr 14, 2020

   Last edited by antoniovs1029

   Contributor

   More actions

   So I still don't know what the 5 "channels" of the input and output of the ONNX model are supposed to be.

   But I tried adding 2 more channels to the RGB channels that were already loaded from the images. I filled these 2 channels with '0's. And after making some changes to your code, I got this output:

   

   I don't know what the expected pre- and post-processing should be for the onnx model (including what to do with the extra channels). So if the output isn't as expected, it's because that would need to be fixed. On the other hand, notice that I was able to run the pipeline with your onnx model, without getting any exceptions. Because of this I'll close this issue. Please feel free to reopen it if you still have problems loading the onnx model into your ml.net pipeline.

   Here I leave you the code I made to generate the above output. Notice that since ML.NET doesn't have transformers to reshape arrays, I had to use a CustomMappingTransformer with a custom transformation to reshape the 3x256x256 loaded image into the 5x1x256x256 array that the onnx model expected as input.

   using System;
   using System.IO;
   using System.Collections.Generic;
   using System.Drawing;
   using System.Linq;
   using Microsoft.ML;
   using Microsoft.ML.Data;
   
   namespace OnnxMlUnet
   {
       class ModelInput
       {
           public string ImagePath { get; set; }
       }
   
       class ReshapeInput
       {
           [VectorType]
           public float[] loadedImage { get; set; }
       }
   
       class ReshapeOutput
       {
           [VectorType]
           public float[] reshapedImage { get; set; }
       }
   
       class ModelOutput
       {
           [VectorType]
           public float[] onnxOutput { get; set; }
       }
   
       class Program
       {
           private const string imagesFolder = @"C:\Users\anvelazq\source\repos\Bugs\is4795onnxerror\assets";
           private static readonly string modelpath = @"C:\Users\anvelazq\source\repos\Bugs\is4795onnxerror\Unet.onnx";
           private const string outputimage = @"C:\Users\anvelazq\source\repos\Bugs\is4795onnxerror\assets\mlout.bmp";
   
           static void Main(string[] args)
           {
               MLContext mlContext = new MLContext();
   
               // Load Data
               IEnumerable<ModelInput> images = LoadImagesFromDirectory(imagesFolder);
               var data = mlContext.Data.LoadFromEnumerable(images);
   
               // Define CustomTransformer action and schema definition
               // To reshape the loaded 3x256x256 images into arrays of 5x1x256x256 that are required by the onnx model input
               var reshapeSchemaDefinition = SchemaDefinition.Create(typeof(ReshapeOutput));
               reshapeSchemaDefinition[nameof(ReshapeOutput.reshapedImage)].ColumnType = new VectorDataViewType(NumberDataViewType.Single, new int[] { 5, 1, 256, 256 });
   
               Action<ReshapeInput, ReshapeOutput> reshapeTransform = (rowIn, rowOut) =>
               {
                   int oldSize = 3 * 256 * 256;
                   int newSize = 5 * 1 * 256 * 256;
                   float[] reshapedOutput = new float[newSize];
                   // Simply copy all the content of the input array into the output array, and leave the extra dimensions as 0.
                   // Notice that inside ML.NET all of these arrays are 1-D and the information about the shape is saved in the schema definition
                   for (int i = 0; i < oldSize; i++)
                       reshapedOutput[i] = rowIn.loadedImage[i];
   
                   rowOut.reshapedImage = reshapedOutput;
               };
   
               // Define and run pipeline
               var pipeline = mlContext.Transforms.LoadImages(outputColumnName: "loadedImage", imageFolder: null, inputColumnName: nameof(ModelInput.ImagePath))
                    .Append(mlContext.Transforms.ResizeImages("loadedImage", inputColumnName: "loadedImage", imageWidth: 256, imageHeight: 256))
                    .Append(mlContext.Transforms.ExtractPixels(outputColumnName: "loadedImage", outputAsFloatArray: true))
                    .Append(mlContext.Transforms.CustomMapping(reshapeTransform, null, outputSchemaDefinition: reshapeSchemaDefinition))
                    .Append(mlContext.Transforms.CopyColumns("input.1", nameof(ReshapeOutput.reshapedImage))) // The onnx model requires an input called "input.1"
                    .Append(mlContext.Transforms.ApplyOnnxModel(modelFile: modelpath))
                    .Append(mlContext.Transforms.CopyColumns(nameof(ModelOutput.onnxOutput), "496")); // The onnx model has an output called "496"
   
               var model = pipeline.Fit(data);
   
               PredictionEngine<ModelInput, ModelOutput> predictionEngine = mlContext.Model.CreatePredictionEngine<ModelInput, ModelOutput>(model);
               ModelInput image = mlContext.Data.CreateEnumerable<ModelInput>(data, reuseRowObject: true).First();
               ModelOutput prediction = predictionEngine.Predict(image);
   
               // Save output prediction as image
               SaveFloatArrayAsImage(prediction.onnxOutput, outputimage);
           }
   
           public static IEnumerable<ModelInput> LoadImagesFromDirectory(string folder)
           {
   
               List<ModelInput> retCollection = new List<ModelInput>();
               var files = Directory.GetFiles(folder, "*", searchOption: SearchOption.AllDirectories);
               foreach (var file in files)
               {
                   if ((Path.GetExtension(file) != ".jpg") && (Path.GetExtension(file) != ".png"))
                       continue;
   
                   yield return new ModelInput(){ImagePath = file};
               }
           }
   
           public static void SaveFloatArrayAsImage(float[] floatArray, string outputFilePath, int width = 256, int height = 256)
           {
               var redOffset = 0;
               var greenOffset = width * height;
               var blueOffset = 2 * width * height;
               Bitmap outImage = new Bitmap(width, height);
   
               for (int x = 0; x < width; x++)
               {
                   for (int y = 0; y < height; y++)
                   {
                       var i = y * width + x;
   
                       // I don't think this is the way the output of onnx should be post-processed
                       // But it gives an acceptable output
                       int r = (int) (floatArray[redOffset + i]*255);
                       int g = (int) (floatArray[greenOffset + i] * 255);
                       int b = (int) (floatArray[blueOffset + i] * 255);
   
                       // Clip values to have Black and White output
                       r = r > 122 ? 255 : 0;
                       g = g > 122 ? 255 : 0;
                       b = b > 122 ? 255 : 0;
   
                       Color c = Color.FromArgb(r, g, b);
                       outImage.SetPixel(x, y, c);
                   }
               }
   
               // Notice that the last 2*256*256 elements of the array
               // Corresponding to the 2 extra 'channels' of the onnx output
               // were ignored above.
               outImage.Save(outputFilePath);
           }
       }
   }
9. antoniovs1029

   closed this as completedon Apr 14, 2020
10. antoniovs1029

    added

    P1

    and removed

    need info

    on Apr 14, 2020
11. antoniovs1029

    added

    onnx

    on Aug 12, 2020
12. ghost locked as resolved and limited conversation to collaborators on Mar 19, 2022