Repository navigation
Add TensorRT support #6212
Description
Is your feature request related to a problem? Please describe.
Currently the fastest way of executing models for Computer Vision inference is by running a TensorRT-optimised model. It is widely available in C/C++ but you cannot really use it in C#.
Describe the solution you'd like
I would like to be able to load the TensorRT engine into C# memory and call it from there using OpenCVSharp's Mat structures.
Describe alternatives you've considered
We are currently using Triton Inference Server but it adds overhead time for data serialisation and transmission.
Additional context
There are certain scenarios that would benefit greatly from calling a TensorRT model in-process such as Quality Control.
Activity
luisquintanilla commentedon Jun 10, 2022
Hi @turowicz
ML.NET offers the ability to export models to ONNX which from my understanding is one of the supported frameworks.
To export an ML.NET model to ONNX you use the
ConvertToOnnxtransform.Here's additional documentation on how to do it as well.
https://docs.microsoft.com/dotnet/machine-learning/how-to-guides/save-load-machine-learning-models-ml-net#save-an-onnx-model-locally
Does that satisfy your requirements?
turowicz commentedon Jun 13, 2022
I meant the other way round. Load up a TRT model in ML.NET and infer on data.
michaelgsharp commentedon Jun 13, 2022
What model format are you thinking? Still onxx? and it sounds a large part of the ask is having a way to not have to copy the data, is that correct?
turowicz commentedon Jun 13, 2022
Model format: TensorRT by NVIDIA
turowicz commentedon Jun 13, 2022
Load it in C#, run inference. This requires C# externs for TensorRT C runtime
[-]TensorRT support[/-][+]Add TensorRT support[/+]michaelgsharp commentedon Jul 11, 2022
@luisquintanilla I'll mark this as future for now, but we need to figure out if this aligns with our goals and if so when we would be able to take a look at this.
lorenzo-park commentedon Feb 10, 2025
Hi, by any chance, is it possible to make the priority of this feature higher? AFIK, the current available option to serve pytorch-trained models in C# is to convert them in ONNX format, howver, converting pytorch-trained models is getting harder in these days. The model authors seems getting less care about ONNX-compatibility these days. I think this is because the pytorch community is introducing the compilation features for Pytorch such as TorchDynamo and Torch-TensorRT. and it removes the necessity of the ONNX as an intermediate artifact to optimize their model for their accelerators. So I thought it is time to consider TensorRT runtime as one of the alternatives of onnxruntime?
(Writing during struggling to convert some open weight to ONNX format...)