TensorRT quits Python if TensorFlow is imported

[daniel-kukiela]

Description

TensorRT code silently quits Python if TensorFlow is imported.

Environment

TensorRT Version: 8.2.3.0
NVIDIA GPU: GTX 1080Ti / RTX 8000
NVIDIA Driver Version: 30.0.15.1123
CUDA Version: 11.4.3
CUDNN Version: 8.2.4.15
Operating System: Windows 10 21H2
Python Version (if applicable): 3.9.6
Tensorflow Version (if applicable): 2.7.0 (tested also with 2.5.0)
Baremetal or Container (if so, version): baremetal

Relevant Files

Used plan file: https://www.dropbox.com/s/mbsnwesfb0wmbp3/upsample_fp16.plan?dl=0

Steps To Reproduce

import tensorrt as trt
import pycuda.driver as cuda
import pycuda.autoinit
import tensorflow


file = 'upsample_fp16.plan'
f = open(file, 'rb')

TRT_LOGGER = trt.Logger(trt.Logger.WARNING)
trt_runtime = trt.Runtime(TRT_LOGGER)
engine = trt_runtime.deserialize_cuda_engine(f.read())
print(engine.get_binding_shape(0))
print(set(engine.get_binding_shape(0)))
print(trt.volume(engine.get_binding_shape(0)))

If you comment out import tensorflow the code works, otherwise silently quits Python.

[samurdhikaru]

@daniel-kukiela Thanks for reporting this issue. FYI, the hang occurs at print(set(engine.get_binding_shape(0))). It's a known compatibility issue between TF and TRT on Windows for TF 2.5+. A fix has been identified and will be available in a near release. In the meantime, here are a few things you could try:

• You could import Tensorflow before TensorRT (in your code TF is imported after TRT). This will not result in the hang.
• You could use a Tensorflow version below 2.5. Then the order of importation does not matter.
• If for some reason, the code itself is untouchable and TF cannot be downgraded, switching to a Linux environment will also solve the issue.

I will update this ticket when the fix is available in a future release.

[rajeevsrao]

Fixed in the 22.04 release.