Repository navigation
BatchNorm and InstanceNormalization tests fail with CUDA 10.1/CUDNN 7.5 #528
Description
The following unit tests fail with CUDA 10.1/CUDNN 7.5
BatchNormTest.PositiveTestCase
BatchNormTest.PositiveTestCaseDefaultEpsilon
BatchNormTest.BatchNorm1d_3d_Pytorch
BatchNormTest.BatchNorm2d_Pytorch
BatchNormTest.BatchNorm3d_Pytorch
BatchNormTest.BatchNorm2d_fp16
InstanceNormalizationOpTest.InstanceNorm
InstanceNormalizationOpTest.InstanceNorm_2
- Windows 10
- ONNX Runtime installed from source
- ONNX Runtime version: master
- Python version: 3.7
- VC++ 15.9
- CUDA/cuDNN version: 10.1, 7.5
- GPU model and memory: GeForce 940MX
Describe steps/code to reproduce the behavior: Build with CUDA enabled, and run unit tests.
Error messages:
BatchNorm:
2019-02-28 10:34:01.0270819 [E:onnxruntime:Default, cuda_call.cc:93 onnxruntime::CudaCall] CUDNN failure 3: CUDNN_STATUS_BAD_PARAM ; GPU=0 ; hostname=MININT-6NOLPDK ; expr=cudnnBatchNormalizationForwardInference( CudnnHandle(), cudnn_batch_norm_mode_, &alpha, &beta, data_desc, x_data, data_desc, y_data, bn_tensor_desc, f_scale.get(), f_B.get(), f_mean.get(), f_var.get(), epsilon_);
InstanceNormalization:
2019-02-28 10:34:02.0367632 [E:onnxruntime:Default, cuda_call.cc:93 onnxruntime::CudaCall] CUDNN failure 3: CUDNN_STATUS_BAD_PARAM ; GPU=0 ; hostname=MININT-6NOLPDK ; expr=cudnnBatchNormalizationForwardTraining( CudnnHandle(), CUDNN_BATCHNORM_SPATIAL, &one, &zero, data_desc, x_data, data_desc, y_data, stats_desc, unused_scale.get(), unused_bias.get(), 1.0f, mean.get(), variance.get(), 0.0, nullptr, nullptr);
2019-02-28 10:34:02.0370557 [E:onnxruntime:Default, provider_test_utils.cc:315
Activity
snnn commentedon Mar 7, 2019
Hi Scott,
I tested it on Linux, all tests passed.
$ ctest -C Debug
Test project /home/chasun/src/onnxruntime/cmake/cmake-build-debug
Start 1: onnxruntime_test_all
1/5 Test #1: onnxruntime_test_all ................................................ Passed 15.48 sec
Start 2: onnxruntime_test_framework_session_without_environment_standalone
2/5 Test #2: onnxruntime_test_framework_session_without_environment_standalone ... Passed 0.20 sec
Start 3: onnx_test_pytorch_converted
3/5 Test #3: onnx_test_pytorch_converted ......................................... Passed 0.22 sec
Start 4: onnx_test_pytorch_operator
4/5 Test #4: onnx_test_pytorch_operator .......................................... Passed 0.10 sec
Start 5: onnxruntime_shared_lib_test
5/5 Test #5: onnxruntime_shared_lib_test ......................................... Passed 18.23 sec
100% tests passed, 0 tests failed out of 5
Cuda version: 10.1.105-1
driver version: 418.39
skottmckay commentedon Mar 7, 2019
My fault - environment error.
Whilst it was building and linking against CUDNN 7.5 correctly and as expected, I had the location of CUDNN 7.1 in the PATH environment variable (forgot to update it when I added 7.5) so it was actually loading that version of cudnn64_7.dll. I guess the vast majority of APIs haven't changed so this worked most of the time, but not quite all (most likely CUDNN_BN_MIN_EPSILON differs between the versions).
Update model path doc (#528)