CUDNN_STATUS_EXECUTION_FAILED issue on Jetson AGX Orin with TensorFlow 2.15 GPU (original) (raw)

Hi,

I’m encountering a runtime GPU inference error when running a frozen TensorFlow model on my Jetson AGX Orin Developer Kit. The error occurs immediately upon the first inference run (sess.run ), with repeated cuDNN registration errors beforehand:

E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9373 Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607 Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1534 Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered

tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
  (0) UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc(6917): 'status'
 [[{{node graph/fe_shared/conv_down_1_0/Conv2D}}]]
 [[graph/reference_output_disparity/_209]]
  (1) UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc(6917): 'status'
 [[{{node graph/fe_shared/conv_down_1_0/Conv2D}}]]
0 successful operations.
0 derived errors ignored.
Original stack trace for 'graph/fe_shared/conv_down_1_0/Conv2D':

The error happens in the Conv2D layer within the imported TensorFlow graph.

Environment Details:
Jetson Platform - NVIDIA Jetson AGX Orin Developer Kit with JetPack 6.1
Python - 3.10.12 (inside a virtual environment)|
TensorFlow Version - 2.15.0+nv24.5 (GPU-enabled build)
TensorFlow Build Info - CUDA 12.2, cuDNN 8
CUDA - 12.2 (nvcc V12.2.140)
cuDNN - 8.9.7.29-1+cuda12.2 (confirmed via dpkg and header file)
GPU Driver - 540.4.0 (checked via nvidia-smi)
OpenCV 4.5.5 (Python pip package),
Other Packages - absl-py 2.3.1, pyyaml installed

No other GPU processes are running during tests (nvidia-smi shows zero GPU utilization

TensorFlow CUDA & GPU Check

python3 -c "import tensorflow as tf; print('CUDA:', tf.test.is_built_with_cuda()); print('GPU Available:', tf.config.list_physical_devices('GPU'))"

Output:

E ... Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E ... Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
E ... Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
CUDA: True
GPU Available: [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

I’ve also tried the below debugging steps:

• Verified tensor input shapes and data types match model expectations.
• Enabled GPU memory growth and limited memory fraction (per_process_gpu_memory_fraction=0.9).
• Ensured no other processes use the GPU.
• Verified CUDA and cuDNN version compatibility with TensorFlow build info.
• Lowered input frame resolution and processed every frame (no batching).
• Tested same frozen model on x86 desktop (works fine with CUDA 11.x and cuDNN 8.x).
• Reviewed model graph and node names, no issues found.
• Verified OpenCV and TensorFlow coexist without known conflicts (video capture starts before TF session).

Request for Assistance

• What could cause the persistent cuDNN factory registration warnings and the CUDNN_STATUS_EXECUTION_FAILED error on Jetson AGX Orin?
• Are there known compatibility issues with TensorFlow 2.15 (TF1 mode), CUDA 12.2, and cuDNN 8.9 on Jetson platforms?
• Should I consider rebuilding TensorFlow from source or downgrading CUDA/cuDNN to known stable versions?
• Any suggestions on further debugging or logs to collect?

If any forum members have hit similar errors or can share experience running custom TF frozen graphs with cuDNN on Jetson AGX Orin, your insights will be highly appreciated! Thanks in advance for your help!

Hi @naveenkumar.kamalakannan ,
I believe Jetson AGX Orin team may assist better here.

Thanks