Backend API Overview — NVIDIA cuDNN Backend (original) (raw)

The NVIDIA CUDA Deep Neural Network (cuDNN) library offers a context-based API that allows for easy multithreading and (optional) interoperability with CUDA streams. This API Reference lists the data types and API functions per sub-library.

The cuDNN version 9 library is reorganized into several sub-libraries. This new library structure separates legacy functionality (an imperative API with a fixed set of operations and fusion patterns in cuDNN version 7 and older) from the graph API (a declarative API, introduced in cuDNN version 8), as well as from the engine implementation. The graph API is now grouped into a single library (libcudnn_graph.so).

We recommend users to make API calls through the graph API, but the older shim layer (libcudnn.so) and legacy libraries (libcudnn_cnn.so, libcudnn_ops.so, libcudnn_adv.so) can still be directly used. The engine libraries (libcudnn_engines_precompiled.so, libcudnn_heuristic.so, and libcudnn_engines_runtime_compiled.so) contain necessary internal functionality to support the aforementioned sub-libraries. This library split is done in a way to allow for more flexibility on the user side in loading libraries. The mechanism for more flexible configurability of libraries will be introduced in a later version of cuDNN.

The API is split into the following libraries, each having corresponding header files in the include directory:

cudnn_graph

• This entity contains the API functions related to cuDNN context creation and destruction, a list of valid cuDNN backend descriptor types, a list of valid attributes, a subset of valid attribute values, and a full description of each backend descriptor type and their attributes. In addition, it is responsible for loading the correct engine libraries.
• Engine and heuristics libraries (cudnn_engines_precompiled, cudnn_engines_runtime_compiled, and cudnn_heuristics) are loaded through dlopen by cudnn_graph, and provide implementations for the API in cudnn_graph. The main idea of separating these engines from each other is to, in a future version of cuDNN, provide a way for the user to only load the functionality they need, and thus provide an opportunity to save memory and disk space.

cudnn_ops

• This entity contains the legacy API functions related to tensor descriptor management, tensor utility API functions, and common machine learning algorithms such as batch normalization, softmax, dropout, and so on. This library depends on cudnn_graph, and indirectly on cudnn_engines_precompiled, as kernels for some APIs reside in the cudnn_engines_precompiled library.

cudnn_cnn

• This entity contains all legacy API functions related to convolutional neural networks. This library depends on cudnn_ops and cudnn_graph. The cudnn_engines_precompiled library must be present for cudnn_cnn to correctly load.

cudnn_adv

• This entity contains all other features and algorithms. This includes RNNs, CTC loss, and multihead attention. This library depends on cudnn_ops and cudnn_graph.

Added, Deprecated, and Removed APIs#

API Changes for cuDNN 9.8.0#

• Added Enums:
  • CUDNN_BACKEND_DEVICEPROP_DESCRIPTOR
  • CUDNN_ATTR_DEVICEPROP_DEVICE_ID
  • CUDNN_ATTR_DEVICEPROP_HANDLE
  • CUDNN_ATTR_DEVICEPROP_JSON_REPRESENTATION
  • CUDNN_ATTR_ENGINE_DEVICEPROP
  • CUDNN_ATTR_ENGINEHEUR_DEVICEPROP
  • CUDNN_ATTR_EXECUTION_PLAN_DEVICEPROP
• Deprecated Enums:
  • CUDNN_ATTR_OPERATIONGRAPH_HANDLE
  • CUDNN_ATTR_EXECUTION_PLAN_HANDLE

API Changes for cuDNN 9.5.0#

• Added APIs:
  • cudnnBackendPopulateCudaGraph()
  • cudnnBackendUpdateCudaGraph()
• Added Enums:
  • CUDNN_STATUS_BAD_PARAM_CUDA_GRAPH_MISMATCH
  • CUDNN_STATUS_NOT_SUPPORTED_CUDA_GRAPH_NATIVE_API
  • CUDNN_BEHAVIOR_NOTE_SUPPORTS_CUDA_GRAPH_NATIVE_API

API Changes for cuDNN 9.2.0#

• Added APIs:
  • CUDNN_BACKEND_OPERATION_BN_FINALIZE_STATISTICS_DESCRIPTOR
• Added Enums:
  • CUDNN_STATUS_SUBLIBRARY_LOADING_FAILED
  • CUDNN_ATTR_ENGINECFG_WORKSPACE_SIZE
  • CUDNN_ATTR_ENGINECFG_SHARED_MEMORY_USED
• Deprecated APIs:
  • cudnnBackendInitialize()

API Changes for cuDNN 9.1.0#

• Added Enums:
  • CUDNN_NUMERICAL_NOTE_STRICT_NAN_PROP
  • CUDNN_POINTWISE_ATAN2
  • CUDNN_POINTWISE_CBRT
  • CUDNN_POINTWISE_CLZ
  • CUDNN_POINTWISE_IS_FINITE
  • CUDNN_POINTWISE_POPC
  • CUDNN_POINTWISE_ROUND_NEAREST_AFZ
  • CUDNN_POINTWISE_ROUND_NEAREST_EVEN
  • CUDNN_POINTWISE_SHIFT_LEFT
  • CUDNN_POINTWISE_SHIFT_RIGHT_ARITHMETIC
  • CUDNN_POINTWISE_SHIFT_RIGHT_LOGICAL
  • CUDNN_POINTWISE_XOR
  • CUDNN_POINTWISE_AND
  • CUDNN_POINTWISE_OR
  • CUDNN_POINTWISE_NOT

API Changes for cuDNN 9.0.0#

• Added Environment Variables:
  • CUDNN_LOGLEVEL_DB
• Added Type:
  • cudnnCTCGradMode_t
• Added APIs:
  • cudnnGetCTCLossDescriptor_v9()
  • cudnnRNNGetClip_v9()
  • cudnnRNNSetClip_v9()
  • cudnnSetCTCLossDescriptor_v9()
  • cudnnGetLastErrorString()
• Added Enums:
  • CUDNN_STATUS_SUBLIBRARY_VERSION_MISMATCH
  • CUDNN_STATUS_SERIALIZATION_VERSION_MISMATCH
  • CUDNN_STATUS_BAD_PARAM_NULL_POINTER
  • CUDNN_STATUS_BAD_PARAM_MISALIGNED_POINTER
  • CUDNN_STATUS_BAD_PARAM_NOT_FINALIZED
  • CUDNN_STATUS_BAD_PARAM_OUT_OF_BOUND
  • CUDNN_STATUS_BAD_PARAM_SIZE_INSUFFICIENT
  • CUDNN_STATUS_BAD_PARAM_STREAM_MISMATCH
  • CUDNN_STATUS_BAD_PARAM_SHAPE_MISMATCH
  • CUDNN_STATUS_BAD_PARAM_DUPLICATED_ENTRIES
  • CUDNN_STATUS_BAD_PARAM_ATTRIBUTE_TYPE
  • CUDNN_STATUS_NOT_SUPPORTED_GRAPH_PATTERN
  • CUDNN_STATUS_NOT_SUPPORTED_SHAPE
  • CUDNN_STATUS_NOT_SUPPORTED_DATA_TYPE
  • CUDNN_STATUS_NOT_SUPPORTED_LAYOUT
  • CUDNN_STATUS_NOT_SUPPORTED_INCOMPATIBLE_CUDA_DRIVER
  • CUDNN_STATUS_NOT_SUPPORTED_INCOMPATIBLE_CUDART
  • CUDNN_STATUS_NOT_SUPPORTED_ARCH_MISMATCH
  • CUDNN_STATUS_NOT_SUPPORTED_RUNTIME_PREREQUISITE_MISSING
  • CUDNN_STATUS_NOT_SUPPORTED_SUBLIBRARY_UNAVAILABLE
  • CUDNN_STATUS_NOT_SUPPORTED_SHARED_MEMORY_INSUFFICIENT
  • CUDNN_STATUS_NOT_SUPPORTED_PADDING
  • CUDNN_STATUS_NOT_SUPPORTED_BAD_LAUNCH_PARAM
  • CUDNN_STATUS_INTERNAL_ERROR_COMPILATION_FAILED
  • CUDNN_STATUS_INTERNAL_ERROR_UNEXPECTED_VALUE
  • CUDNN_STATUS_INTERNAL_ERROR_HOST_ALLOCATION_FAILED
  • CUDNN_STATUS_INTERNAL_ERROR_DEVICE_ALLOCATION_FAILED
  • CUDNN_STATUS_INTERNAL_ERROR_BAD_LAUNCH_PARAM
  • CUDNN_STATUS_INTERNAL_ERROR_TEXTURE_CREATION_FAILED
  • CUDNN_STATUS_EXECUTION_FAILED_CUDA_DRIVER
  • CUDNN_STATUS_EXECUTION_FAILED_CUBLAS
  • CUDNN_STATUS_EXECUTION_FAILED_CUDART
  • CUDNN_STATUS_EXECUTION_FAILED_CURAND
• Deprecated Environment Variables:
  • CUDNN_LOGERR_DBG
  • CUDNN_LOGWARN_DBG
  • CUDNN_LOGINFO_DBG
• Deprecated APIs:
  • cudnnActivationBackward()
  • cudnnActivationForward()
  • cudnnAddTensor()
  • cudnnBatchNormalizationBackward()
  • cudnnBatchNormalizationBackwardEx()
  • cudnnBatchNormalizationForwardInference()
  • cudnnBatchNormalizationForwardTraining()
  • cudnnBatchNormalizationForwardTrainingEx()
  • cudnnConvolutionBackwardBias()
  • cudnnConvolutionBackwardData()
  • cudnnConvolutionBackwardFilter()
  • cudnnConvolutionBiasActivationForward()
  • cudnnConvolutionForward()
  • cudnnCreateAttnDescriptor()
  • cudnnCreateConvolutionDescriptor()
  • cudnnCreateFilterDescriptor()
  • cudnnCreateFusedOpsConstParamPack()
  • cudnnCreateFusedOpsPlan()
  • cudnnCreateFusedOpsVariantParamPack()
  • cudnnCreateSeqDataDescriptor()
  • cudnnCreateTensorTransformDescriptor()
  • cudnnDestroyAttnDescriptor()
  • cudnnDestroyConvolutionDescriptor()
  • cudnnDestroyFilterDescriptor()
  • cudnnDestroyFusedOpsConstParamPack()
  • cudnnDestroyFusedOpsPlan()
  • cudnnDestroyFusedOpsVariantParamPack()
  • cudnnDestroySeqDataDescriptor()
  • cudnnDestroyTensorTransformDescriptor()
  • cudnnFindConvolutionBackwardDataAlgorithm()
  • cudnnFindConvolutionBackwardDataAlgorithmEx()
  • cudnnFindConvolutionBackwardFilterAlgorithm()
  • cudnnFindConvolutionBackwardFilterAlgorithmEx()
  • cudnnFindConvolutionForwardAlgorithm()
  • cudnnFindConvolutionForwardAlgorithmEx()
  • cudnnGetAttnDescriptor()
  • cudnnGetBatchNormalizationForwardTrainingExWorkspaceSize()
  • cudnnGetBatchNormalizationTrainingExReserveSpaceSize()
  • cudnnGetConvolution2dDescriptor()
  • cudnnGetConvolution2dForwardOutputDim()
  • cudnnGetConvolutionBackwardDataAlgorithm_v7()
  • cudnnGetConvolutionBackwardDataAlgorithmMaxCount()
  • cudnnGetConvolutionBackwardDataWorkspaceSize()
  • cudnnGetConvolutionBackwardFilterAlgorithm_v7()
  • cudnnGetConvolutionBackwardFilterAlgorithmMaxCount()
  • cudnnGetConvolutionBackwardFilterWorkspaceSize()
  • cudnnGetConvolutionForwardAlgorithm_v7()
  • cudnnGetConvolutionForwardAlgorithmMaxCount()
  • cudnnGetConvolutionForwardWorkspaceSize()
  • cudnnGetConvolutionGroupCount()
  • cudnnGetConvolutionMathType()
  • cudnnGetConvolutionNdDescriptor()
  • cudnnGetConvolutionNdForwardOutputDim()
  • cudnnGetConvolutionReorderType()
  • cudnnGetCTCLossDescriptor()
  • cudnnGetCTCLossDescriptorEx()
  • cudnnGetCTCLossDescriptor_v8()
  • cudnnGetFilter4dDescriptor()
  • cudnnGetFilterNdDescriptor()
  • cudnnGetFilterSizeInBytes()
  • cudnnGetMultiHeadAttnBuffers()
  • cudnnGetMultiHeadAttnWeights()
  • cudnnGetSeqDataDescriptor()
  • cudnnGetTensorTransformDescriptor()
  • cudnnIm2Col()
  • cudnnInitTransformDest()
  • cudnnMakeFusedOpsPlan()
  • cudnnMultiHeadAttnBackwardData()
  • cudnnMultiHeadAttnBackwardWeights()
  • cudnnMultiHeadAttnForward()
  • cudnnNormalizationBackward()
  • cudnnNormalizationForwardInference()
  • cudnnNormalizationForwardTraining()
  • cudnnOpTensor()
  • cudnnPoolingBackward()
  • cudnnPoolingForward()
  • cudnnReduceTensor()
  • cudnnReorderFilterAndBias()
  • cudnnRNNGetClip_v8()
  • cudnnRNNSetClip_v8()
  • cudnnScaleTensor()
  • cudnnSetAttnDescriptor()
  • cudnnSetConvolution2dDescriptor()
  • cudnnSetConvolutionGroupCount()
  • cudnnSetConvolutionMathType()
  • cudnnSetConvolutionNdDescriptor()
  • cudnnSetConvolutionReorderType()
  • cudnnSetCTCLossDescriptor()
  • cudnnSetCTCLossDescriptorEx()
  • cudnnSetCTCLossDescriptor_v8()
  • cudnnSetFilter4dDescriptor()
  • cudnnSetFilterNdDescriptor()
  • cudnnSetFusedOpsConstParamPackAttribute()
  • cudnnSetFusedOpsVariantParamPackAttribute()
  • cudnnSetSeqDataDescriptor()
  • cudnnSetTensorTransformDescriptor()
  • cudnnTransformFilter()
  • cudnnTransformTensor()
  • cudnnTransformTensorEx
• Deprecated Enums:
  • CUDNN_ATTR_OPERATION_RESAMPLE_BWD_ALPHA
  • CUDNN_ATTR_OPERATION_RESAMPLE_BWD_BETA
  • CUDNN_ATTR_OPERATION_RESAMPLE_FWD_ALPHA
  • CUDNN_ATTR_OPERATION_RESAMPLE_FWD_BETA
  • CUDNN_DATA_INT8x32
  • CUDNN_DATA_INT8x4
  • CUDNN_DATA_UINT8x4
  • CUDNN_STATUS_ALLOC_FAILED
  • CUDNN_STATUS_ARCH_MISMATCH
  • CUDNN_STATUS_INVALID_VALUE
  • CUDNN_STATUS_MAPPING_ERROR
  • CUDNN_STATUS_RUNTIME_PREREQUISITE_MISSING
  • CUDNN_STATUS_VERSION_MISMATCH
  • cudnnBackendAttributeName_t:::CUDNN_ATTR_POINTWISE_NAN_PROPAGATION
  • cudnnBackendAttributeType_t::CUDNN_TYPE_NAN_PROPOGATION
  • CUDNN_TYPE_NAN_PROPOGATION
  • CUDNN_ATTR_POINTWISE_NAN_PROPAGATION
  • CUDNN_KNOB_TYPE_SPLIT_K
  • CUDNN_KNOB_TYPE_USE_TEX
  • CUDNN_KNOB_TYPE_KBLOCK
  • CUDNN_KNOB_TYPE_LDGA
  • CUDNN_KNOB_TYPE_LDGB
  • CUDNN_KNOB_TYPE_CHUNK_K
  • CUDNN_KNOB_TYPE_SPLIT_H
  • CUDNN_KNOB_TYPE_WINO_TILE
  • CUDNN_KNOB_TYPE_CTA_SPLIT_K_MODE
  • CUDNN_KNOB_TYPE_IDX_MODE
  • CUDNN_KNOB_TYPE_SLICED
  • CUDNN_KNOB_TYPE_SPLIT_RS
  • CUDNN_KNOB_TYPE_SINGLEBUFFER
  • CUDNN_KNOB_TYPE_LDGC
  • CUDNN_KNOB_TYPE_TILE_CGA
  • CUDNN_KNOB_TYPE_NUM_C_PER_BLOCK
• Deprecated Types:

  • cudnnActivationDescriptor_t
  • cudnnActivationMode_t
  • cudnnAttnDescriptor_t
  • cudnnBatchNormMode_t
  • cudnnBatchNormOps_t
  • cudnnConvolutionBwdDataAlgoPerf_t
  • cudnnConvolutionBwdFilterAlgoPerf_t
  • cudnnConvolutionDescriptor_t
  • cudnnConvolutionFwdAlgoPerf_t
  • cudnnFilterDescriptor_t
  • cudnnFusedOps_t
  • cudnnFusedOpsConstParamLabel_t
  • cudnnFusedOpsConstParamPack_t
  • cudnnFusedOpsPlan_t
  • cudnnFusedOpsPointerPlaceHolder_t
  • cudnnFusedOpsVariantParamLabel_t
  • cudnnFusedOpsVariantParamPack_t
  • cudnnIndicesType_t
  • cudnnNanPropagation_t
  • cudnnNormAlgo_t
  • cudnnNormMode_t
  • cudnnNormOps_t
  • cudnnOpTensorDescriptor_t
  • cudnnPoolingDescriptor_t
  • cudnnPoolingMode_t
  • cudnnReduceTensorDescriptor_t
  • cudnnReduceTensorIndices_t
  • cudnnReorderType_t
  • cudnnSeqDataDescriptor_t
  • cudnnTensorTransformDescriptor_t

• Removed APIs:
• Removed Types:

  • cudnnRNNPaddingMode_t
  • cudnnAttnQueryMap_t