Check Custom Layer Validity - MATLAB & Simulink (original) (raw)

If you create a custom deep learning layer, then you can use the checkLayer function to check that the layer is valid. The function checks layers for validity, GPU compatibility, correctly defined gradients, and code generation compatibility. To check that a layer is valid, run the following command:

layer is an instance of the layer and layout is a networkDataLayout object specifying the valid sizes and data formats for inputs to the layer. To check with multiple observations, use the ObservationDimension option. To run the check for code generation compatibility, set the CheckCodegenCompatibility option to 1 (true). For large input sizes, the gradient checks take longer to run. To speed up the check, specify a smaller valid input size.

Check Custom Layer Validity

Check the validity of the example custom layer sreluLayer.

The custom layer sreluLayer, attached to this example as a supporting file, applies the SReLU operation to the input data. To access this layer, open this example as a live script.

Create an instance of the layer.

Create a networkDataLayout object that specifies the expected input size and format of a single observation of typical input to the layer. Specify a valid input size of [24 24 20 128], where the dimensions correspond to the height, width, number of channels, and number of observations of the previous layer output. Specify the data has format "SSCB" (spatial, spatial, channel, batch).

validInputSize = [24 24 20 128]; layout = networkDataLayout(validInputSize,"SSCB");

Check the layer validity using checkLayer. When you pass data through the network, the layer expects 4-D array inputs, where the first three dimensions correspond to the height, width, and number of channels of the previous layer output, and the fourth dimension corresponds to the observations.

Skipping GPU tests. No compatible GPU device found.

Skipping code generation compatibility tests. To check validity of the layer for code generation, specify the CheckCodegenCompatibility and ObservationDimension options.

Running nnet.checklayer.TestLayerWithoutBackward .......... .......... Done nnet.checklayer.TestLayerWithoutBackward

Test Summary: 20 Passed, 0 Failed, 0 Incomplete, 14 Skipped. Time elapsed: 0.11032 seconds.

The results show the number of passed, failed, and skipped tests. If you do not have a GPU, then the function skips the corresponding tests.

List of Tests

The checkLayer function uses these tests to check the validity of custom layers.

Some tests run multiple times. These tests also check different data types and for GPU compatibility:

• predictIsConsistentInType
• forwardIsConsistentInType
• backwardIsConsistentInType

To execute the layer functions on a GPU, the functions must support inputs and outputs of type gpuArray with the underlying data typesingle.

Generated Data

To check the layer validity, the checkLayer function generates data with values in the range [-1, 1].

To check for multiple observations, either specify alayout with a batch ("B") dimension or specify the observation dimension using the ObservationDimension option. If you specify the observation dimension, then the checkLayer function checks that the layer functions are valid using generated data with mini-batches of size 1 and 2. If you do not specify this name-value pair, then the function skips the tests that check that the layer functions are valid for multiple observations.

Diagnostics

If a test fails when you use checkLayer, then the function provides a test diagnostic and a framework diagnostic. The test diagnostic highlights any issues found with the layer. The framework diagnostic provides more detailed information.

Function Syntaxes

The test functionSyntaxesAreCorrect checks that the layer functions have correctly defined syntaxes.

For layers with multiple inputs or outputs, you must set the values of the layer properties NumInputs (or alternatively,InputNames) and NumOutputs (or alternatively, OutputNames) in the layer constructor function, respectively.

Multiple Observations

The checkLayer function checks that the layer functions are valid for single and multiple observations. To check for multiple observations, either specify alayout with a batch ("B") dimension or specify the observation dimension using the ObservationDimension option. If you specify the observation dimension, then the checkLayer function checks that the layer functions are valid using generated data with mini-batches of size 1 and 2. If you do not specify this name-value pair, then the function skips the tests that check that the layer functions are valid for multiple observations.

Functions Do Not Error

The tests predictDoesNotError,forwardDoesNotError, andbackwardDoesNotError check that the layer functions do not error when passed inputs of valid size. If you specify an observation dimension, then the function checks the layer for both a single observation and multiple observations.

Outputs Are Consistent in Size

The test backwardIsConsistentInSize checks that thebackward function outputs derivatives of the correct size.

The backward function syntax depends on the type of layer.

• dLdX = backward(layer,X,Y,dLdY,memory) returns the derivativesdLdX of the loss with respect to the layer input, wherelayer has a single input and a single output.Y corresponds to the forward function output anddLdY corresponds to the derivative of the loss with respect to Y. The function input memory corresponds to the memory output of the forward function.
• [dLdX,dLdW] = backward(layer,X,Y,dLdY,memory) also returns the derivative dLdW of the loss with respect to the learnable parameter, where layer has a single learnable parameter.
• [dLdX,dLdSin] = backward(layer,X,Y,dLdY,dLdSout,memory) also returns the derivative dLdSin of the loss with respect to the state input, where layer has a single state parameter anddLdSout corresponds to the derivative of the loss with respect to the layer state output.
• [dLdX,dLdW,dLdSin] = backward(layer,X,Y,dLdY,dLdSout,memory) also returns the derivative dLdW of the loss with respect to the learnable parameter and returns the derivative dLdSin of the loss with respect to the layer state input, where layer has a single state parameter and single learnable parameter.

You can adjust the syntaxes for layers with multiple inputs, multiple outputs, multiple learnable parameters, or multiple state parameters:

• For layers with multiple inputs, replace X anddLdX with X1,...,XN anddLdX1,...,dLdXN, respectively, where N is the number of inputs.
• For layers with multiple outputs, replace Y anddLdY with Y1,...,YM anddLdY1,...,dLdYM, respectively, where M is the number of outputs.
• For layers with multiple learnable parameters, replace dLdW with dLdW1,...,dLdWP, where P is the number of learnable parameters.
• For layers with multiple state parameters, replace dLdSin and dLdSout with dLdSin1,...,dLdSinK anddLdSout1,...,dLdSoutK, respectively, whereK is the number of state parameters.

To reduce memory usage by preventing unused variables being saved between the forward and backward pass, replace the corresponding input arguments with ~.

Tip

If the number of inputs to backward can vary, then usevarargin instead of the input arguments afterlayer. In this case, varargin is a cell array of the inputs, where the first N elements correspond to theN layer inputs, the next M elements correspond to the M layer outputs, the next M elements correspond to the derivatives of the loss with respect to the M layer outputs, the next K elements correspond to the K derivatives of the loss with respect to the K state outputs, and the last element corresponds to memory.

If the number of outputs can vary, then use varargout instead of the output arguments. In this case, varargout is a cell array of the outputs, where the first N elements correspond to theN the derivatives of the loss with respect to theN layer inputs, the next P elements correspond to the derivatives of the loss with respect to the P learnable parameters, and the next K elements correspond to the derivatives of the loss with respect to the K state inputs.

Tip

If the layer forward functions support dlarray objects, then the software automatically determines the backward function and you do not need to specify the backward function. For a list of functions that support dlarray objects, see List of Functions with dlarray Support.

Outputs Are Formatted

Since R2023b

The tests formattableLayerPredictIsFormatted andformattableLayerForwardIsFormatted check that the output of the layer functions are dlarray object with a channel dimension.

Initialization

Since R2023b

These tests check that initialization does not overwrite nonempty parameters.

The testsinitializeDoesNotChangeLearnableParametersWhenTheyAreNotEmpty and initializeDoesNotChangeStatefulParametersWhenTheyAreNotEmpty check that the custom initialize function does not overwrite nonempty learnable and stateful parameters.

Consistent Data Types and GPU Compatibility

The tests predictIsConsistentInType,forwardIsConsistentInType, andbackwardIsConsistentInType check that the layer functions output variables of the correct data type. The tests check that the layer functions return consistent data types when given inputs of the data typessingle, double, andgpuArray with the underlying types single or double.

If the layer forward functions fully support dlarray objects, then the layer is GPU compatible. Otherwise, to be GPU compatible, the layer functions must support inputs and return outputs of type gpuArray (Parallel Computing Toolbox).

Many MATLAB® built-in functions support gpuArray (Parallel Computing Toolbox) and dlarray input arguments. For a list of functions that support dlarray objects, see List of Functions with dlarray Support. For a list of functions that execute on a GPU, see Run MATLAB Functions on a GPU (Parallel Computing Toolbox). To use a GPU for deep learning, you must also have a supported GPU device. For information on supported devices, seeGPU Computing Requirements (Parallel Computing Toolbox). For more information on working with GPUs in MATLAB, see GPU Computing in MATLAB (Parallel Computing Toolbox).

Tip

If you preallocate arrays using functions such aszeros, then you must ensure that the data types of these arrays are consistent with the layer function inputs. To create an array of zeros of the same data type as another array, use the "like" option of zeros. For example, to initialize an array of zeros of size sz with the same data type as the array X, use Y = zeros(sz,"like",X).

Tip

If the layer forward functions support dlarray objects, then the software automatically determines the backward function and you do not need to specify the backward function. For a list of functions that support dlarray objects, see List of Functions with dlarray Support.

Correct Gradients

The test gradientsAreNumericallyCorrect checks that the gradients computed by the layer functions are numerically correct. The testbackwardPropagationDoesNotError checks that the derivatives can be computed using automatic differentiation.

When the optional backward function is not specified, the testbackwardPropagationDoesNotError checks that the derivatives can be computed using automatic differentiation. When the optionalbackward function is specified, the testgradientsAreNumericallyCorrect tests that the gradients computed in backward are numerically correct.

Tip

If the layer forward functions support dlarray objects, then the software automatically determines the backward function and you do not need to specify the backward function. For a list of functions that support dlarray objects, see List of Functions with dlarray Support.

Valid States

For layers with state properties, the testpredictReturnsValidStates checks that the predict function returns valid states. When forward is specified, the testforwardReturnsValidStates checks that the forward function returns valid states. The test resetStateDoesNotError checks that the resetState function returns a layer with valid state properties.

Code Generation Compatibility

If you set the CheckCodegenCompatibility option to1 (true), then thecheckLayer function checks the layer for code generation compatibility.

The test codegenPragmaDefinedInClassDef checks that the layer definition contains the code generation pragma %#codegen. The test layerPropertiesSupportCodegen checks that the layer properties support code generation. The testpredictSupportsCodegen checks that the outputs ofpredict are consistent in dimension and batch size.

In addition, when generating code that uses third-party libraries:

• Code generation supports custom layers with 2-D image or feature input only.
• The inputs and output of the layer forward functions must have the same batch size.
• Nonscalar properties must be a single, double, or character array.
• Scalar properties must have type numeric, logical, or string.

The checkLayer function does not check that functions used by the layer are compatible with code generation. To check that functions used by the custom layer also support code generation, first use the Code Generation Readiness app. For more information, see Check Code by Using the Code Generation Readiness Tool (MATLAB Coder).

See Also

checkLayer | analyzeNetwork

Related Topics

• Define Custom Deep Learning Layers
• Define Custom Deep Learning Layer with Learnable Parameters
• Define Custom Deep Learning Layer with Multiple Inputs
• Define Custom Deep Learning Layer with Formatted Inputs
• Define Custom Recurrent Deep Learning Layer
• Define Custom Deep Learning Layer for Code Generation
• Define Nested Deep Learning Layer Using Network Composition