groupnorm - Normalize data across grouped subsets of channels for each observation

  independently - MATLAB ([original](http://www.mathworks.com/access/helpdesk/help/deeplearning/ref/dlarray.groupnorm.html)) ([raw](?raw))

Normalize data across grouped subsets of channels for each observation independently

Since R2020b

Syntax

Description

The group normalization operation normalizes the input data across grouped subsets of channels for each observation independently. To speed up training of the convolutional neural network and reduce the sensitivity to network initialization, use group normalization between convolution and nonlinear operations such as relu.

After normalization, the operation shifts the input by a learnable offset β and scales it by a learnable scale factor γ.

The groupnorm function applies the group normalization operation todlarray data.Using dlarray objects makes working with high dimensional data easier by allowing you to label the dimensions. For example, you can label which dimensions correspond to spatial, time, channel, and batch dimensions using the"S", "T", "C", and"B" labels, respectively. For unspecified and other dimensions, use the"U" label. For dlarray object functions that operate over particular dimensions, you can specify the dimension labels by formatting thedlarray object directly, or by using the DataFormat option.

[Y](#mw%5F49c523a7-427f-4887-b60f-dfa71ead566f) = groupnorm([X](#mw%5F26485c97-f759-461d-af4d-ce567e83acca%5Fsep%5Fmw%5F04e2502d-c675-44b2-abd1-e144ae7f5377),[numGroups](#mw%5F044fdae5-6535-4973-85de-895811b73e1d),[offset](#mw%5F26485c97-f759-461d-af4d-ce567e83acca%5Fsep%5Fmw%5F4c6bcc13-8062-4f68-ba33-1283ee563946),[scaleFactor](#mw%5F26485c97-f759-461d-af4d-ce567e83acca%5Fsep%5Fmw%5Fdb77bbbd-d7a1-4d76-90d8-f43826970066)) applies the group normalization operation to the input data X using the specified number of groups and transforms it using the specified offset and scale factor.

The function normalizes over grouped subsets of the 'C' (channel) dimension and the 'S' (spatial), 'T' (time), and'U' (unspecified) dimensions of X for each observation in the 'B' (batch) dimension, independently.

For unformatted input data, use the 'DataFormat' option.

example

[Y](#mw%5F49c523a7-427f-4887-b60f-dfa71ead566f) = groupnorm([X](#mw%5F26485c97-f759-461d-af4d-ce567e83acca%5Fsep%5Fmw%5F04e2502d-c675-44b2-abd1-e144ae7f5377),[numGroups](#mw%5F044fdae5-6535-4973-85de-895811b73e1d),[offset](#mw%5F26485c97-f759-461d-af4d-ce567e83acca%5Fsep%5Fmw%5F4c6bcc13-8062-4f68-ba33-1283ee563946),[scaleFactor](#mw%5F26485c97-f759-461d-af4d-ce567e83acca%5Fsep%5Fmw%5Fdb77bbbd-d7a1-4d76-90d8-f43826970066),'DataFormat',FMT) applies the group normalization operation to the unformatted dlarray objectX with format specified by FMT. The outputY is an unformatted dlarray object with dimensions in the same order as X. For example,'DataFormat','SSCB' specifies data for 2-D image input with format'SSCB' (spatial, spatial, channel, batch).

[Y](#mw%5F49c523a7-427f-4887-b60f-dfa71ead566f) = groupnorm(___[Name,Value](#namevaluepairarguments)) specifies options using one or more name-value arguments in addition to the input arguments in previous syntaxes. For example, 'Epsilon',3e-5 sets the variance offset to 3e-5.

Examples

collapse all

Apply Group Normalization

Create a formatted dlarray object containing a batch of 128 28-by-28 images with 6 channels. Specify the format "SSCB" (spatial, spatial, channel, batch).

miniBatchSize = 128; inputSize = [28 28]; numChannels = 6; X = rand(inputSize(1),inputSize(2),numChannels,miniBatchSize); X = dlarray(X,"SSCB");

View the size and format of the input data.

Initialize the scale and offset for group normalization. For the scale, specify a vector of ones. For the offset, specify a vector of zeros.

scaleFactor = ones(numChannels,1); offset = zeros(numChannels,1);

Apply the group normalization operation with three groups using the groupnorm function.

numGroups = 3; Y = groupnorm(X,numGroups,offset,scaleFactor);

View the size and format of the output Y.

Input Arguments

collapse all

X — Input data

dlarray | numeric array

Input data, specified as a formatted dlarray, an unformatteddlarray, or a numeric array.

If X is an unformatted dlarray or a numeric array, then you must specify the format using the DataFormat option. If X is a numeric array, then eitherscaleFactor or offset must be adlarray object.

X must have a "C" (channel) dimension.

numGroups — Number of channel groups

positive integer | 'all-channels' | 'channel-wise'

Number of channel groups to normalize across, specified as a positive integer,'all-channels', or 'channel-wise'.

Data Types: single | double | char | string

offset — Offset

dlarray | numeric array

Offset β, specified as a formatted dlarray, an unformatted dlarray, or a numeric array with one nonsingleton dimension with size matching the size of the 'C' (channel) dimension of the input X.

If offset is a formatted dlarray object, then the nonsingleton dimension must have label 'C' (channel).

scaleFactor — Scale factor

dlarray | numeric array

Scale factor γ, specified as a formatted dlarray, an unformatted dlarray, or a numeric array with one nonsingleton dimension with size matching the size of the 'C' (channel) dimension of the input X.

If scaleFactor is a formatted dlarray object, then the nonsingleton dimension must have label 'C' (channel).

Name-Value Arguments

Specify optional pairs of arguments asName1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Before R2021a, use commas to separate each name and value, and enclose Name in quotes.

Example: 'Epsilon',3e-5 sets the variance offset to3e-5.

DataFormat — Description of data dimensions

character vector | string scalar

Description of the data dimensions, specified as a character vector or string scalar.

A data format is a string of characters, where each character describes the type of the corresponding data dimension.

The characters are:

• "S" — Spatial
• "C" — Channel
• "B" — Batch
• "T" — Time
• "U" — Unspecified

For example, consider an array containing a batch of sequences where the first, second, and third dimensions correspond to channels, observations, and time steps, respectively. You can specify that this array has the format "CBT" (channel, batch, time).

You can specify multiple dimensions labeled "S" or "U". You can use the labels "C", "B", and"T" once each, at most. The software ignores singleton trailing"U" dimensions after the second dimension.

If the input data is not a formatted dlarray object, then you must specify the DataFormat option.

For more information, see Deep Learning Data Formats.

Data Types: char | string

Epsilon — Constant to add to mini-batch variances

1e-5 (default) | positive scalar

Constant to add to the mini-batch variances, specified as a positive scalar.

The software adds this constant to the mini-batch variances before normalization to ensure numerical stability and avoid division by zero.

Before R2023a: Epsilon must be greater than or equal to 1e-5.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

Output Arguments

collapse all

Y — Normalized data

dlarray

Normalized data, returned as a dlarray. The outputY has the same underlying data type as the inputX.

If the input data X is a formatted dlarray,Y has the same dimension labels as X. If the input data is not a formatted dlarray, Y is an unformatted dlarray with the same dimension order as the input data.

Algorithms

collapse all

Group Normalization

The group normalization operation normalizes the elements_xi_ of the input by first calculating the mean_μG_ and variance_σG2_ over spatial, time, and grouped subsets of the channel dimensions for each observation independently. Then, it calculates the normalized activations as

where ϵ is a constant that improves numerical stability when the variance is very small. To allow for the possibility that inputs with zero mean and unit variance are not optimal for the operations that follow group normalization, the group normalization operation further shifts and scales the activations using the transformation

where the offset β and scale factor_γ_ are learnable parameters that are updated during network training.

Deep Learning Array Formats

Most deep learning networks and functions operate on different dimensions of the input data in different ways.

For example, an LSTM operation iterates over the time dimension of the input data, and a batch normalization operation normalizes over the batch dimension of the input data.

To provide input data with labeled dimensions or input data with additional layout information, you can use data formats.

A data format is a string of characters, where each character describes the type of the corresponding data dimension.

The characters are:

• "S" — Spatial
• "C" — Channel
• "B" — Batch
• "T" — Time
• "U" — Unspecified

For example, consider an array containing a batch of sequences where the first, second, and third dimensions correspond to channels, observations, and time steps, respectively. You can specify that this array has the format "CBT" (channel, batch, time).

To create formatted input data, create a dlarray object and specify the format using the second argument.

To provide additional layout information with unformatted data, specify the format using theDataFormat argument.

For more information, see Deep Learning Data Formats.

References

[1] Wu, Yuxin, and Kaiming He. “Group Normalization.” Preprint submitted June 11, 2018. https://arxiv.org/abs/1803.08494.

Extended Capabilities

GPU Arrays

Accelerate code by running on a graphics processing unit (GPU) using Parallel Computing Toolbox™.

The groupnorm function supports GPU array input with these usage notes and limitations:

• When at least one of the following input arguments is a gpuArray or a dlarray with underlying data of typegpuArray, this function runs on the GPU:
  • X
  • offset
  • scaleFactor

For more information, see Run MATLAB Functions on a GPU (Parallel Computing Toolbox).

Version History

Introduced in R2020b

expand all

R2023a: Epsilon supports values less than 1e-5

The Epsilon option also supports positive values less than 1e-5.