FullyConnectedLayer - Fully connected layer - MATLAB (original) (raw)

Description

A fully connected layer multiplies the input by a weight matrix and then adds a bias vector.

Creation

Syntax

Description

`layer` = fullyConnectedLayer([outputSize](#mw%5Fc9bf6c71-174f-47d8-9163-90478a23c56b)) returns a fully connected layer and specifies the OutputSize property.

`layer` = fullyConnectedLayer([outputSize](#mw%5Fc9bf6c71-174f-47d8-9163-90478a23c56b),[Name=Value](#namevaluepairarguments)) sets optional properties using one or more name-value arguments.

example

Input Arguments

expand all

outputSize — Output size

positive integer

Output size for the fully connected layer, specified as a positive integer.

Example: 10

Name-Value Arguments

Specify optional pairs of arguments asName1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Before R2021a, use commas to separate each name and value, and enclose Name in quotes.

Example: fullyConnectedLayer(10,Name="fc1") creates a fully connected layer with an output size of 10 and the name'fc1'.

WeightsInitializer — Function to initialize weights

"glorot" (default) | "he" | "orthogonal" | "narrow-normal" | "zeros" | "ones" | function handle

Function to initialize the weights, specified as one of the following:

• "glorot" – Initialize the weights with the Glorot initializer [1] (also known as Xavier initializer). The Glorot initializer independently samples from a uniform distribution with zero mean and variance2/(InputSize + OutputSize).
• "he" – Initialize the weights with the He initializer [2]. The He initializer samples from a normal distribution with zero mean and variance2/InputSize.
• "orthogonal" – Initialize the input weights with Q, the orthogonal matrix given by the QR decomposition of Z =Q R for a random matrix Z sampled from a unit normal distribution. [3]
• "narrow-normal" – Initialize the weights by independently sampling from a normal distribution with zero mean and standard deviation 0.01.
• "zeros" – Initialize the weights with zeros.
• "ones" – Initialize the weights with ones.
• Function handle – Initialize the weights with a custom function. If you specify a function handle, then the function must be of the form weights = func(sz), where sz is the size of the weights. For an example, see Specify Custom Weight Initialization Function.

The layer only initializes the weights when theWeights property is empty.

Data Types: char | string | function_handle

BiasInitializer — Function to initialize biases

"zeros" (default) | "narrow-normal" | "ones" | function handle

Function to initialize the biases, specified as one of these values:

• "zeros" — Initialize the biases with zeros.
• "ones" — Initialize the biases with ones.
• "narrow-normal" — Initialize the biases by independently sampling from a normal distribution with a mean of zero and a standard deviation of 0.01.
• Function handle — Initialize the biases with a custom function. If you specify a function handle, then the function must have the form bias = func(sz), where sz is the size of the biases.

The layer initializes the biases only when theBias property is empty.

Data Types: char | string | function_handle

Weights — Layer weights

[] (default) | matrix

Initial layer weights, specified as a matrix.

The layer weights are learnable parameters. You can specify the initial value of the weights directly using theWeights property of the layer. When you train a network, if the Weights property of the layer is nonempty, then the trainnet and trainNetwork functions use the Weights property as the initial value. If the Weights property is empty, then the software uses the initializer specified by the WeightsInitializer property of the layer.

At training time, Weights is anOutputSize-by-InputSize matrix.

Data Types: single | double

Bias — Layer biases

[] (default) | matrix

Initial layer biases, specified as a matrix.

The layer biases are learnable parameters. When you train a neural network, if Bias is nonempty, then the trainnet and trainNetwork functions use the Bias property as the initial value. If Bias is empty, then software uses the initializer specified by BiasInitializer.

At training time, Bias is anOutputSize-by-1 matrix.

Data Types: single | double

WeightLearnRateFactor — Learning rate factor for weights

1 (default) | nonnegative scalar

Learning rate factor for the weights, specified as a nonnegative scalar.

The software multiplies this factor by the global learning rate to determine the learning rate for the weights in this layer. For example, ifWeightLearnRateFactor is2, then the learning rate for the weights in this layer is twice the current global learning rate. The software determines the global learning rate based on the settings you specify using the trainingOptions function.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

BiasLearnRateFactor — Learning rate factor for biases

1 (default) | nonnegative scalar

Learning rate factor for the biases, specified as a nonnegative scalar.

The software multiplies this factor by the global learning rate to determine the learning rate for the biases in this layer. For example, if BiasLearnRateFactor is 2, then the learning rate for the biases in the layer is twice the current global learning rate. The software determines the global learning rate based on the settings you specify using the trainingOptions function.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

WeightL2Factor — L2 regularization factor for weights

1 (default) | nonnegative scalar

L2 regularization factor for the weights, specified as a nonnegative scalar.

The software multiplies this factor by the global_L2_ regularization factor to determine the_L2_ regularization for the weights in this layer. For example, ifWeightL2Factor is 2, then the L2 regularization for the weights in this layer is twice the global_L2_ regularization factor. You can specify the global_L2_ regularization factor using the trainingOptions function.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

BiasL2Factor — L2 regularization factor for biases

0 (default) | nonnegative scalar

L2 regularization factor for the biases, specified as a nonnegative scalar.

The software multiplies this factor by the global_L2_ regularization factor to determine the_L2_ regularization for the biases in this layer. For example, ifBiasL2Factor is 2, then the L2 regularization for the biases in this layer is twice the global_L2_ regularization factor. The software determines the global_L2_ regularization factor based on the settings you specify using the trainingOptions function.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

Name — Layer name

"" (default) | character vector | string scalar

Layer name, specified as a character vector or a string scalar. For Layer array input, the trainnet and dlnetwork functions automatically assign names to layers with the name "".

The FullyConnectedLayer object stores the Name property as a character vector.

Data Types: char | string

Properties

expand all

Fully Connected

OutputSize — Output size

positive integer

Output size for the fully connected layer, specified as a positive integer.

Example: 10

InputSize — Input size

'auto' (default) | positive integer

Input size for the fully connected layer, specified as a positive integer or 'auto'. If InputSize is 'auto', then the software automatically determines the input size during training.

Parameters and Initialization

WeightsInitializer — Function to initialize weights

"glorot" (default) | "he" | "orthogonal" | "narrow-normal" | "zeros" | "ones" | function handle

Function to initialize the weights, specified as one of the following:

• "glorot" – Initialize the weights with the Glorot initializer [1] (also known as Xavier initializer). The Glorot initializer independently samples from a uniform distribution with zero mean and variance 2/(InputSize + OutputSize).
• "he" – Initialize the weights with the He initializer [2]. The He initializer samples from a normal distribution with zero mean and variance2/InputSize.
• "orthogonal" – Initialize the input weights with Q, the orthogonal matrix given by the QR decomposition of Z =Q R for a random matrix Z sampled from a unit normal distribution. [3]
• "narrow-normal" – Initialize the weights by independently sampling from a normal distribution with zero mean and standard deviation 0.01.
• "zeros" – Initialize the weights with zeros.
• "ones" – Initialize the weights with ones.
• Function handle – Initialize the weights with a custom function. If you specify a function handle, then the function must be of the form weights = func(sz), where sz is the size of the weights. For an example, see Specify Custom Weight Initialization Function.

The layer only initializes the weights when theWeights property is empty.

Data Types: char | string | function_handle

BiasInitializer — Function to initialize biases

"zeros" (default) | "narrow-normal" | "ones" | function handle

Function to initialize the biases, specified as one of these values:

• "zeros" — Initialize the biases with zeros.
• "ones" — Initialize the biases with ones.
• "narrow-normal" — Initialize the biases by independently sampling from a normal distribution with a mean of zero and a standard deviation of 0.01.
• Function handle — Initialize the biases with a custom function. If you specify a function handle, then the function must have the formbias = func(sz), where sz is the size of the biases.

The layer initializes the biases only when the Bias property is empty.

The FullyConnectedLayer object stores this property as a character vector or a function handle.

Data Types: char | string | function_handle

Weights — Layer weights

[] (default) | matrix

Layer weights, specified as a matrix.

The layer weights are learnable parameters. You can specify the initial value of the weights directly using the Weights property of the layer. When you train a network, if the Weights property of the layer is nonempty, then the trainnet function uses the Weights property as the initial value. If the Weights property is empty, then the software uses the initializer specified by the WeightsInitializer property of the layer.

At training time, Weights is anOutputSize-by-InputSize matrix.

Data Types: single | double

Bias — Layer biases

[] (default) | matrix

Layer biases, specified as a matrix.

The layer biases are learnable parameters. When you train a neural network, if Bias is nonempty, then the trainnet function uses the Bias property as the initial value. IfBias is empty, then software uses the initializer specified by BiasInitializer.

At training time, Bias is anOutputSize-by-1 matrix.

Data Types: single | double

Learning Rate and Regularization

WeightLearnRateFactor — Learning rate factor for weights

1 (default) | nonnegative scalar

Learning rate factor for the weights, specified as a nonnegative scalar.

The software multiplies this factor by the global learning rate to determine the learning rate for the weights in this layer. For example, if WeightLearnRateFactor is 2, then the learning rate for the weights in this layer is twice the current global learning rate. The software determines the global learning rate based on the settings you specify using the trainingOptions function.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

BiasLearnRateFactor — Learning rate factor for biases

1 (default) | nonnegative scalar

Learning rate factor for the biases, specified as a nonnegative scalar.

The software multiplies this factor by the global learning rate to determine the learning rate for the biases in this layer. For example, if BiasLearnRateFactor is 2, then the learning rate for the biases in the layer is twice the current global learning rate. The software determines the global learning rate based on the settings you specify using the trainingOptions function.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

L2 regularization factor for the weights, specified as a nonnegative scalar.

The software multiplies this factor by the global L2 regularization factor to determine the L2 regularization for the weights in this layer. For example, if WeightL2Factor is 2, then the L2 regularization for the weights in this layer is twice the global L2 regularization factor. You can specify the global L2 regularization factor using the trainingOptions function.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

BiasL2Factor — L2 regularization factor for biases

0 (default) | nonnegative scalar

L2 regularization factor for the biases, specified as a nonnegative scalar.

The software multiplies this factor by the global L2 regularization factor to determine the L2 regularization for the biases in this layer. For example, if BiasL2Factor is 2, then the L2 regularization for the biases in this layer is twice the global L2 regularization factor. The software determines the global L2 regularization factor based on the settings you specify using the trainingOptions function.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

Layer

Layer name, specified as a character vector or string scalar. For Layer array input, the trainnet anddlnetwork functions automatically assign names to layers with the name "".

The FullyConnectedLayer object stores this property as a character vector.

Data Types: char | string

NumInputs — Number of inputs

1 (default)

This property is read-only.

Number of inputs to the layer, returned as 1. This layer accepts a single input only.

Data Types: double

InputNames — Input names

{'in'} (default)

This property is read-only.

Input names, returned as {'in'}. This layer accepts a single input only.

Data Types: cell

NumOutputs — Number of outputs

1 (default)

This property is read-only.

Number of outputs from the layer, returned as 1. This layer has a single output only.

Data Types: double

OutputNames — Output names

{'out'} (default)

This property is read-only.

Output names, returned as {'out'}. This layer has a single output only.

Data Types: cell

Examples

collapse all

Create Fully Connected Layer

Create a fully connected layer with an output size of 10 and the name fc1.

layer = fullyConnectedLayer(10,Name="fc1")

layer = FullyConnectedLayer with properties:

      Name: 'fc1'

Hyperparameters InputSize: 'auto' OutputSize: 10

Learnable Parameters Weights: [] Bias: []

Use properties method to see a list of all properties.

Include a fully connected layer in a Layer array.

layers = [ ... imageInputLayer([28 28 1]) convolution2dLayer(5,20) reluLayer maxPooling2dLayer(2,Stride=2) fullyConnectedLayer(10) softmaxLayer]

layers = 6x1 Layer array with layers:

 1   ''   Image Input       28x28x1 images with 'zerocenter' normalization
 2   ''   2-D Convolution   20 5x5 convolutions with stride [1  1] and padding [0  0  0  0]
 3   ''   ReLU              ReLU
 4   ''   2-D Max Pooling   2x2 max pooling with stride [2  2] and padding [0  0  0  0]
 5   ''   Fully Connected   10 fully connected layer
 6   ''   Softmax           softmax

Specify Initial Weights and Biases in Fully Connected Layer

To specify the weights and bias initializer functions, use the WeightsInitializer and BiasInitializer properties respectively. To specify the weights and biases directly, use the Weights and Bias properties respectively.

**Specify Initialization Function

Create a fully connected layer with an output size of 10 and specify the weights initializer to be the He initializer.

outputSize = 10; layer = fullyConnectedLayer(outputSize,'WeightsInitializer','he')

layer = FullyConnectedLayer with properties:

      Name: ''

Hyperparameters InputSize: 'auto' OutputSize: 10

Learnable Parameters Weights: [] Bias: []

Use properties method to see a list of all properties.

Note that the Weights and Bias properties are empty. At training time, the software initializes these properties using the specified initialization functions.

**Specify Custom Initialization Function

To specify your own initialization function for the weights and biases, set the WeightsInitializer and BiasInitializer properties to a function handle. For these properties, specify function handles that take the size of the weights and biases as input and output the initialized value.

Create a fully connected layer with output size 10 and specify initializers that sample the weights and biases from a Gaussian distribution with a standard deviation of 0.0001.

outputSize = 10; weightsInitializationFcn = @(sz) rand(sz) * 0.0001; biasInitializationFcn = @(sz) rand(sz) * 0.0001;

layer = fullyConnectedLayer(outputSize, ... 'WeightsInitializer',@(sz) rand(sz) * 0.0001, ... 'BiasInitializer',@(sz) rand(sz) * 0.0001)

layer = FullyConnectedLayer with properties:

      Name: ''

Hyperparameters InputSize: 'auto' OutputSize: 10

Learnable Parameters Weights: [] Bias: []

Use properties method to see a list of all properties.

Again, the Weights and Bias properties are empty. At training time, the software initializes these properties using the specified initialization functions.

**Specify Weights and Bias Directly

Create a fully connected layer with an output size of 10 and set the weights and bias to W and b in the MAT file FCWeights.mat respectively.

outputSize = 10; load FCWeights

layer = fullyConnectedLayer(outputSize, ... 'Weights',W, ... 'Bias',b)

layer = FullyConnectedLayer with properties:

      Name: ''

Hyperparameters InputSize: 720 OutputSize: 10

Learnable Parameters Weights: [10x720 double] Bias: [10x1 double]

Use properties method to see a list of all properties.

Here, the Weights and Bias properties contain the specified values. At training time, if these properties are non-empty, then the software uses the specified values as the initial weights and biases. In this case, the software does not use the initializer functions.

Algorithms

expand all

Fully Connected Layer

A fully connected layer multiplies the input by a weight matrix and then adds a bias vector.

As the name suggests, all neurons in a fully connected layer connect to all the neurons in the previous layer. This layer combines all of the features (local information) learned by the previous layers across the image to identify the larger patterns. For classification problems, the last fully connected layer combines the features to classify the images. This is the reason that the outputSize argument of the last fully connected layer of the network is equal to the number of classes of the data set. For regression problems, the output size must be equal to the number of response variables.

You can also adjust the learning rate and the regularization parameters for this layer using the related name-value pair arguments when creating the fully connected layer. If you choose not to adjust them, then the software uses the global training parameters defined by thetrainingOptions function.

If the input to the layer is a sequence (for example, in an LSTM network), then the fully connected layer acts independently on each time step. For example, if the layer before the fully connected layer outputs an array X of size _D_-by-_N_-by-S, then the fully connected layer outputs an array Z of size outputSize-by-_N_-by-S. At time step t, the corresponding entry of Z is WXt+b, where Xt denotes time step t of X.

Fully connected layers flatten the output. They encode the spatial data in the channel dimension and remove the spatial dimensions of the output.

Layer Input and Output Formats

Layers in a layer array or layer graph pass data to subsequent layers as formatted dlarray objects. The format of a dlarray object is a string of characters in which each character describes the corresponding dimension of the data. The formats consist of one or more of these characters:

• "S" — Spatial
• "C" — Channel
• "B" — Batch
• "T" — Time
• "U" — Unspecified

For example, you can describe 2-D image data that is represented as a 4-D array, where the first two dimensions correspond to the spatial dimensions of the images, the third dimension corresponds to the channels of the images, and the fourth dimension corresponds to the batch dimension, as having the format "SSCB" (spatial, spatial, channel, batch).

You can interact with these dlarray objects in automatic differentiation workflows, such as those for developing a custom layer, using a functionLayer object, or using the forward and predict functions withdlnetwork objects.

This table shows the supported input formats of FullyConnectedLayer objects and the corresponding output format. If the software passes the output of the layer to a custom layer that does not inherit from the nnet.layer.Formattable class, or aFunctionLayer object with the Formattable property set to 0 (false), then the layer receives an unformatted dlarray object with dimensions ordered according to the formats in this table. The formats listed here are only a subset. The layer may support additional formats such as formats with additional "S" (spatial) or"U" (unspecified) dimensions.

References

[1] Glorot, Xavier, and Yoshua Bengio. "Understanding the Difficulty of Training Deep Feedforward Neural Networks." In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, 249–356. Sardinia, Italy: AISTATS, 2010. https://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf

[2] He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification." In 2015 IEEE International Conference on Computer Vision (ICCV), 1026–34. Santiago, Chile: IEEE, 2015. https://doi.org/10.1109/ICCV.2015.123

[3] Saxe, Andrew M., James L. McClelland, and Surya Ganguli. "Exact Solutions to the Nonlinear Dynamics of Learning in Deep Linear Neural Networks.” Preprint, submitted February 19, 2014. https://arxiv.org/abs/1312.6120.

Extended Capabilities

C/C++ Code Generation

Generate C and C++ code using MATLAB® Coder™.

Usage notes and limitations:

Code generation does not support passingdlarray objects with unspecified (U) dimensions to this layer.

GPU Code Generation

Generate CUDA® code for NVIDIA® GPUs using GPU Coder™.

Refer to the usage notes and limitations in the C/C++ Code Generation section. The same limitations apply to GPU code generation.

Version History

Introduced in R2016a

expand all

R2024a: DAGNetwork and SeriesNetwork objects are not recommend

Starting in R2024a, DAGNetwork and SeriesNetwork objects are not recommended, use dlnetwork objects instead.

There are no plans to remove support for DAGNetwork andSeriesNetwork objects. However, dlnetwork objects have these advantages and are recommended instead:

• dlnetwork objects are a unified data type that supports network building, prediction, built-in training, visualization, compression, verification, and custom training loops.
• dlnetwork objects support a wider range of network architectures that you can create or import from external platforms.
• The trainnet function supports dlnetwork objects, which enables you to easily specify loss functions. You can select from built-in loss functions or specify a custom loss function.
• Training and prediction with dlnetwork objects is typically faster than LayerGraph and trainNetwork workflows.

To convert a trained DAGNetwork or SeriesNetwork object to a dlnetwork object, use the dag2dlnetwork function.

Fully connected layers behave slightly differently in dlnetwork objects when compared to DAGNetwork andSeriesNetwork objects. Fully connected layers flatten the output. They encode the spatial data in the channel dimension by reshaping the output data. Fully connected layers in SeriesNetwork andDAGNetwork objects output data with the same number of the spatial dimensions as the input by outputting data with spatial dimensions of size one. Fully connected layers in dlnetwork objects remove the spatial dimensions of the output.

R2019a: Default weights initialization is Glorot

Starting in R2019a, the software, by default, initializes the layer weights of this layer using the Glorot initializer. This behavior helps stabilize training and usually reduces the training time of deep networks.

In previous releases, the software, by default, initializes the layer weights by sampling from a normal distribution with zero mean and variance 0.01. To reproduce this behavior, set the'WeightsInitializer' option of the layer to'narrow-normal'.