neuronPCA - Principal component analysis of neuron activations - MATLAB (original) (raw)

Principal component analysis of neuron activations

Since R2022b

Description

The compressNetworkUsingProjection function uses principal component analysis (PCA) to identify the subspace of learnable parameters that result in the highest variance in neuron activations by analyzing the network activations using a data set representative of the training data. The PCA step can be computationally intensive. If you expect to compress the same network multiple times (for example, when exploring different levels of compression), then you can perform the PCA step up front using aneuronPCA object.

Depending on the network, projection configuration, and code generation libraries used (including library-free code generation), forward passes of a projected deep neural network can be faster when you deploy the network to embedded hardware.

If you prune or quantize your network, then use compression using projection after pruning and before quantization.

Creation

Syntax

Description

`npca` = neuronPCA([net](#mw%5F6c608f73-c6e1-4e47-a15e-3d5d93de31a9%5Fsep%5Fmw%5Fb6163c1c-0e7a-4d18-81a3-6ef3212bec80),[mbq](#mw%5F6c608f73-c6e1-4e47-a15e-3d5d93de31a9%5Fsep%5Fmw%5F17bf9c2a-ab93-4ae7-a174-34d6b0dcba2f)) computes the principal component analysis of the neuron activations innet using the data in the mini-batch queuembq. The software analyzes the input and output activations of the projectable layers in net. This feature requires the Deep Learning Toolbox™ Model Compression Library support package. This support package is a free add-on that you can download using the Add-On Explorer. Alternatively, see Deep Learning Toolbox Model Compression Library.

example

`npca` = neuronPCA([net](#mw%5F6c608f73-c6e1-4e47-a15e-3d5d93de31a9%5Fsep%5Fmw%5Fb6163c1c-0e7a-4d18-81a3-6ef3212bec80),[X1,...,XN](#mw%5F6c608f73-c6e1-4e47-a15e-3d5d93de31a9%5Fsep%5Fmw%5F654a70e9-ad70-4e71-8b8a-43a81c789f62)) computes the principal component analysis of the neuron activations using the data in thedlarray objects X1,...,XN, whereN is the number of network inputs.

`npca` = neuronPCA(___,[Name=Value](#namevaluepairarguments)) sets the LayerNames property or specifies additional options using one or more name-value arguments.

Input Arguments

expand all

Neural network, specified as an initialized dlnetwork object.

Mini-batch queue that outputs data for each input of the network, specified as a minibatchqueue object.

The PCA step typically works best when using the full training set. However, any dataset that is representative of the training data distribution suffices. The input data must contain two or more observations and sequences must contain two or more time steps.

Note

Do not pad sequence as this can negatively impact the analysis. Instead, truncate mini-batches of data to have the same length or use mini-batches of size 1.

Input data, specified as a formatted dlarray.

For more information about dlarray formats, see the fmt input argument of dlarray.

The PCA step typically works best when using the full training set. However, any dataset that is representative of the training data distribution suffices. The input data must contain two or more observations and sequences must contain two or more time steps.

Note

Do not pad sequence as this can negatively impact the analysis. Instead, truncate mini-batches of data to have the same length or use mini-batches of size 1.

Name-Value Arguments

expand all

Specify optional pairs of arguments asName1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: neuronPCA(mbq,SolverMode="direct") specifies to use a direct PCA solve.

Since R2023b

PCA solver mode, specified as one of these values:

• "auto" — Automatically select PCA solver mode using network heuristics based on layer input and output sizes.
• "direct" — For all eigenvectors and eigenvalues, perform a direct solve of the in-memory covariance matrix. Use this option when the layer inputs and outputs have small numbers of activations or when the activation covariance matrix fits in memory.
• "iterative" — Perform an iterative solve that determines the fewest eigenvectors such that the cumulative sum of the corresponding eigenvalues, in decreasing order, satisfy theMinExplainedVariance option. This option does not store the covariance matrix in memory. Use this option when layer inputs or outputs have large numbers of activations or when the activation covariance matrix does not fit in memory.

Since R2023b

Number of eigenvectors to compute per iteration, specified as a positive integer.

To specify this option, the SolverMode option must be"iterative".

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

Verbosity level, specified as one of these values:

• "summary" — Display a summary of the neuron PCA algorithm.
• "steps" — Display information about the steps of the neuron PCA algorithm.
• "iterations" — Display information about the iterations of the neuron PCA algorithm.
• "off" — Do not display information.

Properties

expand all

Names of layers to analyze, specified as a string array, cell array of character vectors, or a character vector containing a single layer name.

The software, by default, analyzes all the layers in the network that support projection.

The compressNetworkUsingProjection function supports projecting these layers:

• convolution1dLayer (since R2024b)
• convolution2dLayer
• fullyConnectedLayer
• lstmLayer
• gruLayer

To set this property, use the corresponding name-value argument when you create the neuronPCA object. After you create a neuronPCA object, this property is read-only.

Data Types: string | cell

Since R2023b

This property is read-only.

Range of explained variance that the principal components computed for the analyzed layers can attain, returned as a two-element row vector.

Data Types: double

Since R2023b

This property is read-only.

Range of reduction in learnables that the principal components computed for the analyzed layers can attain, returned as a two-element row vector.

If the minimum reduction in learnables is greater than zero, then the explained variance is the same for all learnables reduction levels below the minimum.

Data Types: double

This property is read-only.

Eigenvalues of activations input to projectable layers, specified as a cell array. Each element contains the input eigenvalues for the corresponding projectable layer inLayerNames.

Data Types: cell

This property is read-only.

Eigenvectors of activations input to projectable layers, specified as a cell array. Each element contains the input eigenvectors for the corresponding projectable layer inLayerNames.

Data Types: cell

This property is read-only.

Eigenvalues of activations output by projectable layers, specified as a cell array. Each element contains the output eigenvalues for the corresponding projectable layer inLayerNames.

Data Types: cell

This property is read-only.

Eigenvectors of activations output by projectable layers, specified as a cell array. Each element contains the output eigenvectors for the corresponding projectable layer inLayerNames.

Data Types: cell

Examples

collapse all

Load the pretrained network in dlnetJapaneseVowels and the training data in JapaneseVowelsTrainData.

load dlnetJapaneseVowels load JapaneseVowelsTrainData

Create a mini-batch queue containing the training data. To create a mini-batch queue from in-memory data, convert the sequences to an array datastore.

adsXTrain = arrayDatastore(XTrain,OutputType="same");

Create the minibatchqueue object.

• Specify a mini-batch size of 16.
• Preprocess the mini-batches using the preprocessMiniBatchPredictors function, listed in the Mini-Batch Predictors Preprocessing Function section of the example.
• Specify that the output data has format "CTB" (channel, time, batch).

miniBatchSize = 16;

mbqTrain = minibatchqueue(adsXTrain, ... MiniBatchSize=miniBatchSize, ... MiniBatchFcn=@preprocessMiniBatchPredictors, ... MiniBatchFormat="CTB");

Create the neuronPCA object.

npca = neuronPCA(net,mbqTrain);

Using solver mode "direct". neuronPCA analyzed 2 layers: "lstm","fc"

View the properties of the neuronPCA object.

npca = neuronPCA with properties:

              LayerNames: ["lstm"    "fc"]
  ExplainedVarianceRange: [0 1]
LearnablesReductionRange: [0 0.9690]
        InputEigenvalues: {[12×1 double]  [100×1 double]}
       InputEigenvectors: {[12×12 double]  [100×100 double]}
       OutputEigenvalues: {[100×1 double]  [9×1 double]}
      OutputEigenvectors: {[100×100 double]  [9×9 double]}

Mini-Batch Predictors Preprocessing Function

The preprocessMiniBatchPredictors function preprocesses a mini-batch of predictors by extracting the sequence data from the input cell array and truncating them along the second dimension so that they have the same length.

Note: Do not pad sequence data when doing the PCA step for projection as this can negatively impact the analysis. Instead, truncate mini-batches of data to have the same length or use mini-batches of size 1.

function X = preprocessMiniBatchPredictors(dataX)

X = padsequences(dataX,2,Length="shortest");

end

References

Extended Capabilities

expand all

The neuronPCA function supports GPU array input with these usage notes and limitations:

• This function runs on the GPU if any of these conditions are met:
  • Any of the values of the network learnable parameters insidenet.Learnables.Value are dlarray objects with underlying data of type gpuArray
  • The input argument mbq outputs dlarray objects with underlying data of type gpuArray
  • The input arguments X1,...,XN are dlarray objects with underlying data of type gpuArray

For more information, see Run MATLAB Functions on a GPU (Parallel Computing Toolbox).

Version History

Introduced in R2022b

expand all

Specify the PCA solver mode using the SolverMode option. To specify the number of components per iteration of an iterative PCA solve, use theNumComponentsPerIteration option.

Starting in R2023b, the software, by default, automatically selects the PCA solver mode using neural network heuristics based on layer input and output sizes. In previous versions, the software performs a direct PCA solve for all neural networks. This can lead to differences in the numbers of eigenvalues and eigenvectors in theInputEigenvalues, InputEigenvectors,OutputEigenvalues, and OutputEigenvectors properties. To reproduce the behavior of previous versions, set theSolverMode option to "direct".