Automatic Differentiation with torch.autograd — PyTorch Tutorials 2.7.0+cu126 documentation (original) (raw)

beginner/basics/autogradqs_tutorial

Run in Google Colab

Colab

Download Notebook

Notebook

View on GitHub

GitHub

Note

Click hereto download the full example code

Learn the Basics ||Quickstart ||Tensors ||Datasets & DataLoaders ||Transforms ||Build Model ||Autograd ||Optimization ||Save & Load Model

Created On: Feb 10, 2021 | Last Updated: Jan 16, 2024 | Last Verified: Nov 05, 2024

When training neural networks, the most frequently used algorithm isback propagation. In this algorithm, parameters (model weights) are adjusted according to the gradient of the loss function with respect to the given parameter.

To compute those gradients, PyTorch has a built-in differentiation engine called torch.autograd. It supports automatic computation of gradient for any computational graph.

Consider the simplest one-layer neural network, with input x, parameters w and b, and some loss function. It can be defined in PyTorch in the following manner:

Tensors, Functions and Computational graph

This code defines the following computational graph:

In this network, w and b are parameters, which we need to optimize. Thus, we need to be able to compute the gradients of loss function with respect to those variables. In order to do that, we set the requires_grad property of those tensors.

Note

You can set the value of requires_grad when creating a tensor, or later by using x.requires_grad_(True) method.

A function that we apply to tensors to construct computational graph is in fact an object of class Function. This object knows how to compute the function in the forward direction, and also how to compute its derivative during the backward propagation step. A reference to the backward propagation function is stored in grad_fn property of a tensor. You can find more information of Function in the documentation.

print(f"Gradient function for z = {z.grad_fn}") print(f"Gradient function for loss = {loss.grad_fn}")

Gradient function for z = <AddBackward0 object at 0x7f79cef0cf70> Gradient function for loss = <BinaryCrossEntropyWithLogitsBackward0 object at 0x7f79cef0e110>

Computing Gradients

To optimize weights of parameters in the neural network, we need to compute the derivatives of our loss function with respect to parameters, namely, we need \(\frac{\partial loss}{\partial w}\) and\(\frac{\partial loss}{\partial b}\) under some fixed values ofx and y. To compute those derivatives, we callloss.backward(), and then retrieve the values from w.grad andb.grad:

tensor([[0.3313, 0.0626, 0.2530], [0.3313, 0.0626, 0.2530], [0.3313, 0.0626, 0.2530], [0.3313, 0.0626, 0.2530], [0.3313, 0.0626, 0.2530]]) tensor([0.3313, 0.0626, 0.2530])

Note

Disabling Gradient Tracking

By default, all tensors with requires_grad=True are tracking their computational history and support gradient computation. However, there are some cases when we do not need to do that, for example, when we have trained the model and just want to apply it to some input data, i.e. we only want to do forward computations through the network. We can stop tracking computations by surrounding our computation code withtorch.no_grad() block:

Another way to achieve the same result is to use the detach() method on the tensor:

There are reasons you might want to disable gradient tracking:

More on Computational Graphs

Conceptually, autograd keeps a record of data (tensors) and all executed operations (along with the resulting new tensors) in a directed acyclic graph (DAG) consisting ofFunctionobjects. In this DAG, leaves are the input tensors, roots are the output tensors. By tracing this graph from roots to leaves, you can automatically compute the gradients using the chain rule.

In a forward pass, autograd does two things simultaneously:

The backward pass kicks off when .backward() is called on the DAG root. autograd then:

Note

DAGs are dynamic in PyTorchAn important thing to note is that the graph is recreated from scratch; after each.backward() call, autograd starts populating a new graph. This is exactly what allows you to use control flow statements in your model; you can change the shape, size and operations at every iteration if needed.

Optional Reading: Tensor Gradients and Jacobian Products

In many cases, we have a scalar loss function, and we need to compute the gradient with respect to some parameters. However, there are cases when the output function is an arbitrary tensor. In this case, PyTorch allows you to compute so-called Jacobian product, and not the actual gradient.

For a vector function \(\vec{y}=f(\vec{x})\), where\(\vec{x}=\langle x_1,\dots,x_n\rangle\) and\(\vec{y}=\langle y_1,\dots,y_m\rangle\), a gradient of\(\vec{y}\) with respect to \(\vec{x}\) is given by Jacobian matrix:

\[J=\left(\begin{array}{ccc} \frac{\partial y_{1}}{\partial x_{1}} & \cdots & \frac{\partial y_{1}}{\partial x_{n}}\\ \vdots & \ddots & \vdots\\ \frac{\partial y_{m}}{\partial x_{1}} & \cdots & \frac{\partial y_{m}}{\partial x_{n}} \end{array}\right)\]

Instead of computing the Jacobian matrix itself, PyTorch allows you to compute Jacobian Product \(v^T\cdot J\) for a given input vector\(v=(v_1 \dots v_m)\). This is achieved by calling backward with\(v\) as an argument. The size of \(v\) should be the same as the size of the original tensor, with respect to which we want to compute the product:

First call tensor([[4., 2., 2., 2., 2.], [2., 4., 2., 2., 2.], [2., 2., 4., 2., 2.], [2., 2., 2., 4., 2.]])

Second call tensor([[8., 4., 4., 4., 4.], [4., 8., 4., 4., 4.], [4., 4., 8., 4., 4.], [4., 4., 4., 8., 4.]])

Call after zeroing gradients tensor([[4., 2., 2., 2., 2.], [2., 4., 2., 2., 2.], [2., 2., 4., 2., 2.], [2., 2., 2., 4., 2.]])

Notice that when we call backward for the second time with the same argument, the value of the gradient is different. This happens because when doing backward propagation, PyTorch accumulates the gradients, i.e. the value of computed gradients is added to thegrad property of all leaf nodes of computational graph. If you want to compute the proper gradients, you need to zero out the gradproperty before. In real-life training an optimizer helps us to do this.

Note

Previously we were calling backward() function without parameters. This is essentially equivalent to callingbackward(torch.tensor(1.0)), which is a useful way to compute the gradients in case of a scalar-valued function, such as loss during neural network training.