What are some examples of automatic differentiation in PyTorch?

What are some examples of automatic differentiation in PyTorch?

Automatic differentiation in PyTorch allows you to compute gradients automatically. For example, you can define a simple neural network and use the backward() method to compute gradients.

Can you show me a code snippet for that?

Sure! Here's a basic example:

import torch

# Define a tensor with requires_grad=True
x = torch.tensor([2.0], requires_grad=True)

# Define a simple function
y = x ** 2 + 3 * x + 1

# Compute gradients
y.backward()

# Print the gradient
print(x.grad)  # Output: tensor([7.])

What does the backward() method do?

The backward() method computes the gradient of the output tensor with respect to the input tensors that have requires_grad=True. It uses the chain rule to propagate gradients back through the computation graph.

Are there any limitations to automatic differentiation?

Yes, some limitations include memory consumption for large models and the need for the computation graph to be retained for gradient computation. Additionally, certain operations may not be differentiable.