How can I create custom autograd functions in PyTorch?

How can I create custom autograd functions in PyTorch?

To create custom autograd functions in PyTorch, you need to subclass torch.autograd.Function and implement two static methods: forward and backward.

Can you give me an example of a custom function?

Sure! Here's a simple example of a custom ReLU function:

import torch

class MyReLU(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
        ctx.save_for_backward(input)
        return input.clamp(min=0)

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_tensors
        grad_input = grad_output.clone()
        grad_input[input < 0] = 0
        return grad_input

What do the ctx and saved_tensors do?

ctx is a context object that can be used to stash information for backward computation. saved_tensors allows you to save tensors from the forward pass to use in the backward pass.

How do I use my custom function in a model?

You can use your custom function just like any other PyTorch operation. For example:

output = MyReLU.apply(input)