What are the differences between Leaky ReLU and ReLU?

What are the differences between Leaky ReLU and ReLU?

ReLU (Rectified Linear Unit) outputs zero for negative inputs, while Leaky ReLU allows a small, non-zero gradient for negative inputs.

Why is Leaky ReLU preferred in some cases?

Leaky ReLU helps prevent the 'dying ReLU' problem, where neurons can become inactive and stop learning.

Are there any drawbacks to using Leaky ReLU?

Yes, Leaky ReLU can introduce noise in the model due to its non-zero gradient for negative inputs, which might affect convergence.

In what scenarios should I use ReLU over Leaky ReLU?

ReLU is often preferred in scenarios where the model can benefit from sparsity and when the risk of dying neurons is low.