What are some common mistakes in gradient descent?

What are some common mistakes in gradient descent?

Some common mistakes include using a learning rate that's too high or too low, not normalizing data, and not checking for convergence.

How can I choose the right learning rate?

You can use techniques like learning rate schedules, grid search, or adaptive learning rates to find the optimal value.

What happens if the learning rate is too high?

If the learning rate is too high, the algorithm may overshoot the minimum, leading to divergence instead of convergence.

Can you explain the importance of data normalization?

Data normalization ensures that all features contribute equally to the distance calculations, improving convergence speed.

What are some signs that my gradient descent is not working?

Signs include oscillations in the loss function, failure to decrease the loss, or very slow convergence.