What is the difference between gradient descent and stochastic gradient descent?

What is the difference between gradient descent and stochastic gradient descent?

Gradient descent uses the entire dataset to compute the gradient, while stochastic gradient descent updates the model using one data point at a time.

Why would I choose stochastic gradient descent over regular gradient descent?

Stochastic gradient descent can converge faster and is often better for large datasets, as it updates weights more frequently.

Are there any downsides to using stochastic gradient descent?

Yes, it can introduce more noise in the updates, which might lead to less stable convergence compared to batch gradient descent.

Can you explain how mini-batch gradient descent fits into this?

Mini-batch gradient descent is a compromise between the two; it uses a small batch of data points to compute the gradient, balancing speed and stability.