What are the differences between SGD and Batch Gradient Descent?

What are the differences between SGD and Batch Gradient Descent?

SGD, or Stochastic Gradient Descent, updates the model weights using one training example at a time, while Batch Gradient Descent uses the entire dataset to compute the gradient before updating the weights. This makes SGD faster and more suitable for large datasets.

What are the advantages of using SGD over Batch Gradient Descent?

SGD can converge faster and is less likely to get stuck in local minima due to its noisy updates. It also allows for online learning, where the model can be updated continuously as new data comes in.

Are there any disadvantages to using SGD?

Yes, SGD can be more unstable due to its reliance on individual data points, which can lead to fluctuations in the loss function. It may require careful tuning of the learning rate.

When should I use Batch Gradient Descent instead?

Batch Gradient Descent is preferable when you have a smaller dataset, as it provides a more stable and accurate estimate of the gradient, leading to smoother convergence.