What is the difference between stochastic gradient descent and batch gradient descent?

What is the difference between stochastic gradient descent and batch gradient descent?

Stochastic Gradient Descent (SGD) updates weights using one training example at a time, while Batch Gradient Descent uses the entire dataset.

Which one is better for training machine learning models?

It depends on the situation. SGD is faster and can escape local minima, while Batch Gradient Descent is more stable but slower.

Are there any hybrid methods?

Yes, Mini-batch Gradient Descent combines both methods by using small batches of data, balancing speed and stability.

What are the advantages of using SGD?

SGD can handle large datasets efficiently, introduces randomness which helps in generalization, and often converges faster.

Can you give an example of when to use each?

Use SGD for online learning or when data is too large to fit in memory. Use Batch Gradient Descent for smaller datasets where stability is key.