Gradient Descent

Gradient descent is the core optimisation algorithm that trains neural networks. It iteratively adjusts model weights in the direction that reduces error, like finding the lowest point in a landscape by always walking downhill.

How gradient descent works:

1. Calculate loss (error) for current weights
2. Compute gradients (slope) via backpropagation
3. Update weights: new = old - learning_rate × gradient
4. Repeat until convergence

Variants of gradient descent:

• Batch: Use all training data per update (slow, stable)
• Stochastic (SGD): Use single example per update (noisy, fast)
• Mini-batch: Use small batches (balance of both)
• Momentum: Add velocity to escape local minima
• Adam: Adaptive learning rates per parameter

Key hyperparameters:

• Learning rate: Step size (critical to tune)
• Batch size: Examples per update
• Momentum: Weight of previous updates

Challenges:

• Local minima and saddle points
• Learning rate selection
• Training instability