How Training Actually Moves the Numbers (Gradient Descent)

Why this matters

Gradient descent is the engine of all modern machine learning. Every training loop — from linear regression to GPT — does: compute the gradient of the loss w.r.t. parameters, move parameters a small step opposite to the gradient. Understanding why this works (gradient points uphill; negative gradient points downhill), what can go wrong (saddle points, vanishing gradients, lr too high/low), and the variants (SGD, mini-batch, Adam) means you can diagnose training failures instead of blindly tuning hyperparameters.

Demo

A 2D bowl loss surface — L(w) = w₀² + 4w₁² — has an analytically known minimum at the origin, which means you can verify that gradient descent actually finds it and measure exactly how the learning rate changes the path. Setting the learning rate above 1/(largest eigenvalue) causes the steps to overshoot and diverge; dropping below that threshold produces stable but potentially slow convergence that illustrates why learning-rate choice is the most consequential hyperparameter decision you make before training.

Try it yourself

1. Run with lr=0.3 and watch the loss diverge. The maximum stable lr for this loss is 1/(largest eigenvalue) = 1/8 = 0.125. Verify by trying lr=0.124 vs lr=0.126.
2. Add momentum: v = 0.9*v + lr*grad(w); w -= v (init v = np.zeros(2)). Compare convergence speed at lr=0.01 with and without momentum.
3. Change the bowl to w[0]**2 + 100*w[1]**2. Run with lr=0.01. Notice it converges much slower along the steep dimension. This is the ill-conditioning problem that Adam's per-parameter scaling addresses.
4. Implement mini-batch GD on random 2D data: generate 1000 samples, sample 32 per step, compute gradient on the mini-batch. Does it converge to the same weights? Is it noisier?

References

• Ruder — An overview of gradient descent optimization algorithms
• Kingma & Ba — Adam: A Method for Stochastic Optimization