Module 9: Neural networks · Lesson 9.5 · about 25 min
Training: loss, batches, epochs and learning rate
Training a network is gradient descent on small batches of rows, epoch after epoch. Change the learning rate and the batch size, watch the training and validation loss, and learn the defences against overfitting.
Module 9 · about 25 minutes · read first: 3.7, 9.4. You’ll be able to set up a training loop, choose a learning rate and batch size, read training and validation curves, and use the main defences against overfitting.
1. Why this matters
Backpropagation gives the gradient; the training loop decides how to use it. Most failed neural-network projects fail here: a learning rate that is too high, no validation set, or training long past the point where the network starts memorising. The same knobs appear in every deep-learning library and in every report of how a large language model was trained.
2. The training loop
- Shuffle the training rows.
- Cut them into mini-batches (say 32 rows).
- For each batch: forward pass, loss, backward pass (Lesson 9.4), and one gradient step on every weight.
- One pass over all batches is an epoch. After each epoch, compute the loss on validation data.
- Repeat for many epochs; keep the weights from the epoch with the best validation loss.
Using a batch instead of all rows is stochastic gradient descent (SGD, “stochastic” meaning “with randomness”). Each step’s gradient is a rough estimate from a few rows, like judging a whole pot of soup by one spoonful. But you get many steps per epoch, and the noise even helps escape poor spots.
Watch it happen: one epoch
Shuffle. 1,000 training rows are put in a new random order, so each batch is a fair mix.
Cut into batches. With batch size 50 there are 1,000 ÷ 50 = 20 batches.
First batch. Forward pass on 50 rows, average loss 0.69. Backward pass gives a gradient for every weight. Take one step: every weight moves a little downhill.
The rest of the epoch. Batches 2 to 20 do the same. 20 updates in all, each from a different spoonful of data. The training loss drifts down, with wobbles.
Check on unseen data. Validation loss after this epoch: 0.52, the best so far, so save these weights. If validation loss stops improving for, say, 10 epochs, stop and go back to the saved weights.
3. The knobs
- Learning rate. The most important setting: the size of each step. Too small and training crawls. Too big and it bounces or blows up. Try values a factor of 3 or 10 apart (0.3, 0.1, 0.03, …) and watch the first epochs. Many runs lower it as training goes (a schedule), like slowing down as you near a parking spot.
- Batch size. Small batches: noisy, many steps per epoch. Large batches: smooth, fast on a graphics card, fewer steps. 32 to 512 is common.
- Optimiser. Plain SGD adds . Momentum keeps a running average of recent steps, so the path rolls through small bumps like a ball. Adam also scales each weight’s step by how large its gradients have been. Adam with a rate near 0.001 is a common default.
- Starting weights. Small random numbers, scaled to the layer size, so signals neither die out nor explode through the layers.
- B (mini-batch)
- the handful of rows used for one step
- ∇L (gradient)
- the direction in which the loss rises fastest; we step the opposite way
- η (learning rate)
- how big each step is
- Epoch
- one full pass through all the training rows: ⌈n ÷ |B|⌉ steps
- v, β (momentum)
- a running average of recent steps, with β (about 0.9) saying how much of the past to keep
- Physical meaning
- Walking down a foggy valley by feeling the ground under a few stones at a time. Each guess of "downhill" is rough, but many quick guesses get you down faster than surveying the whole valley before every step. A good stride length (learning rate) gets you there; too long and you leap over the bottom.
4. Defences against overfitting
Networks have many weights and can memorise. The usual defences:
- Early stopping: watch validation loss, keep the best epoch.
- Weight decay: the ridge penalty of Lesson 3.8, applied to every weight.
- Dropout: during training, switch off a random half (or so) of a layer’s neurons at each step, so no neuron can rely on any other. Like a team that practises with random players missing, so everyone learns to play every role.
- More data, or data augmentation (flipped and cropped images, reworded sentences).
“The lower the training loss, the better the model.” Training loss keeps falling as long as the network can memorise more of the training rows, and a big network can memorise all of them. What matters is the loss on rows it has never trained on. When validation loss starts rising while training loss keeps falling, extra training is making the model worse at its real job.
Maths and principle
Principle: take many small, noisy downhill steps, and judge progress on data the steps never see.
Mini-batch step: with batch of rows and learning rate ,
Momentum with around 0.9 keeps a velocity :
Updates per epoch: for training rows.
Analogy: finding the bottom of a foggy valley by feeling the ground under a few stones at a time. Each guess is rough, but many quick guesses get you down faster than surveying the whole valley before every step.
Try it: 60,000 rows, batch 128, 10 epochs. How many weight updates? ( per epoch, 4,690 in all.)
5. Worked example
Made-up numbers. A network classifying product photos, validation loss after each epoch:
| epoch | 1 | 5 | 10 | 15 | 20 | 25 | 30 |
|---|---|---|---|---|---|---|---|
| training loss | 1.90 | 0.92 | 0.58 | 0.41 | 0.28 | 0.18 | 0.11 |
| validation loss | 1.85 | 0.97 | 0.69 | 0.62 | 0.64 | 0.71 | 0.80 |
Validation loss is lowest at epoch 15 and rises after, while training loss keeps falling: overfitting from about epoch 15. With early stopping (patience 10 epochs), training stops at epoch 25 and restores the epoch-15 weights. Adding dropout and retraining moves the best validation loss to 0.55 at epoch 22.
6. Practice
- In the simulation, set the learning rate to 10. What happens to the loss?
- Compare batch size 4 with 60 (all rows) at the same rate. Which needs fewer epochs, and why?
- Why must early stopping use validation loss, not training loss?
Answers
- It jumps and then shoots up (shown clipped at the top): the steps overshoot and diverge.
- Batch 4 usually: it makes 15 updates per epoch against 1, so it gets further per epoch, at the cost of noisier steps.
- Training loss keeps falling as the network memorises; only data it never trains on shows when it stops generalising.
Try it on your own data: the small model trainer shows training and test scores side by side; the same reading of the gap applies to networks.
New words: mini-batch, epoch, stochastic gradient descent, momentum, Adam, learning-rate schedule, weight decay, dropout, early stopping.
- Training loop: shuffle, cut into mini-batches, and for each batch do forward, loss, backward and one gradient step; one pass over all batches is an epoch.
- The learning rate is the most important knob: too small crawls, too big bounces or diverges.
- Momentum and Adam smooth and scale the steps; Adam near 0.001 is a common default.
- Judge training by validation loss; early stopping, weight decay, dropout and more data fight overfitting.
Check yourself
Sources
- Goodfellow, Bengio and Courville (2016), Deep Learning, MIT Press, Chapters 7 (regularisation) and 8 (optimisation) https://www.deeplearningbook.org/
- Kingma and Ba (2015), Adam: a method for stochastic optimization, International Conference on Learning Representations https://arxiv.org/abs/1412.6980
- Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov (2014), Dropout: a simple way to prevent neural networks from overfitting, Journal of Machine Learning Research 15, 1929-1958 https://jmlr.org/papers/v15/srivastava14a.html
Your ticks stay in this browser. Save them as a file on the course home.
Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.