Home / Learn / Data & AI / ML foundations / Lesson 9.2
All lessons

Module 9: Neural networks · Lesson 9.2 · about 20 min

Activation functions

A neuron's activation bends its weighted sum. Without it, a deep network collapses into one straight line; with the wrong one, learning stalls. Compare step, sigmoid, tanh and ReLU and their slopes.

Key points, meanings and formulas on one page: choose "Save as PDF" in the print box.

Module 9 · about 20 minutes · read first: 1.5, 9.1. You’ll be able to explain why networks need activations, compare the common ones and their slopes, and choose one for hidden and output layers.

Without a bend between layers, a deep network is just one straight line in disguise; the activation adds that bend, and its slope decides whether the learning signal survives the trip back.

1. Why this matters

The activation function is the small bend that makes a neural network more than a linear model. The switch from sigmoid to ReLU around 2010 was one of the simple changes that made deep learning work. Most modern networks, language models included, use ReLU or a smooth cousin of it (GELU, SiLU).

2. Why bend at all?

Without an activation, a layer computes 𝐖𝐱+𝐛: a weighted sum. Two such layers give 𝐖2(𝐖1𝐱+𝐛1)+𝐛2, which is again just one weight matrix times 𝐱 plus a bias. However many layers you stack, the result is one linear model with a straight boundary.

It’s like folding paper. Stretching or sliding a sheet keeps every straight line straight. Only a fold, a bend, lets you make new shapes. A non-linear activation between layers is that fold.

Watch it happen

Layer 1, no activation. It computes y = 2x + 1. A straight line.

Layer 2, no activation. It computes 3y − 4. Put the first in: 3(2x + 1) − 4 = 6x − 1. Still one straight line. Two layers did the work of one.

Add ReLU between them. ReLU(y) = max(0, y) keeps positives and turns negatives into 0. Now the output is 3 × max(0, 2x + 1) − 4.

A bend appears. For x below −0.5 the output is flat at −4; above, it climbs with slope 6. One kink. Many neurons, each with its own kink, can be added up to follow any curve.

g(z)
the activation: a function applied to a neuron's weighted sum z
σ(z) (sigmoid)
squashes any number into 0 to 1, like a probability
ReLU(z)
max(0, z): pass positives through, block negatives
g′(z) (slope)
how much the output changes when z changes a little; learning travels back through this
Physical meaning
A dimmer switch versus an on/off switch. With a dimmer you can feel which way to turn to get brighter (a useful slope). An on/off switch gives no hint until it flips (slope 0). And a dimmer that is already fully up (a saturated sigmoid) barely responds however you turn it.

3. The common ones

Try it: compare activation functions
output f(z)
slope f′(z)
slope after 10 layers (f′ⁿ)
Solid: the function; dashed: its slope. The step has slope 0 everywhere (except the jump), so gradient descent cannot learn with it. Sigmoid and tanh flatten for large inputs: their slope there is nearly 0, and multiplied over many layers the learning signal vanishes. ReLU keeps a slope of exactly 1 for every positive input, which is why it made deep networks trainable. The last box shows the slope raised to the tenth power, a rough picture of a signal passing back through 10 such layers.
name formula output range slope used for
step 1 if z ≥ 0, else 0 0 or 1 0 (no learning signal) the original perceptron
sigmoid 1 / (1 + e⁻ᶻ) 0 to 1 at most 0.25 output of yes/no models
tanh (eᶻ − e⁻ᶻ) / (eᶻ + e⁻ᶻ) −1 to 1 at most 1 small networks, older recurrent nets
ReLU max(0, z) 0 and up 0 or 1 hidden layers of most networks

Gradient descent (Lesson 3.3) needs slopes, and backpropagation (Lesson 9.4) multiplies by the activation’s slope once per layer. That explains the history:

  • The step has slope 0, so there is nothing to follow downhill.
  • Sigmoid and tanh saturate: for large inputs they flatten out and their slope is almost 0. The product over many layers vanishes, so deep networks with them learned painfully slowly.
  • ReLU passes the gradient through unchanged wherever it is active, so deep networks train. Its weakness: a neuron whose input is always negative outputs 0, gets slope 0, and can “die”. Variants such as leaky ReLU keep a small slope for negative inputs.

“More layers always means a more powerful network.” Only with an activation between them. Without one, a hundred layers compute exactly what one layer computes. And with a saturating activation like sigmoid, extra layers can make things worse: the learning signal fades before it reaches the early layers, so they barely train.

Maths and principle

Principle: a non-linear function between layers lets a network build curved boundaries; its slope carries the learning signal back.

Slopes:

σ′(z)=σ(z)(1−σ(z)),tanh′⁡(z)=1−tanh2⁡(z)
ReLU′⁡(z)={1z>00z<0

The vanishing product: the sigmoid’s slope is largest at z=0, where it is 0.25. Through 10 sigmoid layers the signal is multiplied by at most 0.2510≈10−6.

Analogy: a dimmer switch versus an on/off switch. You can feel which way to turn a dimmer to get brighter; an on/off switch gives no hint until it flips.

Try it: σ(0)=0.5. What is σ′(0), and σ′(4) if σ(4)≈0.982? (0.5 × 0.5 = 0.25; 0.982 × 0.018 ≈ 0.018.)

4. Worked example

One hidden neuron with weights (0.8, −0.5), bias 0.1, and input (2, 1): z=1.6−0.5+0.1=1.2.

activation output slope at 1.2
step 1 0
sigmoid 0.769 0.178
tanh 0.834 0.305
ReLU 1.2 1

If the input were (−2, 1), z=−2.0: ReLU outputs 0 with slope 0 (this example teaches its weights nothing), sigmoid outputs 0.119 with slope 0.105, tanh −0.964 with slope 0.071.

5. Practice

  1. In the simulation, at z = 4, what are the sigmoid’s and ReLU’s slopes, and their tenth powers?
  2. Why does stacking linear layers add nothing?
  3. What activation would you put on the output of a model predicting a house price?
Answers
  1. Sigmoid about 0.018, tenth power about 3 × 10⁻¹⁸; ReLU 1, tenth power 1.
  2. A linear function of a linear function is linear, so the stack equals one layer.
  3. None (the identity): a price can be any number. ReLU would also work if prices must be positive.

Try it on your own data: the neural network toys shows a perceptron’s hard threshold at work on points you place.

New words: activation function, non-linearity, sigmoid, tanh, ReLU, saturation, vanishing gradient, dead neuron.

  • Without an activation, any stack of layers collapses into a single linear model; the activation’s bend is what lets networks draw curves.
  • Sigmoid squashes to 0 to 1 (used for yes/no outputs); tanh to −1 to 1; ReLU = max(0, z) is the usual choice for hidden layers.
  • Backpropagation multiplies by the activation’s slope at every layer: sigmoid’s slope is at most 0.25, so the signal vanishes in deep stacks, while ReLU’s is 1 where active.
  • Output layers match the task: sigmoid for yes/no, softmax for several classes, none for a number.

Check yourself

1A network stacks 10 layers with no activation function (just weighted sums). It can draw
2Why did ReLU replace sigmoid in hidden layers?
3For the output of a yes/no classifier, the usual activation is:

Sources

Your ticks stay in this browser. Save them as a file on the course home.

Updated 2026-11-22. Numbers in the examples are made up to show the idea unless a source is given. Simulations run in your browser; nothing you type is sent anywhere.