Activation function

What an activation function does

A neural network is made of nodes joined by weighted connections. Each node first combines its inputs with its weights, and an activation function then maps that input and those weights to the node's output ^[1]. The activation function decides how strong a signal a node passes on, and in a modern network almost every layer carries one.

Why the non-linearity matters

The choice of function is not cosmetic. Only non-linear activation functions let a two-layer network act as a universal function approximator; without them a multilayer network collapses to a single linear layer ^[1]. The reason is straightforward: stacking linear maps produces another linear map, so adding linear layers buys no expressive power over one layer. A non-linear activation placed after each weighted sum breaks that collapse and lets the network represent curved, complicated relationships between input and output. The phrase universal function approximator refers to this ability: given enough width in the middle layer, a two-layer network can approximate a very broad class of functions, and that property rests entirely on the non-linearity.

The common shapes

Stanford's CS231n notes describe the sigmoid activation as taking a real-valued input and squashing it to a range between 0 and 1 ^[2]. The same notes state that tanh is simply a scaled sigmoid, with tanh(x) = 2 * sigmoid(2x) - 1 ^[2], which centres the output around zero. CS231n describes the rectified linear unit as computing f(x) = max(0, x), so the activation is simply thresholded at zero ^[2]: positive values pass through unchanged and negative values are cut off.

A small example shows the difference. A node whose weighted sum comes to 2 gives roughly 0.88 through sigmoid, roughly 0.96 through tanh, and stays at 2 through ReLU. The same node with a weighted sum of -2 gives roughly 0.12 through sigmoid, about -0.96 through tanh, and 0 through ReLU.

Where the functions came from

The oldest choice, the binary step function, is almost everywhere non-differentiable, so gradient methods cannot make progress with it ^[1] — training by gradient descent needs a slope to follow. Sigmoid or logistic functions were used by Hinton and colleagues for speech recognition in 2012 ^[1]. Nair and Hinton proposed the rectified linear unit, ReLU, in 2010, and it was used in AlexNet in 2012 ^[1]. Later came the leaky ReLU from Maas, Hannun and Ng in 2013 ^[1]; the Gaussian error linear unit, GELU, from Hendrycks and Gimpel in 2016 ^[1]; and the SiLU or Swish function from Ramachandran and colleagues in 2017 ^[1].

Where they are used today

GELU became the dominant activation in modern transformers after BERT adopted it in 2018 ^[1]. ReLU's rise owed a great deal to speed: the CS231n notes report that ReLU was found to accelerate the convergence of stochastic gradient descent by a factor of 6 compared with sigmoid and tanh in the work of Krizhevsky et al. ^[2].

Limits

Saturating activations such as sigmoid and tanh cause gradients to approach zero — the vanishing gradient problem — while the non-saturating ReLU mitigates it ^[1]. Deep networks are trained by propagating error gradients backwards through the layers, and the derivative of the activation function acts as a multiplier on the gradient at every layer; when those derivatives are uniformly small, the product shrinks quickly with depth.

ReLU has a failure mode of its own: it has zero derivative for negative inputs, so a neuron can become permanently inactive, a failure known as the dying ReLU problem ^[1]. The binary step function fails differently, being almost everywhere non-differentiable ^[1]. Choosing an activation function is therefore a trade between smoothness, speed of convergence and the risk of dead units.