Activation Functions: Types and Properties

Activation function role in neural networks

In neural networks, an activation function determines a node’s output from its weighted inputs. Its role is crucial for enabling nontrivial learning: when activations are nonlinear, even relatively shallow networks can approximate complex functions (captured by the Universal Approximation Theorem). Common activations such as sigmoid/logistic, tanh, ReLU, and GELU are widely used because they introduce nonlinearity and shape how information and gradients propagate during training. Activation functions also differ in mathematical properties that affect optimization. Nonlinear activations support universal approximation, while the identity activation makes a multi-layer network equivalent to a single layer. The range (finite vs. infinite) influences training stability and efficiency, and differentiability matters for gradient-based methods: differentiable activations enable effective backpropagation, whereas nondifferentiable or poorly behaved activations (e.g., binary step) can prevent learning. Saturating activations can lead to vanishing gradients, while nonsaturating ones like ReLU are often preferred. Mathematically, activations can be organized into categories such as ridge functions (depending on a linear combination of inputs), radial functions (often used in radial basis function networks), and folding functions (performing aggregation like pooling and used in multiclass outputs such as softmax). The topic also includes specialized variants like periodic and quadratic activations, and even quantum activation functions, where nonlinearity can be implemented in quantum neural network circuits without measuring each perceptron output at every layer.

0/2 modes complete

Nonlinearity and universal function approximation

0/2 modes complete