Activation Functions: Types and Properties

Activation function definition in neural networks

In artificial neural networks, an activation function is a node-level function that computes the output from a node’s inputs and their weights. Using nonlinear activation functions enables neural networks to represent complex relationships; in particular, with nonlinearity, even relatively shallow networks can approximate arbitrary functions (as formalized by the Universal Approximation Theorem). Activation functions also differ in mathematical and practical properties that affect training, such as whether they are nonlinear, their output range (finite vs. infinite), and whether they are continuously differentiable—properties that influence gradient-based optimization and issues like vanishing gradients. Mathematically, activation functions can be categorized by how they transform inputs. Commonly, they are grouped into ridge functions (depending on a linear combination of inputs), radial functions (often related to radial basis functions and depending on distance from a center), and fold functions (used for aggregation such as mean/min/max, and including softmax for multiclass outputs). Activation functions may be saturating or nonsaturating depending on whether their gradients approach zero as inputs grow large, which impacts learning dynamics. Examples include logistic/sigmoid, tanh, ReLU and its smooth variants (e.g., GELU), as well as radial basis functions like Gaussian RBFs, and softmax/maxout for multi-output settings.

0/2 modes complete

Nonlinearity and Universal Approximation Theorem

0/2 modes complete