Activation function definition in neural networks

Summary

In artificial neural networks, an activation function is a node-level function that computes the output from a node’s inputs and their weights. Using nonlinear activation functions enables neural networks to represent complex relationships; in particular, with nonlinearity, even relatively shallow networks can approximate arbitrary functions (as formalized by the Universal Approximation Theorem). Activation functions also differ in mathematical and practical properties that affect training, such as whether they are nonlinear, their output range (finite vs. infinite), and whether they are continuously differentiable—properties that influence gradient-based optimization and issues like vanishing gradients. Mathematically, activation functions can be categorized by how they transform inputs. Commonly, they are grouped into ridge functions (depending on a linear combination of inputs), radial functions (often related to radial basis functions and depending on distance from a center), and fold functions (used for aggregation such as mean/min/max, and including softmax for multiclass outputs). Activation functions may be saturating or nonsaturating depending on whether their gradients approach zero as inputs grow large, which impacts learning dynamics. Examples include logistic/sigmoid, tanh, ReLU and its smooth variants (e.g., GELU), as well as radial basis functions like Gaussian RBFs, and softmax/maxout for multi-output settings.

Key Points

• 1

  An activation function maps a node’s weighted inputs to its output; nonlinearity is essential for universal function approximation.

• 2

  Activation functions differ by properties such as nonlinearity, output range, differentiability, and whether they are saturating vs. nonsaturating, which affects gradient-based training.

• 3

  Activation functions can be organized into ridge, radial, and fold categories, with common examples like ReLU (ridge), Gaussian RBFs (radial), and softmax/maxout (fold/multivariate).