Understanding Residual Networks (ResNet)

Residual connection motif x plus f(x)

The residual connection motif in residual networks is the architectural pattern that combines the input x with a learned transformation f(x) to form the output x + f(x). Formally, if f is an arbitrary neural network module, the block computes F(x) = f(x) + x (often written as x ↦ f(x) + x). This is implemented using a skip connection that performs an identity mapping from the block input to its output, allowing the network to learn a residual function relative to the input rather than learning the full mapping from scratch. This design improves training and convergence for very deep feedforward models by stabilizing both forward signal flow and backward gradient flow. In forward propagation, the identity skip ensures that signals from earlier (shallower) blocks can directly contribute to later (deeper) block inputs. In backward propagation, the gradient to a shallower layer includes an added term that directly passes through the identity path, which helps mitigate vanishing gradients and supports optimization. When dimensions do not match (e.g., f maps R^n to R^m with n ≠ m), a projection connection P(x) is used, yielding y = f(x) + P(x), where P is typically a learned linear projection.

0/2 modes complete

Residual learning and residual functions H(x)

0/2 modes complete