Residual connection motif x plus f(x)

Summary

The residual connection motif in residual neural networks uses the architectural pattern x ↦ f(x) + x, where f(x) is a learned transformation (any neural module). The “+ x” part is implemented as a skip connection that performs an identity mapping from the block input to its output, allowing the network to learn a residual function relative to the input rather than learning the full mapping from scratch. In practice, this is packaged as a residual block, and deep networks are built by stacking many such blocks. This motif improves training and convergence for very deep models by making both forward signal flow and backward gradient flow more stable. In forward propagation, the identity skip ensures that signals from earlier (shallower) blocks can reach later (deeper) blocks through direct addition. In backward propagation, the residual formulation adds a direct gradient term to shallower layers, helping mitigate vanishing gradients. When input and output dimensions differ, a projection connection y = F(x) + P(x) is used, where P(x) is typically a learned linear projection to match dimensions.

Key Points

• 1

  Residual connections use the motif x ↦ f(x) + x, implemented via identity skip connections to form residual blocks.

• 2

  Stacking residual blocks enables stable training of very deep networks by improving forward signal propagation and adding direct gradient pathways in backpropagation.

• 3

  When dimensions change across a residual block, projection connections replace the identity skip with a learned linear projection P(x).