Gradient Descent Learning Path

Gradient descent update rule and intuition

Gradient descent updates parameters by repeatedly moving in the opposite direction of the gradient of a differentiable objective function. If the current point is a_n, the basic update rule is a_{n+1} = a_n - η ∇f(a_n), where η > 0 is the learning rate (step size). Intuitively, the negative gradient points in the direction of steepest decrease, so subtracting it makes the function value decrease (for a sufficiently small step size), ideally driving the iterates toward a local minimum. The step size can vary by iteration (η_n), and convergence can be guaranteed under assumptions such as convexity and Lipschitz continuity of the gradient. A geometric intuition is that the gradient is orthogonal to level (contour) curves: moving along the negative gradient crosses contours most rapidly toward lower values, like descending the bottom of a bowl-shaped landscape. Choosing η is crucial: too small makes progress slow, while too large can overshoot and diverge. More generally, one can use an update a_{n+1} = a_n - η_n p_n with a direction p_n that still has a positive component along the negative gradient; then the guaranteed decrease depends on both the alignment (angle) between p_n and the descent direction and how quickly the gradient changes along the step. In practice, line-search methods (e.g., Wolfe conditions) or backtracking line search help select η efficiently without requiring expensive extra gradient evaluations.

0/2 modes complete

Negative gradient as steepest descent direction

0/2 modes complete