Loading your learn pack…
Read the summary, then start training!
Gradient descent uses the update a_{n+1} = a_n - η ∇f(a_n), moving against the gradient to reduce the objective.
Gradient descent updates parameters by repeatedly moving in the opposite direction of the gradient of a differentiable objective function. If the current point is a_n, the basic update rule is a_{n+1} = a_n - η ∇f(a_n), where η > 0 is the learning rate (step size). Intuitively, the negative gradient points in the direction of steepest decrease, so subtracting it makes the function value decrease (for a sufficiently small step size), ideally driving the iterates toward a local minimum. The step size can vary by iteration (η_n), and convergence can be guaranteed under assumptions such as convexity and Lipschitz continuity of the gradient. A geometric intuition is that the gradient is orthogonal to level (contour) curves: moving along the negative gradient crosses contours most rapidly toward lower values, like descending the bottom of a bowl-shaped landscape. Choosing η is crucial: too small makes progress slow, while too large can overshoot and diverge. More generally, one can use an update a_{n+1} = a_n - η_n p_n with a direction p_n that still has a positive component along the negative gradient; then the guaranteed decrease depends on both the alignment (angle) between p_n and the descent direction and how quickly the gradient changes along the step. In practice, line-search methods (e.g., Wolfe conditions) or backtracking line search help select η efficiently without requiring expensive extra gradient evaluations.
Gradient descent uses the update a_{n+1} = a_n - η ∇f(a_n), moving against the gradient to reduce the objective.
The negative gradient gives the steepest descent direction; the gradient is orthogonal to level sets, explaining the “contour-crossing” intuition.
Step size η must be chosen carefully: too small slows convergence, too large can cause overshoot/divergence; line search or backtracking helps select η.
More general updates a_{n+1} = a_n - η_n p_n work as long as p_n is sufficiently aligned with the negative gradient and the decrease can be bounded using gradient-change information.
An iterative optimization step of the form a_{n+1} = a_n - η ∇f(a_n) that moves parameters in the negative gradient direction scaled by a step size η.
The direction -∇f(a) that points toward the steepest local decrease of the objective function f at point a.
A positive scalar controlling how far gradient descent moves along the chosen descent direction each iteration.
A set of points where the objective function has the same value; the gradient is orthogonal to these contours.
A chosen direction in a generalized update a_{n+1} = a_n - η_n p_n that must have a positive inner product with the negative gradient to ensure decrease.
A procedure that selects an appropriate step size η (often satisfying conditions like Wolfe) by evaluating the objective along the proposed direction.
A practical line-search method that starts with a large step size and repeatedly shrinks it until sufficient decrease is achieved.
Criteria used in line search to ensure the chosen step size provides adequate decrease and curvature properties.
“Can you explain what "Gradient descent uses the update a_{n+1} = a_n - η ∇f(a_n), moving against the gradient to reduce the objective." means in simple terms?”