Loss Functions and Expected Loss

Definition of loss and objective functions

In mathematical optimization and decision theory, a loss (or cost/error) function is a real-valued function that assigns a “cost” to an event or to values of one or more variables. Optimization typically aims to minimize this loss. An objective function is closely related: in many settings it is either the loss itself or its opposite (e.g., reward/profit/utility/fitness), in which case the objective is maximized. In statistics, loss functions are commonly used for parameter estimation, where the “event” reflects the discrepancy between estimated and true values for observed data. The expected loss leads to decision criteria: in the frequentist framework, the expected loss over the sampling distribution is the risk function; in the Bayesian framework, the expectation is taken using a prior over parameters, producing the Bayes risk, which is minimized by the Bayes decision rule. Common examples include squared error loss (used in least squares) and 0–1 loss (used in classification). Selecting an appropriate loss function depends on the practical consequences of being wrong and on desired mathematical properties (e.g., continuity/differentiability), with real-world costs often being asymmetric or non-smooth.

0/2 modes complete

Optimization as loss minimization

0/2 modes complete