Definition of loss and objective functions

Summary

In mathematical optimization and decision theory, a loss (or cost/error) function is a real-valued function that assigns a “cost” to an event or to values of one or more variables. Optimization typically aims to minimize this loss. An objective function is closely related: in many settings it is either the loss itself or its opposite (e.g., reward/profit/utility/fitness), in which case the objective is maximized. In statistics, loss functions are commonly used for parameter estimation, where the “event” reflects the discrepancy between estimated and true values for observed data. The expected loss leads to decision criteria: in the frequentist framework, the expected loss over the sampling distribution is the risk function; in the Bayesian framework, the expectation is taken using a prior over parameters, producing the Bayes risk, which is minimized by the Bayes decision rule. Common examples include squared error loss (used in least squares) and 0–1 loss (used in classification). Selecting an appropriate loss function depends on the practical consequences of being wrong and on desired mathematical properties (e.g., continuity/differentiability), with real-world costs often being asymmetric or non-smooth.

Key Points

• 1

  A loss function maps outcomes (or variable values) to a real-valued cost; optimization usually minimizes it, while objective functions may be maximized if they are the negative of loss (reward/utility).

• 2

  Frequentist expected loss yields a risk function (expectation over data given parameters), while Bayesian expected loss yields Bayes risk (expectation over parameters using a prior), minimized by the Bayes decision rule.

• 3

  Different loss functions (e.g., squared error, absolute error, 0–1 loss) encode different notions of error and can change which estimators/decision rules are optimal; practical cost considerations and mathematical tractability both influence the choice.