Surrogate Loss Functions Overview

Updated 10 February 2026

Surrogate loss functions are alternative loss definitions designed to approximate nonconvex, discontinuous target losses for efficient empirical risk minimization.
They ensure calibration through regret transfer bounds, linking the minimization of the surrogate loss to the control of the true loss and affecting sample complexity rates.
They are widely applied in binary, multiclass, and structured prediction tasks, with recent advances including learned surrogate losses to optimize complex metrics.

A surrogate loss function is an alternative loss defined to approximate, smooth, or upper-bound a target loss that is nonconvex, discrete, non-differentiable, or otherwise intractable for direct empirical risk minimization. Surrogate losses are ubiquitous across supervised and structured learning, algorithmic fairness, active/online learning, and learning-to-defer, serving as the foundation for gradient-based or convex/monotone optimization in binary, multiclass, multilabel, and structured prediction scenarios. Their principled design, calibration theory, and computational tractability have become central topics across modern statistical machine learning.

1. Motivation for Surrogate Losses

The canonical goal in supervised learning is to minimize a target loss $\ell^0$ (for example, the 0-1 loss for classification or a domain-specific metric such as F1, Jaccard, mIoU). However, these loss functions are often nonconvex, discontinuous, or operationally defined only on the prediction of discrete outputs. Minimizing the risk $\mathbb{E}_{(x,y)\sim D}[\ell^0(y, \hat{y}(x))]$ directly is NP-hard in most realistic settings (e.g., empirical 0-1 minimization for general linear classifiers) (Ben-David et al., 2012).

Surrogate losses $\ell$ are constructed to be convex (where possible), continuous, and differentiable in the model's output or score, while maintaining a tight formal connection to the original loss. Surrogates enable tractable empirical risk minimization via stochastic gradient methods, convex optimization, or efficient subgradient/cutting-plane/approximate-inference solvers (Ben-David et al., 2012, Choi, 2018). In structured domains, where output spaces are exponentially large, surrogate losses are also designed to permit efficient maximization or optimization over outputs.

2. Calibration, Regret, and Consistency

A crucial theoretical property is calibration: minimization of the surrogate loss should guarantee control over the true loss. This is formalized via calibration functions (or surrogate regret transfer bounds), which provide explicit upper bounds of the form:

$R_{0-1}(f) \leq \psi^{-1}(R_\ell(f) - R_\ell^*)$

for binary or multiclass loss, where $R_\ell$ denotes surrogate risk and $\psi$ is a nondecreasing calibration mapping (Ben-David et al., 2012, Pires et al., 2016, Hanneke et al., 2012).

In binary classification:

For any classification-calibrated convex surrogate $\ell$ , there is a calibration function $G_\ell$ such that

$R_{0-1}(w, b) \leq G_\ell(R_\gamma(w, b))$

where $R_\gamma$ is the $\gamma$ -margin error (Ben-David et al., 2012).

In multiclass (and more general) settings:

Calibration functions must be computed per-loss, but many classical surrogates including OVA, multiclass hinge (LLW), decoupled background-discrimination loss, and multiclass logistic regression admit explicit, tight $\delta$ -functions, often through reduction to binary calibration (Pires et al., 2016).

Importantly, the rate at which surrogate risk converges to optimum governs the rate at which excess risk in the target loss diminishes. For piecewise-linear convex surrogates (polyhedral), the regret transfer is linear, while for smooth, strongly convex losses, only a square-root relationship can be achieved (Frongillo et al., 2021). This dichotomy has a direct impact on sample complexity and statistical efficiency.

Surrogate Family	Risk Transfer	Sample Complexity Rate
Polyhedral (e.g., hinge)	Linear ( $\zeta(\varepsilon) = C\varepsilon$ )	$O(1/n)$
Non-polyhedral (smooth)	Square-root ( $\zeta(\varepsilon) = c\sqrt{\varepsilon}$ )	$O(1/\sqrt{n})$

3. Classical Convex Surrogates: Definitions and Optimality

In binary classification with linear predictors, canonical choices include:

Hinge loss: $\ell_{\mathrm{hinge}}(z) = \max\{0, 1-z\}$
Logistic loss: $\ell_{\mathrm{logistic}}(z) = \log(1 + e^{-z})$
Exponential loss: $\ell_{\mathrm{exp}}(z) = e^{-z}$
Squared hinge loss: $\ell_{\mathrm{sqhinge}}(z) = [\max\{0, 1-z\}]^2$

Ben-David et al. (Ben-David et al., 2012) prove that hinge loss achieves essentially optimal calibration bounds in terms of the margin error for linear predictors:

$R_{0-1}(w, b) \leq (B+1)R_{1/B}(w,b)$

where $B$ is the inverse margin. Lower bounds for other convex losses (e.g., exponential, logistic, squared-hinge) demonstrate that no convex loss can have strictly better dependence on $B$ and $v$ than hinge up to a constant. Notably, smooth/strongly-convex surrogates (logistic, exponential) entail worse qualitative scaling as $B$ increases.

4. Surrogates for Multiclass, Structured, and Custom Losses

Multiclass Classification

Multiclass surrogates generalize binary surrogates by defining per-class scores and loss functions. Unified calibration theory shows that multiclass hinge (LLW), one-vs-all (OVA), and coupled/logistic surrogates admit calibration functions directly inherited from their binary building blocks (Pires et al., 2016). The explicit expressions for calibration functions allow tight generalization guarantees and support a wide variety of loss functions.

Structured Prediction

Structured prediction requires surrogates capable of handling exponentially large output spaces, rich dependencies, and non-modular losses:

Margin-rescaling and slack-rescaling surrogates extend the hinge mechanism to structured outputs, employing maximizations or scaling in the surrogate to upper bound the task loss (Choi, 2018).
Bi-criteria surrogates capture a broad family via bivariate, quasi-concave combinations of structural loss and margin. Convex-hull or angular search over label space is used for efficient inference (Choi, 2018).
General non-modular loss surrogates use submodular-supermodular decomposition: a discrete loss is decomposed uniquely into submodular and supermodular components, each upper bounded by respective convex surrogates (Lovász hinge, slack-rescaling), then summed to form a globally convex, extension-loss that matches the true loss at binary points, preserves piecewise linearity, and is subgradient-computable in polynomial time (Yu et al., 2016).
Differentiable learned surrogates: Recent approaches parameterize the surrogate loss itself (e.g., via neural networks), learning differentiable surrogates for user-specified structured losses, with performance validated on graph and sequence prediction tasks (Yang et al., 2024).

Non-Differentiable Metric Surrogates

Automated methods search the space of parameterized and constrained differentiable approximations to replace non-differentiable metric components (e.g., argmax, logic operators) with smooth surrogates, enabling gradient-based training using direct metric optimization (e.g., mIoU, boundary F1) (Li et al., 2020).
For confusion-matrix-based metrics (F1, Jaccard, etc.), differentiable surrogates (e.g., sigmoidF1) are constructed by replacing indicator functions with smooth functions such as sigmoid, with analytic differentiation and batch-wise implementation (Bénédict et al., 2021).

5. Surrogates Beyond Standard Supervised Learning

Performance Metric Optimization

For performance metrics expressed as linear-fractional functions of FP/FN (e.g., F $_\beta$ , Jaccard), minimizing a strongly proper composite surrogate (e.g., logistic, squared, exponential) followed by threshold tuning yields upper bounds on target metric regret in terms of the surrogate regret. This holds for both binary and multilabel tasks (Kotłowski et al., 2015).

Differentiable surrogates aligned with $F_\beta$ via gradient path matching have been explicitly proposed for imbalanced data, yielding improved empirical metric convergence under severe class imbalance (Lee et al., 2021).

Fairness and Group-Objective Surrogates

Fairness-centered surrogates implement per-group or per-individual weighting schemes. For example, the $\alpha$ - $\beta$ FML family interpolates between ERM (average loss) and minimax fair objectives via $\beta$ -powered integrals of the per-example loss, with theoretically justified SGD optimization and smooth fairness-accuracy trade-off (Xu et al., 21 Mar 2025).

Learning to Defer and Abstention

In learning-to-defer and abstention tasks, surrogate loss families parameterized by non-increasing $\Psi$ (e.g., softmax-type, generalized cross-entropy, MAE) achieve realizable $H$ -consistency and Bayes-consistency, sometimes outperforming classical surrogates in theoretical calibration and empirical accuracy (Mao et al., 2024, Mao et al., 2023).

Causal Inference, Individualized Policies

Causal inference and dynamic treatment regime optimization require surrogates that upper-bound individualized treatment effect loss. Minimax surrogates (max over treated/control group risks), specifically hinge-based, yield convex SVM-type formulations and tight generalization error bounds (Goh et al., 2018). For multi-stage (DTR) settings, only surrogate families exhibiting suitable product-form non-concavity (e.g., $\psi(x,y) = \phi(x)\phi(y)$ with symmetric, bounded $\phi$ ) can guarantee Fisher consistency, outperforming concave surrogate approaches (Laha et al., 2021).

6. Prospective and Learned Surrogates

Recent advances include:

Learned surrogates via bilevel optimization: Neural network parameterized surrogates are trained to approximate non-differentiable or set-wise losses (e.g., F1, AUC, Jaccard) at the batch level. Training uses permutation-invariant architectures (DeepSets) and joint optimization of predictive and surrogate parameters, enabling the use of arbitrary true loss functions with gradient-based learners (Grabocka et al., 2019).
Rank correlation-based surrogate learning: Rather than exact value-matching, surrogates are learned to preserve the relative ordering of models by Spearman’s rank correlation, directly targeting improvement in the evaluation metric through rank preservation. This relaxes regression demands and improves stability and test performance across benchmarks in image and NLP tasks (Huang et al., 2022).

7. Limitations and Open Problems

Despite the breadth of calibration and tractable surrogate families, several open issues remain:

Tight bounds for smooth, non-polyhedral surrogates can be suboptimal in high-complexity settings. Polyhedral surrogates are theoretically optimal for regret transfer, but may underperform empirically in certain distributions with rare or ambiguous labels (Frongillo et al., 2021).
The design of surrogates for highly structured and application-specific losses (e.g., chemical graphs, text generation metrics) often requires careful parameterization or search, and may hinge on the quality of relaxations and contrastive learning strategies (Li et al., 2020, Yang et al., 2024).
There exist settings (notably in sequential decision problems such as DTR) where classical surrogate families (concave, hinge-type) fail Fisher consistency, requiring fundamentally novel surrogate forms (Laha et al., 2021).

References (arXiv IDs)

Surrogate loss optimality in linear classification: (Ben-David et al., 2012)
Regret bounds for polyhedral/non-polyhedral losses: (Frongillo et al., 2021)
Surrogate calibration in active and passive learning: (Hanneke et al., 2012)
Structured surrogate construction via submodular-supermodular decomposition: (Yu et al., 2016)
Surrogate regret for performance metrics (F1, Jaccard): (Kotłowski et al., 2015), smooth $F_\beta$ surrogate (Lee et al., 2021)
Learning smooth surrogate losses via neural nets (bilevel, ranking-based): (Grabocka et al., 2019, Huang et al., 2022)
Differentiable surrogate search for semantic segmentation metrics: (Li et al., 2020)
Fairness-oriented surrogate family ( $\alpha$ - $\beta$ FML): (Xu et al., 21 Mar 2025)
Score-based abstention surrogates and $H$ -consistency: (Mao et al., 2023), learning-to-defer: (Mao et al., 2024)
Minimax SVM surrogates for treatment effect/causal inference: (Goh et al., 2018)
Surrogate loss design in DTR / policy optimization: (Laha et al., 2021)
Multiclass calibration functions and general surrogate theory: (Pires et al., 2016)
Bi-criteria and efficiently inferable structured surrogates: (Choi, 2018)
Smooth F1 approximations in multilabel learning: (Bénédict et al., 2021)
Differentiable surrogate losses for structured prediction via contrastive learning: (Yang et al., 2024)

Table: Major Surrogate Losses and Their Properties

Surrogate	Target Scenario	Convexity	Calibration Tightness	Regret Transfer	Computational Notes
Hinge	Binary/class.	Convex	Optimal (linear)	Linear	Exp. active learning
Logistic	Binary/class.	Strongly convex	Slightly suboptimal	$\sqrt{\cdot}$	Smooth, stable
Square-Hinge	Binary/class.	Convex, smooth	Quadratic	$\sqrt{\cdot}$	Easy to optimize
Multiclass LLW/OVA	Multiclass	Convex	As binary	Linear	Efficient per-sample
Polyhedral	General discrete	Convex, pl	Optimal	Linear	Favors sample-complexity
Learned DeepSet	Metric/structured	Nonconvex	Data-dependent	Empirical (fast)	Batch, bilevel opt.
Bi-criteria	Structured prediction	Convex (qc)	Application-determined	Data-dependent	Convex-hull/angle search

pl = piecewise linear; qc = quasi-concave

By synthesizing calibration theory, computational tractability, robustness, and practical metric alignment, surrogate loss functions are an indispensable tool for modern machine learning. Their design—spanning from classical convex constructions to modern data-driven, task-specific surrogates—remains a central domain for both theoretical development and applied performance.