Sigmoid Capability Boundaries in Neural Networks

Updated 21 February 2026

Sigmoid Capability Boundaries are defined limits arising from intrinsic sigmoidal activation properties, affecting function approximation, network expressivity, and model verification.
They illustrate how factors such as network depth, input domain structure, and bounded activations govern approximation accuracy and classification efficacy.
Analytical methods including linear relaxation, optimal tangent/secular bounds, and α-sig tuning demonstrate practical improvements in certification tightness and gradient flow.

Sigmoid capability boundaries define the exact mathematical and operational limits of what neural networks employing sigmoidal activation functions can express, approximate, or verify—either in function space, optimization, or formal verification regimes. These boundaries arise from the inherent properties of sigmoidal nonlinearities: their bounded images, smooth “S-shaped” curvature, and unique saturation/vanishing gradient profiles. The research landscape spans function approximation (on bounded/unbounded domains), network robustness, formal verification, and architectural design, with rigorous identification and characterization of points where sigmoid-based models fundamentally fail or achieve optimality.

1. Universal Approximation and Domain Criticality

The approximation capabilities of shallow neural networks with monotone sigmoidal activations admit a sharp dichotomy according to the input domain structure. In the setting of function spaces $L^p(\Omega)$ , the critical findings are as follows (Wang et al., 2019):

On domains of the form $\mathbb{R}\times[0,1]^n$ (one unbounded direction, $2\leq p<\infty$ ), a single-hidden-layer network with a monotone sigmoid, ReLU, ELU, Softplus, or LeakyReLU activation is a universal approximator: for any $f\in L^p(\mathbb{R}\times[0,1]^n)$ , for every $\varepsilon>0$ there exists a network $g$ such that $\|f-g\|_{L^p}<\varepsilon$ .
On the full plane $\mathbb{R}^2$ (two unbounded directions), any shallow (depth=2) network composed of bounded sigmoidals fails to approximate any nontrivial $L^p$ function: $\mathcal S_2(\varphi)\cap L^p(\mathbb{R}^2)=\{0\}$ for any reasonable $p$ .

This boundary is proven via Hahn–Banach separation and Fourier-analytic arguments (positive case), and, in the negative case, by showing linear (or constant, for sigmoidal) asymptotics of ridge units preclude $L^p$ integrability unless all components cancel identically.

Domain	Shallow Sigmoid Universal ?	Reference
$\mathbb{R}\times[0,1]^n$	Yes	(Wang et al., 2019)
$\mathbb{R}^2$ (or $\mathbb{R}^m$ , $m\geq2$ unbounded)	No	(Wang et al., 2019)

The phase transition is sharp: one unbounded direction grants universality, two or more destroy $L^p$ -expressivity, regardless of hidden size. Deepening the network (depth $\geq3$ ) restores universal approximation over all $L^p(\mathbb{R}^n)$ .

2. Expressivity, Approximation Rate, and Sharpness

The best possible (i.e., sharp) rates at which single-hidden-layer sigmoid networks approximate univariate targets are governed by moduli of smoothness and do not exceed $O(1/n^r)$ for activation smoothness $C^r$ , with possible log-factor slowdowns for analytic sigmoidals with unrestricted scaling (Goebbels, 2018):

For any $f$ $f$ with $r$ $r$ -th smoothness modulus $\omega_r(f,\cdot)$ $ω_{r} (f, \cdot)$ ,
- Jackson-type upper bounds: $E(\Phi_{n,\sigma};f)_p \leq C\,\omega_r(f,1/n)_p$ .
- Sharpness: There exist $f$ for which $E(\Phi_{n,\sigma};f)_p$ cannot be improved $o$ -wise, i.e., no generically faster decay.
- For logistic sigmoid (analytic), lower bounds include a log-factor: $E(\Phi_{n,\sigma_l};f)_p \neq o(1/[n\log n]^r)$ ; uniform scaling restores matching rates.

This points to strict limits on efficiency: width growth is necessary to improve approximation error.

3. Robustness Verification and Linear Relaxation Boundaries

In formal verification and robustness, sigmoid capability boundaries are formalized as the family of linear upper and lower bounds (tangent and secant relaxations) enclosing sigmoid activations neuron-wise (local) or network-wise (global) (Zhang et al., 2022, König et al., 2024, Chevalier et al., 2024). The core developments include:

Neuron-wise vs. network-wise tightness: Neuron-wise tightest bounds minimize the integral gap between linear and sigmoid over an interval but might not yield the best global network output bounds. Network-wise tightness seeks affine envelopes yielding the best (tightest) global output certifications (Zhang et al., 2022).
Parameter search and automation: Efficient algorithms (gradient ascent, SMAC configuration, or dual-space projected search) can globally tune tangent points to maximize certification bounds. For instance, SMAC-driven hyperparameter optimization achieved up to 184% tighter bounds in practical benchmarks (König et al., 2024).
$\alpha$ -sig method: By rotating affine bounds around a contact point parameterized by $\alpha$ , and tuning $\alpha$ per neuron in the dual optimization, the tightest convex relaxations for formal verification can be achieved, improving both certification rate and computational speed compared to static LiRPA/α-CROWN cuts (Chevalier et al., 2024).

Relaxation Approach	Tightness Regime	Empirical Improvement	Reference
Single-layer, convex search	Network-wise optimal	Up to 160%	(Zhang et al., 2022)
SMAC configuration	Network-wide (global)	$\sim$ 25%	(König et al., 2024)
$\alpha$ -sig dual tuning	Per-neuron, projected dual	+1–14% (faster)	(Chevalier et al., 2024)

These boundaries are of primary importance for practical DNN certification, allowing for stronger (less conservative) provable robustness.

4. Bounded-Range, Activation Bottlenecks, and Extrapolation Limits

A crucial capability boundary for sigmoidal networks is due to their bounded image: any architecture with a path through strictly bounded-activation layers suffers an activation bottleneck, sharply limiting the expressivity over unbounded targets:

Theorem: For any function $f:D\to\mathbb{R}$ unbounded on $D$ , a network composed of bounded activation(s) and post-activation Lipschitz mappings cannot produce predictions differing from $f$ by less than an unbounded error; i.e., $\sup_{t\in D} |f(t)-g(t)| = \infty$ (Toller et al., 2024).
LSTM/GRU bottleneck: Despite gating/recurrence, LSTM and GRU hidden states are trapped in $[-1,1]^d$ due to their sigmoidal and tanh bottlenecks, preventing trend or straight-line extrapolation.
Empirical outcome: Linear or ReLU-based architectures track unbounded sequences, sigmoidal models saturate and fail as ground truth leaves the training interval. Remedies involve skip connections, linear residuals, or unbounded output activations.

5. Sequential Modeling, Vanishing Gradients, and SST Extension

In recurrent/sequential architectures, classical sigmoid gating causes rapid gradient attenuation due to the maximal derivative (at most $1/4$) and exponential decay over time steps or layers:

Gradient propagation limit: For $T$ time steps, the Jacobian norm decays as $(1/4)^T$ ; beyond $T\sim 5-10$ , gradients vanish to machine precision (Subramanian et al., 2024).
Capability boundary: Fails to preserve information for long sequences, sparse data (missingness $>20\%$ ), or small dataset regimes ( $<400$ sequences). Classical GRU/LSTM thus underperform in these scenarios.
SST (Squared Sigmoid–Tanh) extension: Applies squaring to gate activations, amplifying strong signals, and partially restoring gradient flow. Empirically, this extends the learning boundary—delivering 4–5% gains in accuracy under high sparsity, recovery of rare pattern recall in sign language datasets, and reduction of test MSE by 70% in long-horizon regression (Subramanian et al., 2024).

6. Universality in Convolutional and Compact Settings

On compact domains $E^n$ , non-overlapping convolutional networks with sigmoidal activations retain the universality of classical MLPs: for every continuous target and every $\varepsilon>0$ , such architectures can achieve uniform approximation error below $\varepsilon$ (Chang, 2022). The only requirements are classical sigmoidal limit behavior and continuity. The complexity of approximation (e.g., network width or depth required for a given $\varepsilon$ ) follows the MLP theory, offering no advantage over densely connected layers but extending expressivity guarantees to CNN-style models.

7. Capability Boundaries in Separation and Classification

Shallow sigmoidal networks can perfectly classify any dataset sampled from a $k$ -separable distribution with positive $\delta$ -margin, tuning sharp transition layers via high gain and leveraging sigmoid saturation. The critical property is that the regions of decision uncertainty (the transition bands) can be made arbitrarily narrow relative to the margin, yielding zero classification error for well-separated data (Min et al., 2019). This mechanism sets a boundary: outside strict separability (e.g., nonzero measure mass close to decision boundaries or overlapping support), perfect classification is unattainable.

References:

(Wang et al., 2019): Approximation capabilities of neural networks on unbounded domains
(Goebbels, 2018): On Sharpness of Error Bounds for Single Hidden Layer Feedforward Neural Networks
(Zhang et al., 2022): Provably Tightest Linear Approximation for Robustness Verification of Sigmoid-like Neural Networks
(König et al., 2024): Automated Design of Linear Bounding Functions for Sigmoidal Nonlinearities in Neural Networks
(Chevalier et al., 2024): Achieving the Tightest Relaxation of Sigmoids for Formal Verification
(Toller et al., 2024): Activation Bottleneck: Sigmoidal Neural Networks Cannot Forecast a Straight Line
(Subramanian et al., 2024): Enhancing Sequential Model Performance with Squared Sigmoid TanH (SST) Activation Under Data Constraints
(Chang, 2022): Continuous approximation by convolutional neural networks with a sigmoidal function
(Min et al., 2019): Shallow Neural Network can Perfectly Classify an Object following Separable Probability Distribution