Sigmoid-Bounded Entropy Term

• Sigmoid-bounded entropy term is defined by applying a temperature-controlled sigmoid mapping to bounded surprisal measures, ensuring a strictly positive regularization bonus.
• It stabilizes learning by preventing runaway entropy in low-density regions, mitigating out-of-distribution optimization and controlling Q-value oscillations.
• Flexible parameters like temperature and offset allow tuning of exploration versus stability, making it applicable in RL, time-series complexity analysis, and uncertainty modeling.

A sigmoid-bounded entropy term is a mathematical modification of classical entropy regularization strategies that applies a temperature-controlled sigmoid mapping to the surprisal or distance measure, resulting in a bounded and strictly positive entropy bonus. This approach appears in reinforcement learning (RL) regularization, time-series complexity measures, and generalized information-theoretic models, unifying the benefits of exploration with stability and robustness against degenerate behaviors.

1. Formal Definition and Mathematical Formulation

In RL, the sigmoid-bounded entropy term is defined for a tanh-squashed Gaussian policy $a = \tanh(x)$, with $$x = \mu_\theta(s) + \sigma_\theta(s) \odot \epsilon$$, $\epsilon \sim \mathcal{N}(0,I)$. For each action dimension $i$, let the per-dimension log-density be $\log \pi_{\theta,i}(a_i|s)$, and define surprisal $s_i \equiv -\log \pi_{\theta,i}(a_i|s)$. The sigmoid-bounded entropy reward is then

$$h_i(s_i) = h_\mathrm{max} \cdot \sigma\left( \frac{s_i - m}{t} \right)$$

where $\sigma(z) = \frac{1}{1+e^{-z}}$ is the sigmoid function, $h_\mathrm{max} > 0$ is the per-dimension maximum, $m$ a center offset, and $t > 0$ a temperature. Summing over $d$ action dimensions yields the total

$$\mathcal{H}_\mathrm{sig}(s,a) = \sum_{i=1}^d h_i(s_i) \in (0, d h_\mathrm{max})$$

which replaces the conventional unbounded entropy bonus in the policy and value update equations (Wu et al., 22 Jan 2026).

In time-series analysis, the sigmoid-based membership function for refined composite multiscale fuzzy entropy (SRCMFE) is given by

$$\mu_S(d_{ij}; a, b) = \frac{1}{1 + \exp[- a (d_{ij} - b)] }$$

where $d_{ij}$ is a Chebyshev distance between embedding vectors, $a$ controls slope, and $b$ the threshold. The resulting entropy feature at scale $\tau$ is

$$\mathrm{SRCMFE}(m,r,a,b;\tau) = -\ln\frac{n_\tau^{m+1}}{n_\tau^{m}}$$

with $n_\tau^m$ counts aggregated over all offsets and pairings (Jiang et al., 2017).

In generalized information-theoretic form, the entropy summand for outcome $i$ is

$$K_i = -p(K_i) \ln p(K_i), \quad p(K_i) = \frac{1}{1 + e^{-K_i/E}}$$

where $p(K_i)$ is an informational “performance” variable, $E > 0$ is a scaling parameter (0811.0139).

2. Mitigation of Negative-Entropy-Driven Out-of-Distribution Optimization

Standard entropy regularization (as in SAC) operates using $-\log \pi(a|s)$, which is unbounded above and can dominate Bellman backups in regions of low policy density ($\pi(a|s) \to 0$), artificially inflating $Q(s,a)$ and driving optimization toward out-of-distribution (OOD) actions. The resulting entropy bonus can destabilize training, producing spikes in Q-values and leading policies into unsupported regimes.

With a sigmoid mapping, as in $h_i(s_i)$ above, the entropy bonus for each dimension is strictly bounded ($0 < h_i(s_i) < h_\mathrm{max}$). High-surprisal (very unlikely) actions saturate the bonus at $d h_\mathrm{max}$, while low-surprisal actions receive almost no additional bonus. This restricts OOD exploration and stabilizes updates, with the Q-function landscape forming a “bowl shape”—higher in the interior, lower near boundaries—rather than ever-rising edges (Wu et al., 22 Jan 2026). SRCMFE’s use of bounded membership similarly avoids degenerate or undefined entropy values for uncommon patterns (Jiang et al., 2017). In Jaeger’s entropy model, the sigmoid performance mapping ensures all terms are continuous and finite, never diverging under low-probability assignments (0811.0139).

3. Integration into Learning Frameworks

Reinforcement Learning (SigEnt-SAC):

• Critic Update: Incorporates the sigmoid-bounded entropy in the soft Bellman backup:

$$\hat{Q}^{k+1}(s,a) = (1-\alpha) \hat{Q}^k(s,a) + \alpha [r(s,a) + \gamma \mathbb{E}_{a'\sim\pi}[\hat{Q}^k(s', a') + \mathcal{H}_\mathrm{sig}(s',a')]]$$

A CQL-style regularizer adds a conservative penalty for in- and out-of-distribution actions.

• Actor Update: Optimizes a joint maximum-entropy and gated behavioral cloning (BC) objective:

$$J(\pi_\theta) = \mathbb{E}_{s, a \sim \pi_\theta}[Q^k(s, a) + \alpha \mathcal{H}_\mathrm{sig}(s, a)] - \lambda \mathbb{E}_{(s, a_\mathrm{exp}) \sim D_\mathrm{exp}}[p_\mathrm{gate} \|a_\mathrm{mean}(s) - a_\mathrm{exp}\|_2^2]$$

Both critic and actor updates ensure gradients are well-behaved due to the entropy bound (Wu et al., 22 Jan 2026).

Time-Series Complexity (SRCMFE):

• The sigmoid-bounded membership function replaces the exponential weighting, yielding well-defined entropy estimates even for small samples and at all coarse-graining scales (Jiang et al., 2017).

General Entropy Models:

• Jaeger’s framework utilizes the sigmoid as a probability–information link, ensuring bounded, interpretable entropy contributions, facilitating robust combination of classifier confidence scores (0811.0139).

4. Comparison to Traditional Entropy Formulations

Feature	Standard (SAC, FuzzyEn, Shannon)	Sigmoid-Bounded Variant
Bonus Magnitude	Unbounded as $\pi \to 0$ or $\Phi^{m+1} \to 0$	Bounded in $(0,h_\mathrm{max})$ or $(0,1)$
OOD Optimization Risk	High—pulls toward unsupported actions	Mitigated—bonus saturates
Gradient Stability	Unstable in low-density regions	Stable everywhere
Behavior Near Data Support	Weakly regularized—can collapse	Positive but limited—retains exploration
Robustness to Sample Size	Sensitive (entropy may be undefined)	Robust to short series and low counts

Classical entropy (e.g., $-\log \pi(a|s)$, $-p\ln p$) can suffer from instability and over-exploration under extreme probabilities or distances. Sigmoid-bounded formulations constrain entropy bonuses, yielding numerically stable, interpretable regularization across RL and time-series domains (Wu et al., 22 Jan 2026, Jiang et al., 2017, 0811.0139).

5. Theoretical and Empirical Properties

Theoretical:

• Boundedness: Entropy regularizers are provably finite for all inputs, precluding collapse or runaway gradients.
• Strict Positivity: Even for high-density actions (low surprisal, close patterns), the bonus remains strictly positive, maintaining a minimal stochastic incentive.
• Parameterization: Temperature $t$ and center $m$ (or slope $a$ and shift $b$ for SRCMFE) control the active support of the entropy bonus, tuning exploration versus stability.
• Continuity and Concavity: The sigmoid ensures all terms are continuous and differentiable, with concave behavior in probability space (0811.0139).

Empirical:

• RL Performance: SigEnt-SAC achieves 100% success rate faster than baselines, reduces OOD action ratio, and generalizes across four robot embodiments with minimal real-world interaction (Wu et al., 22 Jan 2026).
• Time-Series Analysis: SRCMFE yields reduced variance and robust entropy estimates for mechanical fault diagnosis, outperforming classical MFE under realistic conditions (Jiang et al., 2017).
• Classifier Combination: In pattern recognition, sigmoid-bounded confidence aggregation improves accuracy when combining multiple sources (0811.0139).

Ablation studies in RL demonstrate that omitting the sigmoid bound leads to increased Q-function oscillations and instability, validating its regularizing efficacy (Wu et al., 22 Jan 2026).

6. Applicability and Extensions

The sigmoid-bounded entropy term finds utility in diverse domains:

• Off-policy RL with limited expert data: Enables stable and efficient policy learning from a single trajectory, avoiding collapse or divergence even in the presence of sparse rewards (Wu et al., 22 Jan 2026).
• Complexity quantification in time series: The SRCMFE method provides a pragmatic approach to diagnose faults in machinery, extracting stable multiscale entropy features from short signals (Jiang et al., 2017).
• Generalized uncertainty modeling: Jaeger’s framework embeds the sigmoid-bound in information integration for decision-making, notably in combining classifier confidences or modeling perceptual uncertainty (0811.0139).

This suggests further research may extend these principles to domains requiring robust regularization of probabilistic models under data scarcity, adversarial conditions, or high-dimensional state spaces.

7. Mathematical Properties and Interpretations

Sigmoid-bounded entropy terms are characterized by:

• Bounded support in $[0,h_\mathrm{max}]$ (RL) or $(0,1)$ (SRCMFE/general entropy), constraining regularization.
• Smooth transitions between the encouraging and saturating regimes, defined by the temperature and center/threshold parameters.
• A plausible implication is that the sigmoid-bound naturally limits exploration to the support of the empirical data, preventing divergence toward unlikely states or actions.
• In perceptual models, the sigmoid mapping bridges “true” and “perceived” uncertainty, with congruence only at specific points (e.g., golden ratio solution) (0811.0139).

In summary, the sigmoid-bounded entropy term modifies classical entropy instruments by introducing bounded, differentiable, and tunable regularization, enhancing stability and interpretability in reinforcement learning, time-series complexity, and information-theoretic modelling while retaining core exploration benefits.