Temperature-Free InfoNCE for Contrastive Learning

• The paper introduces a temperature-free variant of InfoNCE by replacing temperature scaling with an arctanh mapping, removing the need for hyperparameter tuning.
• The method ensures improved gradient dynamics with uniformly scaled gradients and exact vanishing at perfect alignment, enhancing convergence stability.
• Empirical evaluations demonstrate comparable or improved performance across modalities such as image classification, graph representation, NLP, and recommendation tasks.

Temperature-free InfoNCE refers to a class of contrastive learning objectives that eliminate the temperature hyperparameter from the conventional InfoNCE loss, thereby simplifying deployment and yielding desirable gradient and convergence properties. The canonical formulation replaces the temperature scaling with an algebraic mapping, notably the inverse hyperbolic tangent function ($\arctanh$), allowing representation learning without the need for extensive temperature tuning.

1. Classical InfoNCE Loss and Temperature Scaling

The InfoNCE (NT-Xent) loss is foundational in contrastive self-supervised learning, operating on the principle of maximizing agreement between augmented views of the same data instance while minimizing agreement across instances. For a batch $\{x_1,\ldots,x_N\}$ with encoder $f(\cdot)$, the loss is defined as:

$$L = \sum_{i=1}^N -\log \left ( \frac{\exp( s_{i,i+}/\tau )}{\sum_{j=1}^N \exp( s_{ij}/\tau )} \right ),$$

where $$s_{ij} = \cos \theta_{ij} = f(x_i)\cdot f(\tilde{x}_j)$$ denotes cosine similarity, $\tau>0$ is the temperature, and $i+$ indexes the positive pair.

Temperature ($\tau$) is critical in calibrating the sharpness of the softmax distribution over similarities. Its choice affects both the expressivity and training dynamics, but its sensitivity often necessitates exhaustive hyperparameter search and consequently complicates pipeline design (Kim et al., 29 Jan 2025).

2. Temperature-Free Loss Formulation via $\arctanh$ Mapping

The temperature-free InfoNCE loss substitutes the “divide-by-$\tau$” operation in the logits with an invertible algebraic transformation. Specifically, letting $s_{ij}\in(-1,1)$,

1. Re-scale similarity to probability-like $p_{ij} = (1 + s_{ij})/2 \in (0,1)$.
2. Apply logit mapping:

$\logit(p_{ij}) = \log\frac{p_{ij}}{1-p_{ij}} = \log\left( \frac{1 + s_{ij}}{1 - s_{ij}} \right)$

Algebraically, $\logit \left( \frac{1+s}{2} \right ) = 2 \arctanh(s)$.

1. The temperature-free logit for each pair:

$\tilde \ell_{ij} = 2 \arctanh( s_{ij} )$

Inserted into the softmax cross-entropy:

$L = \sum_{i=1}^N -\log \frac{ \exp(2\,\arctanh(s_{i,i+})) }{ \sum_{j=1}^N \exp(2\,\arctanh(s_{ij})) }$

This reformulation is hyperparameter-free and does not require temperature calibration at any stage (Kim et al., 29 Jan 2025).

3. Analysis of Gradient Dynamics

Temperature-free InfoNCE exhibits improved gradient behavior over the standard variant. For a minimal setting (one positive $s^+=C$, one negative $s^-= -C$, $C\in(0,1)$):

• Standard (temperature-scaled):

$$\frac{\partial L_i}{\partial C} = \frac{2/\tau}{1+ \exp(2C/\tau)}$$

As $C\to 1$, the gradient does not vanish unless $\tau$ is very small, impeding exact optimum convergence. Conversely, if $\tau$ is made small to allow vanishing, gradients for moderate $C$ become negligible, causing training stagnation. The trade-off is further entangled with batch size $N$.

• Temperature-free (arctanh-based):

$$\frac{\partial L^{free}_i}{\partial C} = \frac{4 (N-1)(1-C)}{ (1+C) [ N(1-C)^2 + 4C ] }$$

As $C\to 1$, the gradient reliably vanishes for all $N$, guaranteeing exact alignment. Away from $C=1$, gradients remain well-scaled and monotonically decreasing with $C$, avoiding vanishing zones (Kim et al., 29 Jan 2025).

This mapping ensures both "zero-gradient at perfect alignment" and "uniformly alive gradients" throughout training, independent of batch size or application domain.

4. Empirical Evaluation Across Modalities

The temperature-free InfoNCE framework was evaluated on five distinct contrastive learning settings:

Task	Baseline Best Acc./Metric	Temp-Free Acc./Metric	Setup/Model
Image Classification	84.43 % (τ=0.25)	84.65 %	ResNet-18, SimCLR
Graph Representation	67.33 % F₁ (τ=0.5)	67.95 % F₁	GRACE
Anomaly Detection	≈97.215 (τ=0.25-0.5)	97.279	MSC, ResNet-152
Bias Mitigation in NLP	80.6 % LM	81.0 % LM	BERT+MABEL
Sequential Recommendation	0.1336 HR@1 (τ=0.8)	0.1360 HR@1	DCRec, MovieLens

On each task, the temperature-free method matches or marginally exceeds the best manually tuned InfoNCE variant, providing robust generalization across modalities without the cost of hyperparameter sweeps (Kim et al., 29 Jan 2025).

5. Implications for Pipeline Design and Hyperparameter Optimization

Temperature-free InfoNCE obviates the need for laborious temperature hyperparameter searches, removing one of the principal bottlenecks in large-scale contrastive model deployment. This is particularly advantageous in cross-domain retraining and when operational costs per trial are prohibitive. The method guarantees stable gradients without regard to batch size, model architecture, or data modality, unifying implementation strategies for vision, graph, language, recommendation, and anomaly detection tasks. A plausible implication is that future contrastive frameworks may converge toward temperature-free objectives to standardize training regimes and minimize sensitivity to architecture-specific properties (Kim et al., 29 Jan 2025).

6. Connections to Dual-Temperature and Dictionary-Free Contrastive Learning

Related work such as Zhang et al.'s dual-temperature InfoNCE (Zhang et al., 2022) decomposes the effects of temperature into “intra-anchor” and “inter-anchor” terms, allowing the scalar temperature to be set arbitrarily large to neutralize anchor hardness—thus enabling dictionary-free designs like SimMoCo and SimCo that outperform MoCo v2, without the complexity of queue or momentum encoders. Although not fully eliminating all temperature tuning (the intra-anchor temperature $\tau_\alpha$ still requires selection), it shows that decoupling the roles of temperature greatly simplifies pipeline design and supports high performance with much reduced negative sample sizes.

This suggests that the temperature-free InfoNCE is a strict advancement in removing brittle hyperparameters, whereas dual-temperature methods chart a compromise by restricting tuning to a single, much less sensitive component. Both trends indicate a shift toward automating or eliminating temperature selection in contemporary contrastive learning frameworks (Zhang et al., 2022).

7. Summary and Future Directions

The temperature-free InfoNCE loss, instantiated via the arctanh mapping, delivers all the benefits of conventional contrastive learning without the need for temperature calibration. It ensures improved optimization dynamics and generalizable results across a broad spectrum of applications. As the theory and empirical results validate, removing temperature as a hyperparameter streamlines reproducibility, robustness, and scalability of self-supervised pipelines. Future research may investigate alternate mappings or further unify other hyperparameter dependencies, developing fully automatic contrastive objectives suitable for heterogeneous data and model classes.