Learning in Curved Weight Space:Exponential-Linear Weight Reparameterization for Improved Optimization

Abstract: Many neural networks operations have a multiplicative nature rather than additive: halving or doubling a norm are analogous relatively but require unequal optimization distances when taking linear steps. Adaptive optimizers such as Adam normalize updates per coordinate, but update steps remain additive; weights with very different magnitudes receive similarly sized absolute changes, producing very different relative perturbations. We introduce \textbf{\method} (\textbf{\methodshort}), a weight reparameterization for neural networks that combines a sign-aware symmetric-exponential pathway with an identity-like linear pathway. The symmetric-exponential pathway is near-linear for small raw weights but increasingly curved at larger magnitudes. Additive updates in logarithmic space map to magnitude-proportional changes in effective weight space. The linear pathway provides a direct route through the transform that we hypothesize stabilizes optimization, while learnable scale, curvature, and offset parameters control balance between pathways and the curvature of the exponential pathway. These components create a curved parameter-space geometry that empirically improves speed of loss descent over standard linear parameterization. We also identify a useful \emph{mismatched initialization}: raw weights are chosen so a symmetric version of the transform matches Xavier statistics, but training uses an asymmetric forward transform that leaves positive weights at full strength while making negative weights smaller in magnitude; in small-model ablations, this improves early optimization and may act as a form of symmetry breaking. We train transformers on OpenWebText over nine width$\times$depth configurations, \methodshort reaches matched validation loss in 1.32--1.49$\times$ fewer training steps, with the largest widths seeing the biggest gains.

• The paper introduces SEL, a hybrid exponential-linear reparameterization that aligns additive optimizer updates with multiplicative computation in neural networks.
• It leverages dual pathways—learnable symmetric-exponential and linear—to dynamically adjust weight updates for improved convergence speed.
• Experimental results demonstrate SEL reduces training steps by up to 1.49× while maintaining negligible inference overhead.

SymExpLin: Exponential-Linear Weight Reparameterization for Improved Optimization

Motivation: Addressing the Mismatch Between Optimizer Dynamics and Multiplicative Computation

Neural network training, especially with optimizers like Adam, fundamentally operates through additive updates in weight space. However, many transformations in modern architectures, notably matrix multiplications in transformers, exhibit multiplicative computational effects. While adaptive optimizers attempt to normalize gradient dynamics, they remain agnostic to the parameter magnitude, which leads to inconsistent relative updates: small weights can be perturbed heavily in relative terms, whereas large weights receive diminutive relative updates. As networks scale, the distribution of trained weights develops heavy tails—few parameters migrate far from their initialization to support functional outlier phenomena critical for large model expressivity.

Figure 1: Weight distributions in pretrained models shift to heavy tails, with maximum weights exceeding Xavier initialization by over $30\times$ in Qwen3-4B, while most weights remain near zero.

Attempts to statically correct such mismatches with schemes like Maximal Update Parameterization ($\mu$P) have proven effective at normalization across tensors, but fail to provide per-parameter dynamism as training progresses and weight distributions spread. The result is a persistent inefficiency: traversing from $1$ to $2$ in absolute weights (doubling effect) is linearly "harder" than traversing in the inverse direction, despite having the same relative effect on the function.

The SymExpLin (SEL) Transform: Architecture and Implementation

SEL introduces a nonlinear, sign-aware weight reparameterization that combines two pathways: a symmetric-exponential function and an identity-like linear contribution. The parameterization is as follows:

$$w_{\mathrm{eff}} = \operatorname{sign}(w) [E_\theta(|w|) + L_\theta(w)] + \Delta_\phi$$

where

• $E_\theta(|w|)$ is the learnable symmetric-exponential pathway,
• $L_\theta(w)$ is the linear pathway,
• $\Delta_\phi$ is an (optional) additive correction (e.g., LoRA-style).

The exponential component is parameterized with curvature control ($\beta$), offset ($n$), multipliers, and scaling factors, enabling per-parameter tuning of the transition between near-linear (small $\mu$0) and exponentially curved (large $\mu$1) regimes.

Figure 2: The SEL symmetric-exponential transform generalizes symexp with learnable curvature $\mu$2 and offset $\mu$3, enabling model-scale adaptation to typical weight magnitudes.

Crucially, SEL is designed to be maximally practical: all transformations are elementwise and can be efficiently compiled, with all parameterization folded away at inference into standard weight matrices, ensuring zero inference cost overhead.

Mismatched Initialization and Combination Modes

A key empirical finding is the benefit of mismatched initialization: initializing weights such that a congruent pathway combination matches target Xavier statistics, but evaluating with a mismatched (asymmetric) forward-path combination—where the sign wraps both linear and exponential pathways—leads to improved early optimization. This asymmetry results in negative weights being under-initialized (by 20–25%), breaking initialization symmetry and facilitating more effective training dynamics, particularly in small-scale settings.

Figure 3: Congruent and mismatch combination modes coincide for positive weights; the mismatch mode downscales effective negative weights, driving initialization asymmetry.

Experimental Evaluation and Ablations

SEL was tested on transformer models across a $\mu$4 grid of widths ($\mu$5, $\mu$6, $\mu$7) and depths ($\mu$8, $\mu$9, $1$0) on OpenWebText. Across all configurations, SEL consistently outperformed baseline parameterizations in speed and final validation loss. SEL achieved matched performance in $1$1 to $1$2 fewer steps compared to baseline, with the widest models gaining the largest acceleration. For the largest profiled model ($1$3), a $1$4 reduction in steps and only $1$5 per-step overhead equated to an estimated $1$6 wall-clock speedup.

Figure 4: Across all model scales, SEL variants (solid) consistently drive faster and deeper evaluation loss descent than baseline (dashed), with $1$7 indicating the final loss gap.

Ablation studies further support SEL's necessity and modularity:

• Removal of the linear pathway (exponential-only) leads to clear training instability and divergence.

  Figure 5: Exponential-only ablation; without the linear pathway, SEL fails to learn, highlighting the necessity of additive-local optimization routes.

• Learning rate (LR) annealing for scale parameters facilitates early commitment to useful exponential magnitudes, improving stability and convergence.

Figure 6: LR annealing for scale parameters triggers early adaptation in pathway balances (right) and consistently marginally outperforms fixed multipliers (left).

SEL also results in trained weight distributions with heavier tails compared to baseline parameterizations.

Figure 7: Training with SEL yields more pronounced heavy tails in effective weight distributions than standard linear parameterization.

The learned transform exhibits per-module adaptation, with exponential branches amplifying large raw weights while retaining control near zero.

Figure 8: Across sampled modules, learned SEL transforms retain near-linear control near the origin but amplify larger inputs, reflecting hybrid adaptation.

Analysis and Practical Considerations

SEL offers a dynamic, per-parameter analog to static parameterizations like $1$8P. By making $1$9 magnitude-dependent, SEL ensures that the optimization dynamics naturally scale update sensitivity for large-magnitude weights, accelerating their convergence relative to the rest. This is critical for supporting emergent phenomena in large models such as massive outlier weights ("super weights") that are disproportionately functionally important and challenging to obtain through purely additive optimization [dettmers2022llm], [yu2024superweight].

SEL's structured scale parameterization (row$2$0column multiplicative) achieves a balance between expressivity and computational cost, enabling module and position-specific adaptation without excessive parameter overhead. Because all transformation parameters (exponential/linear scales, curvature, offsets) can be folded post-training, SEL imposes no architectural change or inference penalty, and is fully compatible with existing infrastructure.

Implications and Future Directions

SEL's methodology addresses both a well-motivated theoretical inefficiency in neural optimization and a practically emergent challenge at larger scales, where heavy-tailed weight distributions and functional outlier weights dominate transformer dynamics. Its compatibility with standard optimizers and architectural conventions makes SEL viable for wide deployment without inference cost.

Potential future developments include:

• Pretrained model conversion and fine-tuning in SEL space, leveraging the invertibility of the transformation.
• Extensions to other tasks and architectures, as current evidence is limited to language modeling.
• Theoretical characterization of the SEL optimization landscape and further analysis of initialization asymmetry.

Finally, related parameterizations such as sinh-based transforms may offer improved smoothness with comparable performance, indicating further avenues for investigation.

Conclusion

SymExpLin presents a principled, practical solution to the incompatibility between additive optimizer dynamics and the multiplicative nature of neural network computations. Its exponential-linear hybrid parameterization, combined with structured scaling and initialization asymmetry, yields substantial speed improvements in convergence, better support for heavy-tailed and outlier-prone weights, and robust, inference-neutral deployment. Collectively, SEL demonstrates that modifying parameter-space geometry, rather than only optimizer dynamics, is a compelling pathway for neural network training innovation.