Federated Freeze A LoRA (FFALORA)

• The paper introduces FFALORA, which freezes one LoRA adapter matrix (A) and optimizes only the other (B) to achieve exact model aggregation in federated learning.
• FFALORA reduces communication overhead and enhances noise robustness under differential privacy, ensuring stability even with heterogeneous client data.
• Variants such as alternating freeze, adaptive rank selection, and personalized approaches allow targeted trade-offs between expressivity, efficiency, and robustness in diverse settings.

Federated Freeze A LoRA (FFALORA) is a family of parameter-efficient federated fine-tuning techniques for large-scale neural networks using Low-Rank Adaptation (LoRA). The core principle is to freeze one LoRA adapter matrix (typically the "down" projection A) across all clients and rounds, while only updating and communicating the other matrix (B). This simple constraint yields exact model aggregation, reduced communication overhead, and robust theoretical guarantees, especially under privacy-preserving constraints and heterogeneous data distributions. FFALORA variants include permanent freeze, alternating freeze, adaptive rank selection, and extensions for personalized federated learning in multimodal and statistical settings.

1. Mathematical Foundations

Under LoRA, the adapted weight for any linear layer of a model is parameterized as

$$W = W_0 + \Delta W = W_0 + \left(\frac{\alpha}{r}\right) B A$$

with:

• $W_0 \in \mathbb{R}^{d \times k}$: frozen, pretrained base weight,
• $A \in \mathbb{R}^{r \times k}$: "down" projection, initialized (e.g. $A \sim \mathcal{N}(0, \sigma^2)$),
• $B \in \mathbb{R}^{d \times r}$: "up" projection, initialized as $B=0$,
• $r$: LoRA rank, $\alpha$: scaling factor.

In standard LoRA, both $A$ and $B$ are trainable.

In FFALORA (permanent freeze variant), $A$ is fixed once and never updated, while $B$ is optimized locally on each client. The forward weight at round $t$ is

$W^{(t)} = W_0 + B^{(t)} A_0$

where $A_0$ is the globally broadcast, frozen adapter. Only $B$ is updated by local gradient methods and then aggregated by the central server.

Under classic federated averaging (FedAvg), FFALORA ensures

$$\sum_{i}p_i (B_i A_0) = \left(\sum_{i}p_i B_i \right) A_0$$

yielding exact aggregation of updates at the server with no cross-term aggregation bias or need for high-rank residual corrections (Sun et al., 2024, Singhal et al., 2024). This is in contrast to standard LoRA, where

$$\sum_i p_i B_i A_i \neq \left(\sum_i p_i B_i \right) \left(\sum_i p_i A_i \right)$$

unless all $A_i$ are equal.

2. Algorithmic Structure and Variants

The FFALORA workflow is:

1. Server Initialization: Broadcast frozen $W_0$, initialize $A_0$ (random Gaussian), initialize $B_0$.
2. Local Client Training: For each communication round,
   • Clients receive the latest global $B$.
   • $A_0$ is fixed, only $B$ is locally updated using (potentially DP-protected) gradients.
   • After $\tau$ local steps, clients send $B$ updates to the server.
3. Server Aggregation: The server performs FedAvg on received $B_i$, computes $B^{t+1} = (1/K) \sum_{i=1}^K B_i$, and broadcasts new $B$ for the next round.

Alternating Freeze FFALORA: To avoid the expressivity bottleneck of permanently frozen $A$, an alternating schedule optimizes $B$ in odd rounds (with $A$ frozen) and $A$ in even rounds (with $B$ frozen), enabling exploration of the full low-rank parameter space (Koo et al., 2024, Zhou et al., 29 Oct 2025).

Adaptive Rank FFALORA: Upload budgets per client can be tailored by using local importance score masking, where each client selects a subset of ranks to communicate based on the Frobenius norm of their update's contribution. This mechanism ensures communication efficiency and robustness in resource-heterogeneous federated environments (Koo et al., 2024).

Personalized FFALORA (Two-Level): A bilevel adaptation structure injects shared (global) low-rank adapters ($A^{(0)},B^{(0)}$) and tiny, client-specific adapters ($A^{(c)},B^{(c)}$) per client, supporting personalized ranking and federated fine-tuning with negligible added communication cost (Hao et al., 5 Mar 2025).

3. Theoretical Properties

• Exact Aggregation: Fixing one LoRA factor (typically $A$), FFALORA ensures that product-of-averages coincides with average-of-products, eliminating all cross-terms and aggregation bias in federated learning updates (Sun et al., 2024, Singhal et al., 2024).
• Noise Robustness: Under differential privacy, FFALORA propagates additive noise only along one adapter channel (e.g., $(B+\xi)A_0$), avoiding second-order noise amplification present in joint $A$-$B$ update schemes ($(B+\xi_B)(A+\xi_A)$) (Sun et al., 2024, Singhal et al., 21 Feb 2025).
• Smoothness: If $F(W)$ is $L$-smooth, then $F(W_0 + B A_0)$ is $L\|A_0\|^2$-smooth in $B$, ensuring FedAvg convergence. If both $A$ and $B$ are optimized jointly, uniform Lipschitz continuity does not hold (Sun et al., 2024).
• Expressivity and Robustness: Permanent freeze restricts the solution space to those reachable by the frozen factor ($A_0$). Alternating freeze restores full expressivity over two rounds but incurs more communication cost (Koo et al., 2024). Adaptive rank masking further enables selective exploration of important subspaces.
• DP Guarantees: FFALORA's reduction in trainable parameters lowers the amount of additive DP noise required, leading to improved performance under the same privacy budget (Sun et al., 2024, Singhal et al., 21 Feb 2025).

4. Empirical Evaluation

Experiments consistently demonstrate:

• Performance: For RoBERTa-large (GLUE: MNLI, SST-2, QQP, QNLI), GSM-8K, and LLaMA-7B, FFALORA matches or outperforms vanilla federated LoRA and full-model fine-tuning under both privacy-preserving (DP) and standard FL (Sun et al., 2024, Koo et al., 2024). Example accuracies (ε=6):
  • MNLI-matched: LoRA 82.0±10.7 vs FFALORA 85.0±1.1
  • MNLI-mismatched: 82.5±10.9 vs 85.6±1.0
  • GSM-8K (LLaMA-7B): FFALORA 17.12% vs LoRA 15.68%
• Robustness to Heterogeneity: FFALORA is more stable under label/class-based non-i.i.d. splits and severe data skew. Alternating freeze provides additional robustness in extreme heterogeneity or low-rank settings (Koo et al., 2024, Hao et al., 5 Mar 2025).
• Communication Savings: FFALORA halves the communication cost compared to conventional federated LoRA, as only one adapter matrix (typically $B$) is exchanged. Alternating freeze further reduces uplink cost to 42.97% in MIMO settings (Zhou et al., 29 Oct 2025).
• Computation Efficiency: Backpropagation is performed only over the unfrozen adapter, resulting in nearly $2\times$ reduction in adapter-only layer computation (Sun et al., 2024).

Empirical Results Table (selected, (Sun et al., 2024), ε=6):

Task	LoRA (%)	FFALORA (%)	Variance (LoRA/FFALORA)
MNLI-matched	82.0±10.7	85.0±1.1	High/Low
MNLI-mismatched	82.5±10.9	85.6±1.0	High/Low
SST-2	94.3±2.1	94.3±1.7	Comparable
QQP	83.5±3.3	84.4±0.6	High/Low
QNLI	89.0±6.7	90.4±1.9	High/Low

5. Extensions, Adaptive Mechanisms, and Limitations

• Alternating Freeze (LoRA-A²/Fed-PELAD): Alternates optimization between $A$ and $B$ adapters over rounds, avoiding permanent expressivity loss. Learning-rate ratio tuning ($\eta_B/\eta_A \approx 5$) provides convergence stability (Koo et al., 2024, Zhou et al., 29 Oct 2025). Empirically, alternating freeze yields several percentage points gain in extreme heterogeneity compared to permanent freeze.
• Adaptive Rank Selection: Per-client masking of adapter ranks based on update importance scores enables efficiency and robustness in settings with severe client resource heterogeneity (Koo et al., 2024).
• Two-Level Adaptation (PF2LoRA): Embeds both shared and client-specific LoRA modules; each client automatically discovers its effective rank using a bilevel objective. Communication remains minimal—only shared adapters are transmitted (Hao et al., 5 Mar 2025).
• LoRA-FAIR: Incorporates server-side bias correction and unified client initialization to reduce drift and aggregation errors (Bian et al., 2024).

Limitations include reduced adaptation capacity if rank is too low and loss of expressivity under severe data variation with permanent freeze. Exact-aggregation methods (FedEx-LoRA, Fed-SB) may outperform FFALORA in some centralized tasks (Singhal et al., 2024, Singhal et al., 21 Feb 2025). Adaptive schedules and module-wise freeze strategies are active areas of research.

6. Practical Guidelines and Hyperparameter Choices

• Rank ($r$): 8–16 is typically optimal; higher ranks provide diminishing returns under strong privacy constraints (Sun et al., 2024).
• Learning Rate ($\eta$): Wide search is recommended (0.1–1.0 for $B$); no tuning of scaling parameter $\alpha$ since $A$ is fixed.
• Clipping Norm ($C$): Values in {2, 5, 10}; monitor $\|\nabla_B\|$ distribution under DP.
• DP Budget ($\epsilon$): FFALORA tolerates lower $\epsilon$, maintaining accuracy even under strong DP.
• Initialization: Standard Gaussian for $A_0$ works; orthogonal initialization may reduce variance marginally.
• Heterogeneity: FFALORA is preferred in cross-silo non-i.i.d. regimes; alternation or adaptive masking provides further resilience.
• Resource Constraints: FFALORA naturally extends to mobile or edge scenarios; for ultra-low bandwidth, low ranks ($r = 8$) offer 1–2-dB performance degradation for a fourfold decrease in communication (Zhou et al., 29 Oct 2025).
• Secure Aggregation & Privacy: FFALORA also provides a stronger privacy guarantee, as only task-level coefficients are exchanged and the attack surface for membership inference is reduced (Mao et al., 2024).

• "Improving LoRA in Privacy-preserving Federated Learning" (Sun et al., 2024)
• "Towards Robust and Efficient Federated Low-Rank Adaptation with Heterogeneous Clients" (Koo et al., 2024)
• "Fed-PELAD: Communication-Efficient Federated Learning for Massive MIMO CSI Feedback with Personalized Encoders and a LoRA-Adapted Shared Decoder" (Zhou et al., 29 Oct 2025)
• "A Survey on LoRA of LLMs" (Mao et al., 2024)
• "Personalized Federated Fine-tuning for Heterogeneous Data: An Automatic Rank Learning Approach via Two-Level LoRA" (Hao et al., 5 Mar 2025)
• "FedEx-LoRA: Exact Aggregation for Federated and Efficient Fine-Tuning of Foundation Models" (Singhal et al., 2024)
• "Fed-SB: A Silver Bullet for Extreme Communication Efficiency and Performance in (Private) Federated LoRA Fine-Tuning" (Singhal et al., 21 Feb 2025)
• "LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinement" (Bian et al., 2024)
• "Frugal Federated Learning for Violence Detection: A Comparison of LoRA-Tuned VLMs and Personalized CNNs" (Thuau et al., 20 Oct 2025)

In summary, Federated Freeze A LoRA introduces a conceptually simple yet powerful freezing constraint into federated LoRA, achieving exact aggregation, communication/computation reduction, stability under privacy and heterogeneity, and competitive accuracy on diverse FL benchmarks across language, vision, and wireless domains. Its adaptability to alternating schedules and adaptive rank selection makes it a foundational scheme for parameter-efficient federated adaptation.