Precision-Weighted FedAvg in Federated Learning

• Precision-Weighted FedAvg is a federated learning aggregation method that uses statistical precision metrics to weight client model updates, enhancing convergence and personalization.
• It employs methodologies such as Fisher information and inverse-variance weighting to provide direction-aware aggregation for mitigating non-IID client drift.
• Empirical results demonstrate notable improvements in accuracy and convergence speed across tasks like image classification and segmentation.

Precision-Weighted FedAvg (PW-FedAvg) refers to a class of federated learning (FL) aggregation strategies that enhance or replace the canonical FedAvg scheme by applying parameter-level or client-level weights based on statistical measures of precision—often derived from Fisher information, inverse-variance, or progress in local optimization—to mitigate client drift, handle heterogeneity, and improve global convergence and personalized performance. The literature encompasses several instantiations, including Fisher-weighted function-space aggregation (FedFish), variance-based inverse weighting, and performance-driven weighting schemes, with empirical and theoretical support across diverse FL tasks (Dhawan et al., 2023, Chang et al., 20 Jan 2026, Mächler et al., 2021, Reyes et al., 2021).

1. Motivation and Conceptual Overview

Standard FedAvg aggregates client model updates using weights proportional to client dataset sizes, uniformly applying the same scalar to all parameters per client. This approach neglects the information content and statistical certainty of individual client updates, making it vulnerable to misaligned (non-IID) client distributions and producing suboptimal global models when models drift after intensive local training. Precision-weighted approaches replace or augment this data-size weighting with measures that reflect the informativeness or reliability of each client's update per parameter, leveraging local uncertainty (variance), Fisher information, or actual loss improvement to produce direction-, parameter-, or performance-aware aggregation (Dhawan et al., 2023, Chang et al., 20 Jan 2026).

2. Algorithmic Formulations

2.1 Fisher-Weighted Aggregation

FedFish (Dhawan et al., 2023) and Fisher-Informed Parameterwise Aggregation (FIPA) (Chang et al., 20 Jan 2026) use the empirical Fisher Information Matrix (FIM) computed by each client to derive per-parameter "precision" weights. For FedFish, the server aggregate is given as:

$$\theta_G^* = \frac{\sum_{i=1}^N [\operatorname{diag}(F_i) \circ \theta_i]}{\sum_{i=1}^N \operatorname{diag}(F_i)}$$

where $F_i$ is the empirical Fisher diagonal from client $i$, $\,\circ\,$ denotes element-wise multiplication. This aggregation arises from a quadratic approximation of the function-space KL divergence between global and client models.

FIPA generalizes this by using the (low-rank) full FIM, yielding an update:

$$\theta^{(k+1)} = \theta^{(k)} + \sum_{m=1}^M B_m^{(k)}\Delta\theta_m^{(k)}$$

with each $$B_m^{(k)} = (N_m/N)\,[\widehat H^{(k)}]^{\dagger}\,\widehat H_m^{(k)}$$, replacing scalar client weights with parameter-wise matrices (Chang et al., 20 Jan 2026).

2.2 Variance- or Progress-based Weighting

Precision-Weighted FedAvg (PW-FedAvg) (Reyes et al., 2021) and FedCostWAvg (Mächler et al., 2021) implement aggregation based on client-level inverse variance or loss-improvement statistics:

• PW-FedAvg: Clients estimate local precision via $\lambda_k = 1/\operatorname{Var}(g_k)$ (variance of stochastic gradient or Adam's second moment) and the server aggregates:

$$w_{t+1} = \frac{\sum_{k=1}^K \lambda_k\,w_{t+1}^k}{\sum_{k=1}^K \lambda_k}$$

• FedCostWAvg: Blends data-size and "precision" (relative loss reduction) per client:

$$w^{t+1} = \sum_k \left[ \alpha \frac{n_k}{N} + (1-\alpha) \frac{\Delta_k^t}{K} \right] w_k^{t+1}$$

with $\Delta_k^t = \ell_k(w_k^t)/\ell_k(w_k^{t-1})$, and $K = \sum_k \Delta_k^t$. A tuning hyperparameter $\alpha\in[0,1]$ controls the trade-off.

3. Theoretical Rationale and Interpretation

All precision-weighted schemes seek to (a) preserve client updates that are informative or reliable, (b) dampen contributions from clients or parameter directions with high uncertainty or low signal-to-noise, and (c) match the global model more closely to actual client outcome distributions. The Fisher-based methods (FedFish, FIPA) explicitly interpret precision as local curvature or sensitivity in function space, ensuring that parameters well-constrained by client data dominate the aggregate (Dhawan et al., 2023, Chang et al., 20 Jan 2026). PW-FedAvg and FedCostWAvg interpret precision either through variance of stochastic gradients (fixed effects meta-analysis analogy) or empirical cost decrease (Bayesian confidence in update direction) (Reyes et al., 2021, Mächler et al., 2021).

Mathematically, Fisher weighting minimizes a quadratic surrogate of function-space KL divergence across clients. The fixed-effect estimator analogy for variance-based weighting penalizes noisy, unreliable parameter updates (Reyes et al., 2021).

4. Practical Implementation and Computational Aspects

Precision-weighted aggregation can be implemented with varying granularity and computational overhead:

• Diagonal Fisher weighting (FedFish/PW-FedAvg): Simple to compute via running gradient or Adam's second moment, negligible overhead, practical for large models (Dhawan et al., 2023, Reyes et al., 2021).
• Full or low-rank FIM (FIPA): Employs rank-$r$ sketches, subspace iteration on the client, QR-based merging on the server, maintaining efficiency by reducing $\mathcal{O}(p^2)$ cost to $\mathcal{O}(pr)$ per round, with moderate extra communication (Chang et al., 20 Jan 2026).
• Loss-improvement weighting (FedCostWAvg): Adds only a scalar per client per round and minimal client computation (Mächler et al., 2021).

Key considerations include ensuring comparability of loss scales across clients (FedCostWAvg), regularizing and tuning step sizes for FIPA, and using warmup or two-stage schemes to stably enter a training basin before precision-weighted aggregation (Chang et al., 20 Jan 2026, Mächler et al., 2021).

5. Empirical Performance and Benchmarks

Experimental results across multiple domains validate the advantages of precision-weighted Federated Averaging:

Method	Domain / Task	Non-IID Resilience	Accuracy Improvement	Speedup	Reference
FedFish	EMNIST, CIFAR100, C4->Stack	Robust to drift	+2.5 pp (EMNIST), +3 pp (CIFAR100), +7% (C4)	-	(Dhawan et al., 2023)
FIPA	Regression, PDE, CIFAR-10	Reduced drift	+9.5 pp (CIFAR-10), 30–50% lower L2 PDE err	-	(Chang et al., 20 Jan 2026)
PW-FedAvg	MNIST, Fashion, CIFAR-10	High heterogeneity	+9 pp (MNIST), +13 pp (Fashion, non-IID)	20–37× (FMNIST)	(Reyes et al., 2021)
FedCostWAvg	Tumor Segmentation (FETS)	Improved generaliz.	+3–4 pp DICE (all classes)	-	(Mächler et al., 2021)

Table: Summary of empirical improvements of representative precision-weighted aggregation methods.

Empirical findings consistently report: (1) faster convergence; (2) systematically higher final accuracy, especially as local epochs increase and under highly heterogeneous (non-IID) data splits; (3) improved personalization and transfer performance without added communication rounds (Dhawan et al., 2023, Reyes et al., 2021, Chang et al., 20 Jan 2026, Mächler et al., 2021).

6. Limitations and Practical Considerations

Precision weighting introduces new design and tuning choices. FedFish and FIPA rely on accurate estimation of Fisher information, which can be noisy with small client batch sizes or poorly estimated gradients. For cost- or loss-based weightings (FedCostWAvg), misalignments in loss scaling across clients can bias aggregation; normalization or clipping may be required. Communication costs rise when transmitting low-rank FIM sketches (FIPA), though this is minor relative to model parameter size for moderate $r$ (Chang et al., 20 Jan 2026). There is no fundamental increase in privacy risk over baseline FedAvg, but no mitigation of inference attacks is provided (Reyes et al., 2021). Some approaches lack formal convergence proofs; their guarantees remain under practical, empirically validated regimes (Reyes et al., 2021, Mächler et al., 2021).

7. Broader Impact and Future Directions

Precision-weighted aggregation strategies have demonstrated robust benefits in FL scenarios typified by data heterogeneity, client drift, and personalized inference requirements. They admit a principled Bayesian or meta-analytic interpretation and have been integrated with state-of-the-art client optimization routines. Ongoing research continues to refine efficient FIM approximation, better heterogeneity modeling, robust weighting under adversarial or noisy clients, and extensions to model personalization and privacy-enhancing settings. The documented empirical successes across image, text, and scientific data domains establish precision-weighted FedAvg as a central technique for scalable and robust federated learning (Dhawan et al., 2023, Mächler et al., 2021, Reyes et al., 2021, Chang et al., 20 Jan 2026).