DP-FedSOFIM: Efficient DP Federated Learning

• The paper introduces DP-FedSOFIM, a federated optimization algorithm that uses a server-side Fisher Information Matrix preconditioner to accelerate differentially private learning.
• It reduces client-side memory and computation to O(d) by shifting costly FIM operations to the server, thereby overcoming the limitations of dense second-order methods.
• Empirical evaluations on CIFAR-10 demonstrate that DP-FedSOFIM achieves faster convergence and higher test accuracy across various privacy regimes compared to first-order methods.

DP-FedSOFIM is a federated optimization algorithm designed to accelerate differentially private federated learning (DP-FL) using an efficient, server-side approximation of the Fisher Information Matrix (FIM) as a natural gradient preconditioner. It addresses key bottlenecks of existing second-order methods in high-dimensional or resource-constrained federated contexts, improving convergence under strict privacy regimes through careful algorithmic structure and privacy-preserving updates (Nair et al., 14 Jan 2026).

1. Background and Motivation

Federated learning with differential privacy (DP-FL) requires noise injection (typically via the Gaussian mechanism) and per-example gradient clipping to ensure formal $(\epsilon, \delta)$-DP guarantees. Tight privacy budgets, such as $\epsilon < 2$, necessitate high noise levels, which can significantly degrade signal quality and slow optimization. Conventional first-order methods (e.g., DP-FedGD, DP-FedAvg) that employ isotropic learning rates suffer especially in ill-conditioned optimization tasks. Recent second-order approaches like DP-FedNew attempt to address curvature adaptivity but require clients to store and communicate dense $O(d^2)$ covariance matrices, limiting scalability for large models or constrained devices.

DP-FedSOFIM fundamentally re-architects the deployment of second-order information by shifting all FIM-related computation to the server, while requiring only $O(d)$ memory and computation on the client—a critical advantage in federated settings where model dimensionality $d$ can be large and client resources limited.

2. Algorithmic Framework

DP-FedSOFIM operates by aggregating privatized client gradients and employing a regularized FIM preconditioner, updated in rank-one form, at the server. The FIM is used to construct a natural gradient direction that significantly enhances conditioning compared to first-order approaches, while the use of the Sherman–Morrison formula ensures computational efficiency.

2.1. Fisher Information Matrix Preconditioning

The empirical Fisher Information Matrix at parameter $\theta$ is defined as

$$\mathcal{F}(\theta) = \mathbb{E}_{x\sim D} \left[ \nabla_\theta \ell(\theta; x) \nabla_\theta \ell(\theta; x)^\top \right] + \lambda I_d$$

for regularization parameter $\lambda>0$. The server maintains a momentum buffer $M^t$, yielding a rank-one approximation: $$\widehat{\mathcal{F}^t} = M^t (M^t)^\top + \rho I_d$$ where $\rho$ is a regularization parameter. Sherman–Morrison inversion enables the update: $$H^t = \frac{1}{\rho} I_d - \frac{M^t (M^t)^\top}{\rho^2 + \rho \|M^t\|_2^2}$$ All server-side matrix operations thus reduce to $O(d)$ time and memory.

2.2. Client- and Server-Side Steps

The protocol separates local gradient computation (with clipping and noise addition) from server-side aggregation and preconditioned updates. Each client computes and transmits clipped, noisy gradients; the server aggregates, updates the FIM estimate, and computes a natural gradient update.

Algorithmic Steps Overview:

Step	Location	Complexity
Gradient computation, clipping, noise	Client	$O(d)$
Rank-one FIM update, inversion	Server	$O(d)$
Preconditioned parameter update	Server	$O(d)$

No $O(d^2)$ buffers or client-side matrix inversions are required, unlike previous second-order approaches.

3. Privacy Guarantees

Each client's gradient release is implemented via the Gaussian mechanism, with sensitivity $\Delta = 2C/|\mathcal{D}_{\min}|$ and noise scale $$\sigma_{\text{release}} = C\,\sigma/(\sqrt{n}\;|\mathcal{D}_{\min}|)$$. Across $T$ communication rounds, privacy is composed using techniques such as hockey-stick divergence or Gaussian-DP accounting.

Crucially, the FIM-based preconditioning and all subsequent server computations are pure post-processing applied to privatized aggregates. Thus, by the post-processing theorem, these operations do not incur additional privacy loss (Nair et al., 14 Jan 2026). The entire protocol maintains end-to-end ($\epsilon, \delta)$-DP as defined by standard DP theory.

4. Convergence Properties and Computational Complexity

Under the assumptions of $L$-smoothness, strong convexity ($\mu$), and bounded variance/bias in clipped gradients, DP-FedSOFIM achieves linear convergence to a noise- and clipping-determined neighborhood of the optimum. Specifically, for step size $\eta$ chosen according to regularization and problem constants, the main convergence guarantee is

$$\mathbb{E}[F(\theta^T) - F(\theta^*)] \leq r^T (F(\theta^0) - F(\theta^*)) + O\left( \frac{d (C\,\sigma)^2}{\mu\,\rho^2\,n^2\,|\mathcal{D}|^2} \right) + O(\zeta_{\max}^2)$$

for contraction coefficient $r$ depending on conditioning and step size. The $O(d)$ per-round computation matches the complexity of first-order methods and surpasses prior second-order techniques in scalability. In contrast, DP-FedNew and related methods require $O(d^2)$ memory and $O(d^3)$ computations for each client.

5. Empirical Evaluation

DP-FedSOFIM was empirically benchmarked on CIFAR-10 using a federated IID partition across 20 clients, each training a frozen ResNet-20 feature extractor with a linear head. Across privacy regimes defined by $\epsilon \in \{0.5, 1, 2, 5, 10\}$ and $\delta=10^{-5}$, DP-FedSOFIM consistently achieved higher test accuracy at round 70 than first-order baselines:

Privacy Regime	DP-FedGD Acc (\%)	DP-FedSOFIM Acc (\%)	Gain
No DP	66.41	69.53	+3.12
$\epsilon=10$	66.60	68.97	+2.37
$\epsilon=5$	66.44	68.00	+1.56
$\epsilon=2$	65.59	66.75	+1.16
$\epsilon=1$	64.19	64.61	+0.42
$\epsilon=0.5$	60.43	61.03	+0.60

Convergence analysis showed that for relaxed privacy budgets ($\epsilon \geq 5$), DP-FedSOFIM consistently outpaces first-order methods after early rounds. For tight privacy ($\epsilon \leq 2$), it exhibits lag in initial stages due to the noisy FIM estimate but recovers and overtakes in later rounds.

6. Discussion and Prospective Directions

DP-FedSOFIM maintains the rapid convergence hallmarks of second-order (natural gradient) methods, eliminating the prohibitive memory and communication costs at the client. The server-side preconditioning does not compromise privacy guarantees due to the post-processing property. Empirical results demonstrate that DP-FedSOFIM outperforms first-order alternatives under all tested ($\epsilon,\delta)$ regimes.

However, high-variance FIM estimates under extremely tight privacy constraints ($\epsilon \leq 1$) can cause early instability. Theoretical guarantees currently address only strongly convex objectives typical of shallow models (e.g., linear heads) and do not extend to deep nonconvex neural networks.

Potential research directions include broadening the convergence guarantees to nonconvex settings, incorporating richer curvature structures (such as block-diagonal or K-FAC approximations) while retaining client efficiency, adapting to user-level DP with varying client participation, and automating hyperparameter selection for ($\rho$, $\beta$, $\eta$) in response to privacy budgets and data properties (Nair et al., 14 Jan 2026).