SiFAR: Synchronization-Free All-Reduce for Low-Latency LLM Inference

Abstract: The rise of reasoning models and agentic systems has made LLM token-generation latency a key bottleneck. Unlike chatbots, whose latency gains saturate at human reading speed, these systems generate intermediate reasoning tokens not consumed by humans. Thus, per-token latency directly determines end-to-end response time. Low-latency inference uses minimal batching, making token generation bandwidth-bound. Tensor Parallelism addresses this by sharding model weights across GPUs and loading them in parallel. However, scaling to more GPUs introduces All-Reduce overheads that grow with GPU count. Removing All-Reduce improves token throughput by 43% for Llama-3.1-8B on 8 H200 GPUs. We propose Synchronization-Free All-Reduce (SiFAR), which reduces synchronization overhead during low-latency inference. Existing oneshot and twoshot algorithms incur overheads from barriers before and after communication. First, we find that the bottom barrier in oneshot enforces a WAW dependency and eliminate it by co-designing communication and model execution to enable dual buffering. However, oneshot scales poorly with GPU count. Twoshot performs better at higher TP degrees but incurs an unavoidable bottom barrier. To overcome this, we leverage in-switch reduction in modern switches. We propose redundant pull, where each GPU reduces the full All-Reduce payload at the switch. This improves oneshot scalability while retaining its no-bottom-barrier advantage. Finally, to reduce top-barrier overhead, we observe that each decode step issues multiple All-Reduce operations, keeping GPUs tightly synchronized after the first. We therefore propose speculative reduction, which initiates data transfer before the top barrier and ensures correctness via lightweight validation. SiFAR reduces All-Reduce latency by up to 52% and improves end-to-end throughput by 18.6% for Llama-3.1-8B and 13.1% for Qwen3.5-397B-17B at TP=8.

• The paper introduces a synchronization-free All-Reduce design that reduces latency by up to 52% through a co-design of dual buffering, redundant pull, and speculative reduction.
• Dual buffering eliminates write-after-write hazards and redundant pull leverages in-switch reduction to significantly decrease per-GPU data transfer.
• Speculative reduction minimizes top-barrier synchronization overhead while ensuring correctness, thereby boosting throughput for low-batch LLM inference.

SiFAR: Synchronization-Free All-Reduce for Low-Latency LLM Inference

Motivation and Problem Statement

The shift towards agentic systems and high-resolution multi-token reasoning models has placed stringent demands on LLM inference latency, particularly the per-token generation time (TPOT). While batching is traditionally used to amortize memory access costs, in fast reasoning tasks, minimal batching is necessary, resulting in memory bandwidth-bound token generation. Tensor Parallelism (TP) partially solves this by sharding model weights across GPUs, leveraging aggregate memory bandwidth. However, scaling TP introduces substantial communication overheads, predominantly in All-Reduce operations, which rapidly become the limiting factor in low-latency settings. Synchronization barriers before and after data transfer account for up to 62% of All-Reduce latency in small payload scenarios typical of high interactivity and reasoning-centric workloads.

Figure 1: Insights into synchronization bottlenecks in existing All-Reduce variants and the three SiFAR optimizations: dual buffering removes the bottom barrier, redundant pull leverages in-switch reduction, and speculative reduction minimizes top-barrier overhead.

All-Reduce Algorithmic and System Analysis

Conventional LLM inference systems (vLLM, SGLang, TRT-LLM) utilize either oneshot or twoshot All-Reduce variants, depending on payload size and degree of TP. Oneshot entails each GPU pulling the entire buffer from all peers and reducing locally, introducing heavy synchronization at both ends to prevent WAW hazards. Twoshot splits the process into reduce-scatter and broadcast phases, maintaining constant data transfer but reinforcing an uneliminable bottom barrier due to RAW dependencies. At scale (i.e., TP=8), oneshot's data transfer grows linearly, whereas twoshot's synchronization costs dominate as payloads shrink.

Figure 2: Oneshot All-Reduce incurs top and bottom synchronization barriers ensuring correctness but at significant latency cost.

Figure 3: Twoshot All-Reduce limits data movement but retains costly barriers around reduce-scatter and broadcast phases.

Empirical latency breakdowns on H200 hardware confirm that, for payloads 8–32 KB, barriers alone contribute between 32–62% of total latency (Figure 4). This necessitates algorithmic and execution co-design targeting synchronization overhead.

Figure 4: Synchronization overhead is consistently the main contributor to All-Reduce latency across all TP degrees and payload sizes.

SiFAR: Synchronization-Free All-Reduce

SiFAR's core design combines three orthogonal optimizations:

Dual Buffering: By alternating between two input buffers across successive All-Reduce calls, the system eliminates WAW hazards and obviates the bottom barrier. This co-design between the communication layer and the inference engine (e.g., Megakernels) enables barrier-free transition without restructuring memory management.

Figure 5: Dual buffering ensures buffer reuse is delayed until safe, eliminating the write-after-write dependency.

Redundant Pull: Leveraging in-switch computation (multimem.ld_reduce), SiFAR enables each GPU to issue reduction over the full payload, rather than only a disjoint partition (as in reduce-scatter). This approach reduces payload data received per GPU from $K \times (N-1)$ to $K$, matching twoshot's data transfer but without the associated bottom barrier.

Figure 6: Latency comparison of reduce-scatter and redundant pull demonstrates near-constant overhead for small payloads in redundant pull, motivating its use in SiFAR.

Figure 7: Redundant pull design: switches aggregate reduction for full payload for each GPU, radically decreasing per-GPU data receipt.

Figure 8: Redundant pull with dual buffering outperforms both oneshot and twoshot by combining low data transfer with barrier elimination.

Speculative Reduction: The top barrier remains necessary to ensure RAW correctness but often incurs only flag-exchange overhead as GPUs are tightly synchronized post-first All-Reduce (via fused CUDA Graphs/Megakernels). SiFAR speculates that peers have produced valid partials, begins fetch immediately, and validates correctness by reducing a flag buffer alongside the payload. Mis-speculation is detected and retried, yet incurs negligible tail latency in practice due to hardware monotonicity guarantees and token-by-token workload regularity.

Figure 9: Profiling reveals flag-exchange, not GPU divergence, as the dominant cost in top barrier synchronization.

Figure 10: SiFAR design: redundant pull, dual buffering, and speculative reduction work in concert to remove synchronization bottlenecks.

Figure 11: Lightweight verification logic guarantees correctness despite speculative data transfer, retrying only on rare mis-speculations.

Empirical Results and Numerical Claims

Standalone All-Reduce latency benchmarks across TP=2, 4, 8 (payloads 8–32 KB) indicate that SiFAR consistently outperforms all baselines, achieving up to 52% latency reduction at TP=8 compared to conventional twoshot/oneshot (Figure 12).

Figure 12: SiFAR achieves dominant latency reductions against oneshot, twoshot, and other optimized routines, most notably at high TP degrees.

Compared against NCCL 2.30 auto-selection, SiFAR routinely outstrips even custom low-latency kernels, confirming the pervasiveness of synchronization overhead in the status quo and the necessity of SiFAR’s co-designed approach (Figure 13).

End-to-end throughput tests on Llama-3.1-8B and Qwen3.5-397B-17B display throughput increases of up to 18.6% at TP=8, with performance gains increasing at higher TP degrees, and substantial improvement retained across varied input sequence lengths and batch sizes (Figures 17, 18, 19).

Figure 14: Throughput gains with SiFAR escalate as TP increases, maximizing at TP=8.

Figure 15: SiFAR’s benefits persist across input sequence lengths, with improvements tapering on attention-dominated workloads.

Mis-speculation rates grow with TP but their penalty is marginal—maximum 1.4% lost performance—and tail latency is not inflated (Figures 24, 21, 25).

Figure 16: Mis-speculation rate is monotonic with TP but negligibly affects throughput.

Figure 17: SiFAR reliably reduces median and tail latencies, with mis-speculation retries not introducing significant noise.

Figure 18: Average latency remains below baseline even under enforced retries, confirming practical robustness of speculative reduction.

Ablation (Figure 19) demonstrates meaningful contributions from all three optimizations, with their complementary effect most pronounced at large TP degrees.

Practical and Theoretical Implications

SiFAR’s suite of optimizations disrupts the communication-vs-computation tradeoff in LLM inference. By attacking synchronization overhead directly, rather than merely hiding it through batching or compute-communication fusion, SiFAR enables latency reductions that cannot be achieved through traditional overlapping or microbatching schemes (Figure 20). This is crucial in the context of agentic and automated reasoning at scale where per-token latency compounds, and human-read thresholds no longer limit performance requirements.

On the theoretical front, SiFAR’s approach validates the principle that synchronization-avoidance is best achieved through tightly co-designed algorithm-architecture pairs, leveraging hardware primitives (in-switch reduction, symmetric memory) and work profile regularity. The speculative reduction strategy raises interesting questions about monotonicity assumptions in GPU execution, correctness validation under speculative communication, and future applicability to straggler-prone distributed environments.

Hardware and software architects may push for tighter coupling between communication APIs and inference engines, specialized support for dual buffering at the framework level, and expansion of switch-side computation primitives.

Future Directions

• Generalization: The speculative reduction approach could extend to broader classes of collective communication where application-level monotonicity and regularity are prevalent.
• Hardware Integration: Further switch-side compute primitives and memory models should be exposed to software, possibly enabling push-based reduction with tighter mesh architectures.
• Multi-node and Heterogeneous Topologies: SiFAR’s principles remain to be validated in multi-node environments, NIC-constrained topologies, and under heterogeneous compute/memory regimes.
• Automated Buffering: Integration at the framework level (e.g., automatic dual buffering in PyTorch/vLLM/SGLang) without manual intervention would enable mainstream adoption.

Conclusion

SiFAR offers a synchronization-free All-Reduce mechanism for low-latency LLM inference, harnessing dual buffering, redundant pull, and speculative reduction to systematically eliminate synchronization overheads. This approach yields up to 52% reduction in All-Reduce latency and boosts end-to-end throughput for major LLM architectures, with improvements most significant at high TP degrees and low batch sizes. SiFAR demonstrates that direct attack on synchronization, rather than indirect overlap, is essential for sub-millisecond token generation in reasoning-centric LLM serving, setting the stage for future co-designed algorithmic and hardware advances (2607.08973).