Fused AR-A2A Algorithm for Hybrid Parallelism

• The paper demonstrates a fused AR-A2A approach that hybridizes tensor and expert parallelism to reduce inter-node communication by 1/n_proc, improving latency.
• The algorithm uses nonblocking intra-node all-reduce and inter-node all-to-all to overlap communication phases and achieve up to 3.8× acceleration in time-to-first-token.
• Empirical results show throughput improvements of up to 50% and significant reductions in communication overhead, validating the method on modern accelerator platforms.

A fused AR-A2A communication algorithm refers to a distributed communication mechanism that overlaps or hybridizes All-Reduce (AR) operations, typically used in tensor parallelism (TP), with All-to-All (A2A) operations, commonly used in expert parallelism (EP), in order to optimize network utilization and reduce overall communication latency in large-scale distributed systems. This approach is particularly impactful for serving Mixture-of-Experts (MoE) models and other workloads heavily dependent on both intra-node and inter-node communication primitives (Zhou et al., 13 Jan 2026).

1. Theoretical Motivation and Hybrid Parallelism

Classical model parallelism for LLMs and MoE architectures relies on either TP (AR) or EP (A2A). Tensor parallelism distributes each tensor along its hidden dimension across $d$ ranks, synchronizing partial results via AR (often as a Reduce-Scatter (RS) followed by All-Gather (AG)), which is communication-efficient on intra-node high-bandwidth interconnects but becomes the bottleneck over slower inter-node links (e.g., InfiniBand, RoCE). Expert parallelism, conversely, dispatches tokens across experts via A2A, which scales better between nodes but can be impaired by load imbalance and large inter-node message volumes (Zhou et al., 13 Jan 2026).

The fused AR-A2A communication algorithm addresses these constraints by combining TP (intra-node AR) and EP (inter-node A2A) in a single hybrid scheme. Within each node, intermediate tensors are sharded using AR before A2A dispatch is performed between nodes, dramatically reducing inter-node communication volume by a factor of $1/n_\mathrm{proc}$, where $n_\mathrm{proc}$ is the number of intra-node devices. This design enables concurrent intra-node aggregation and inter-node routing through nonblocking primitives, effectively overlapping the fast intra-node AR with the slower inter-node A2A (Zhou et al., 13 Jan 2026).

2. Algorithmic Structure and Overlap Mechanisms

Two principal fused routines are integrated into the MoE block’s forward pass:

a) Fused AG–Dispatch

Input tensors ($$X \in \mathbb{R}^{b/d_\mathrm{DP} \times s \times h}$$) are split among $m=n_\mathrm{proc}$ intra-node GPUs along the hidden dimension. Each TP rank $r_\mathrm{TP}$ shards its portion of the tensor further into $n=n_\mathrm{node}$ slices for A2A dispatch. Nonblocking communication (isend/irecv) is used to send slices to matching ranks on remote nodes. Upon receipt of all slices, each rank recovers the full subset required for its experts by performing an intra-node AG. The key is that, except for the initial and final communication rounds, inter-node dispatch and intra-node AG are substantially overlapped in time (Zhou et al., 13 Jan 2026).

b) Fused RS–Combine

After local expert computation, each rank contains a partial result sharded along $h$. Each intra-node rank splits its output into $n_\mathrm{node}$ blocks and exchanges as before with peer ranks using A2A. Reduce-Scatter (RS) operations run in parallel within the node, enabling local aggregation weighted by router probabilities. The process completes with an All-Gather to reconstruct the full output tensor. In this overlap architecture, RS and A2A proceed asynchronously, with only the AG step forming a sequential dependency at the end (Zhou et al., 13 Jan 2026).

3. Mathematical Formulation and Communication Costs

The fused AR-A2A strategy is formalized by the following communication cost models:

• AR/TP for a tensor of size $\text{size}$ across $d$ ranks:

$$\text{AR}(\text{size}, d) = \text{RS}\left( \frac{\text{size}}{d}, d \right) + \text{AG}\left( \frac{\text{size}}{d}, d \right)$$

• A2A/EP for the same tensor:

$$\text{A2A}(\text{size}, d) \propto (d-1) \times \frac{\text{size}}{d}$$

• Without overlap (pure EP combine):

$$\lambda_{\mathrm{EP}} = \text{AR}(bsh, n_\mathrm{proc}) + 2 \times \text{A2A}(bshk, n_\mathrm{node})$$

• With fused algorithm (MixServe):

$$\lambda_{\mathrm{mix}} = \text{AR}(bsh, n_\mathrm{proc}) + \text{AG}\left( \frac{bshk}{n_\mathrm{proc}}, n_\mathrm{proc} \right) + 2 \times \text{A2A}\left( \frac{bshk}{n_\mathrm{proc}}, n_\mathrm{node} \right)$$

The core speedup arises from overlapping intra-node RS and inter-node A2A:

$$T_{\mathrm{overlap}} = \max\{ T_{\mathrm{RS}}, T_{\mathrm{A2A}} \} + T_{\mathrm{AG}}$$

as opposed to the sum in sequential execution. In ideal conditions ($T_{\mathrm{RS}} = T_{\mathrm{A2A}}$), a $2\times$ speedup is achievable for the combine stage (Zhou et al., 13 Jan 2026).

4. System Implementation and Hardware Considerations

All inter-node communications leverage non-blocking isend/irecv and completion tests to maximize asynchrony with local computation and intra-node collectives. Two communication streams per device—one for intra-node (AR/AG), one for inter-node (A2A)—prevent head-of-line blocking. Tensor shards use zero-copy views or pointer slicing to avoid memory transfers. Temporary buffers of up to $O(bsh)$ per rank are utilized for lower-latency aggregation. Thread and rank mapping is optimized so that identical TP ranks across nodes communicate along matching channels (Zhou et al., 13 Jan 2026).

Hardware requirements include multiple GPUs per node (NVLink, HCCS, etc.) for high-bandwidth intra-node collectives and a lower-bandwidth inter-node link (InfiniBand, RoCE). The algorithm is compatible with contemporary accelerators, as evidenced by implementation on Nvidia H20 and Huawei Ascend 910B platforms (Zhou et al., 13 Jan 2026).

5. Empirical Results and Quantitative Improvements

The fused AR-A2A communication design enables substantial performance gains:

• Time-to-First-Token (TTFT): $1.08\times$–$3.80\times$ acceleration (e.g., Qwen3 prefill drops from 600 ms to 157 ms on Ascend910B, a $3.8\times$ reduction).
• Inter-Token Latency (ITL): $1.03\times$–$1.66\times$ speedup (e.g., Qwen3 ITL drops from 134 ms to 81 ms).
• Throughput: Increase by $5.2\%$–$50.3\%$ (e.g., DeepSeek-R1 streaming throughput on H20 increases from 363 tokens/s to 545 tokens/s, $50\%$ improvement).

These improvements are attributable to three primary factors:

1. Shrinking inter-node message size by $1/n_\mathrm{proc}$ through hidden-dimension sharding,
2. Overlapping expensive A2A traffic with inexpensive intra-node collectives,
3. Avoidance of additional memory copies and host launches due to fused routines.

The algorithm was validated on state-of-the-art MoE models (DeepSeek-R1 and Qwen3) and outperformed vLLM and Tutel-based hybrid and expert-parallel implementations (Zhou et al., 13 Jan 2026).

6. Related Approaches and Comparative Techniques

Prior works such as “Optimizing Distributed ML Communication with Fused Computation–Collective Operations” illustrate analogous principles of fusing collective communication, particularly in scale-up and scale-out settings for deep learning workloads (Punniyamurthy et al., 2023). These approaches indicate that persistent kernels, nonblocking GPU-initiated primitives (e.g., ROC_SHMEM/NVSHMEM), and zero-copy buffer management can yield 12–31% lower per-layer latency and a 21% training pass reduction at scale. These findings corroborate the performance of fused AR-A2A designs and reinforce the importance of concurrent arithmetic and network operations for large distributed AI systems.

7. Summary and Significance

The fused AR-A2A communication algorithm constitutes a scalable, high-efficiency method for hybrid parallelism in distributed serving systems, particularly for large MoE and LLM architectures. By concurrent exploitation of intra-node all-reduce and inter-node all-to-all, combined with nonblocking and zero-copy mechanisms, the algorithm achieves measurable reductions in communication overhead, end-to-end latency, and throughput constraints. This approach is corroborated by MixServe’s empirical results and is recognized as foundational for modern, resource-efficient distributed AI inference (Zhou et al., 13 Jan 2026, Punniyamurthy et al., 2023).

Dimension	Classical (TP/EP)	Fused AR–A2A
Inter-node communication	High (AR/EP)	Sharply reduced ($1/n_\mathrm{proc}$)
Overlap of collectives	None (sequential)	Substantial (max concurrency)
Throughput/latency gain	Modest	Up to $3.8\times$ TTFT, 50%+ throughput

This table situates the fused AR-A2A algorithm with respect to classical TP/EP designs using published metrics (Zhou et al., 13 Jan 2026).