Beyond Exascale: Dataflow Domain Translation on a Cerebras Cluster

Abstract: Simulation of physical systems is essential in many scientific and engineering domains. Commonly used domain decomposition methods are unable to deliver high simulation rate or high utilization in network computing environments. In particular, Exascale systems deliver only a small fraction their peak performance for these workloads. This paper introduces the novel \algorithmpropernoun{} algorithm, designed to overcome these limitations. We apply this method and show simulations running in excess of 1.6 million time steps per second and simulations achieving 84 PFLOP/s. Our implementation can achieve 90\% of peak performance in both single-node and clustered environments. We illustrate the method by applying the shallow-water equations to model a tsunami following an asteroid impact at 460m-resolution on a planetary scale running on a cluster of 64 Cerebras CS-3 systems.

• The paper presents a novel Domain Translation algorithm that cyclically shifts processor-grid associations to effectively hide communication latency in stencil computations.
• It achieves up to 84.7 PFLOPS and over 98% scaling efficiency on 64-node Cerebras CS-3 clusters, redefining performance limits for distributed PDE solvers.
• The implementation supports complex simulations, including Earth-scale tsunami modeling, proving its practical potential for high-fidelity hazard prediction.

Dataflow Domain Translation and Exascale Stencil Computation on Cerebras CS-3 Clusters

Introduction and Motivation

This work addresses the core bottleneck for cluster-scale PDE solvers: the network latency and bandwidth limitations inherent in traditional domain decomposition methods, especially for stencil computations ubiquitous in scientific simulation. Exascale systems typically deliver a small fraction of their architectural peak when applied to such problems due to the frequent synchronizations imposed by halo/ghost zone exchanges. The paper introduces the Domain Translation algorithm, demonstrating its implementation and performance on a cluster of 64 Cerebras Wafer Scale Engine (WSE) CS-3 nodes. The key claims include over 1.6 million timesteps/s and up to 84 PFLOP/s sustained performance on stencil workloads—achieving 90% of peak utilization in both single-node and clustered settings.

Figure 1: A 1D stencil example illustrates the latency bottleneck of static domain decomposition and the latency hiding of domain translation.

The Domain Translation Algorithm

Traditional Decomposition Limitations

Conventional distributed implementations for grid-based PDEs utilize static domain decomposition. Each node updates a local subdomain and exchanges boundary (ghost) data with neighbors. As node speed increases, domain sizes shrink per node, and the per-timestep network latency can become the dominant rate-limiting factor (Figure 1b). Employing larger overlapping regions (ghost zones) hides latency but increases redundant computation and diminishes efficiency rapidly as network latency grows.

Domain Translation Approach

Domain Translation exploits spatial architectures by translating the subdomain assignment cyclically: at each timestep, the association of processors to grid points is shifted by the stencil width $p$. This rotation converts the neighbor communication pattern into a unidirectional dataflow, so each grid point crosses the inter-node boundaries only once per sweep rather than at every timestep, thereby amortizing the network latency over the full subdomain width rather than incurring it at every iteration.

Figure 2: Space-time schematic of domain translation: diagonal subdomain boundaries indicate staggered computation and communication cycles that pipeline network latency.

This approach fundamentally removes the per-timestep dependency on network latency as long as the subdomain size per node exceeds a critical threshold. Thus, only bandwidth and computational throughput determine the simulation rate at scale.

Performance Model and Comparison

Figure 3: Time-stepping rate transitions between latency-bound, bandwidth-bound, and ultimately compute-bound regimes as per-node subdomain size increases.

Where prior methods face an inherent trade-off between timestep rate and ghost-zone cost (with utilization falling polynomially in dimensionality), domain translation offers ideal utilization in the compute-bound regime, fully hiding both latency and bandwidth as long as minimal grid sizes per node are maintained. The paper models the efficiency in terms of per-node grid size ($n$), stencil width ($p$), network latency ($\lambda$), and local point update cost ($c$), showing that

$$\text{compute-bound regime:}\quad n^2 > 2p\lambda/c$$

yields near-optimal scaling.

Spatial Architecture and Cerebras CS-3 Implementation

Cerebras Wafer-Scale Engine Design

The WSE is a 2D array of hundreds of thousands of PEs, each with tightly coupled SRAM and a high-radix Network-on-Chip. All nodes are connected by high-bandwidth links, and switches are configured so that all same-index ports for each wafer map to the same edge switch (Figure 4).

Figure 4: Switch topology for WSE clusters, achieving line-rate communication by port-index mapping.

Specialized routing and mirroring techniques (checkerboarding node roles and code) ensure that physical connections align with logical neighbors, minimizing switch traversals and thereby reducing effective network latency (Figure 5).

Figure 5: Alternating mirrored spatial stencils ensure same-switch entry/exit, optimizing communication latency.

On-wafer routing leverages multiple virtual channels and daisy-chains (horizontal and vertical flows) for high-throughput inter-core message passing (Figure 6).

Figure 6: On-wafer message routing using daisy-chained flows and virtual channels for efficient intra-wafer and off-wafer communication.

Software Framework

A lean ∼1,000-line Tungsten dataflow codebase encapsulates stencil computation, communication, and translation primitives. By abstracting the per-timestep translation, the code remains agnostic to the underlying hardware scaling, efficiently mapping computations to variable node/topology configurations.

Numerical Methods and Benchmarks

Benchmarks: Heat Equation & Shallow Water Equations

The evaluative kernels are canonical for structured grid simulation:

• 5-point/9-point heat equation stencils: scalar parabolic PDEs, 9 and 17 FLOPs per update respectively.
• Shallow Water Equations (SWE): non-linear hyperbolic system, solved with Lax-Wendroff spatial discretization and RK2 time integration, requiring 155 FLOPs per time step per point.

The performance model accurately predicts the transition from IO- to compute-boundedness and asymptotic efficiency values for each kernel.

Experimental Results

Scaling and Efficiency

Exploiting the domain translation method, the CS-3 cluster achieves strong and weak scaling efficiencies above 98%, sustaining near-theoretical performance regardless of node count (up to 64 nodes) and grid size per node.

Figure 7: Weak scaling of 5-point heat equation on 4–60 CS-3 nodes, showing near-constant performance regardless of node count.

The system matches the modeled crossover point from IO-bound to compute-bound behavior, and at scale, achieves 1.32 Flops/cycle per core, corresponding to 66% of peak hardware efficiency.

Figure 8: Measured and predicted FLOPs per core across problem regimes, revealing clear IO- to compute-bound transition.

Record-Setting Performance

Strong scaling on large problems maintains perfect efficiency up to 64 nodes. Under clock-boosted (1.2 GHz, power-aware) execution, the system attains 84.7 PFLOPS on 192B grid points with 64 nodes, a striking result for a stencil computation (Figure 9, right panel). The power efficiency also surpasses typical sparse workloads, attaining 57 GFLOPS/W (cf. Green500’s 72.7 GFLOPS/W, but on dense algebra).

Figure 9: (left) Strong scaling with standard 750 MHz; (right) 84.7 PFLOPS achieved for I/O-bound, large-scale stencil computation at 1.2 GHz on 64 nodes.

Applied Science: Planetary-Scale Tsunami Simulation

The methodology is demonstrated through simulation of Earth-scale tsunami propagation (SWE) following an asteroid impact. The simulation runs at 462m horizontal resolution, utilizing the global GEBCO_2024 grid and attaining planetary coverage within a practical walltime.

Figure 10: Global tsunami wave propagation 14 hours post-impact, simulated at 460m resolution on 64-node cluster.

A further zoom simulates the impact on San Francisco Bay, opening the possibility for rapid, high-fidelity hazard prediction.

Figure 11: Simulation detail of tsunami impact at San Francisco Bay, illustrating localized resolution of global event.

Practical and Theoretical Implications

The presented work establishes a scalable algorithmic and hardware-software stack for latency-agnostic, high-throughput, stencil-based PDE simulation. The domain translation paradigm eliminates the utilization/latency trade-off, previously foundational in distributed physic simulations. The implications are:

• Practical:
  • Near-linear scaling for scientific and engineering workloads formerly bottlenecked by inter-node communication.
  • Enables real-time and uncertainty-quantified studies (e.g., digital twinning, rapid disaster response, ensemble climate/weather forecasting).
  • Step-change in the power efficiency for structured scientific computing (beyond what has been achieved in sparse or unstructured problems).
• Theoretical:
  • Opens the pathway to extend robust stencil solvers to true exascale clusters without algorithmic restructuring.
  • Provides a reference architecture and algorithm for future spatial/DSP architectures and upcoming multi-wafer systems.
  • Shifts the research focus from latency amortization and asynchrony strategies to bandwidth allocation and compute optimization at ultra-large scales.

Looking forward, the authors expect minimal obstacles scaling to even larger clusters. As noted, extension to stacked shallow water and full atmospheric models should exhibit similar efficiency due to nearly identical nearest-neighbor communication patterns. The approach promises substantial advances for coupled Earth system simulations, as full dynamical cores remain communication-bound under current architectures.

Conclusion

The domain translation algorithm implemented on the Cerebras CS-3 cluster delivers a transformative advance in scalable stencil PDE simulation—demonstrating, for the first time, end-to-end cluster efficiency that rivals single-node performance on distributed finite-difference physics problems, with recorded performance of 84.7 PFLOPS and efficiencies over 60%. This work decisively demonstrates that current limitations in cluster scientific computing for structured grids stem not from hardware or physics, but from algorithmic choices; with appropriate dataflow translation, exascale performance is feasible and sustainable for broad classes of simulation codes. The algorithm and system offer a template for the future trajectory of large-scale scientific computing in both hardware and numerical algorithm design.