PyFR: An Open Source Framework for Solving Advection-Diffusion Type Problems on Streaming Architectures using the Flux Reconstruction Approach

Abstract: High-order numerical methods for unstructured grids combine the superior accuracy of high-order spectral or finite difference methods with the geometric flexibility of low-order finite volume or finite element schemes. The Flux Reconstruction (FR) approach unifies various high-order schemes for unstructured grids within a single framework. Additionally, the FR approach exhibits a significant degree of element locality, and is thus able to run efficiently on modern streaming architectures, such as Graphical Processing Units (GPUs). The aforementioned properties of FR mean it offers a promising route to performing affordable, and hence industrially relevant, scale-resolving simulations of hitherto intractable unsteady flows within the vicinity of real-world engineering geometries. In this paper we present PyFR, an open-source Python based framework for solving advection-diffusion type problems on streaming architectures using the FR approach. The framework is designed to solve a range of governing systems on mixed unstructured grids containing various element types. It is also designed to target a range of hardware platforms via use of an in-built domain specific language based on the Mako templating engine. The current release of PyFR is able to solve the compressible Euler and Navier-Stokes equations on grids of quadrilateral and triangular elements in two dimensions, and hexahedral elements in three dimensions, targeting clusters of CPUs, and NVIDIA GPUs. Results are presented for various benchmark flow problems, single-node performance is discussed, and scalability of the code is demonstrated on up to 104 NVIDIA M2090 GPUs. The software is freely available under a 3-Clause New Style BSD license (see www.pyfr.org).

• The paper introduces PyFR, an open-source computational fluid dynamics framework that employs high-order Flux Reconstruction methods on unstructured grids for simulating advection-diffusion problems.
• Validation shows PyFR achieves high accuracy, including super accuracy for Euler vortex simulations, and demonstrates strong scalability on GPUs.
• PyFR leverages symbolic manipulation for runtime code generation and targets multiple hardware architectures like GPUs and CPUs, enhancing adaptability and efficiency.

An Evaluation of PyFR: A High-Order Computational Framework for Advection-Diffusion Simulation

The paper "PyFR: An Open Source Framework for Solving Advection-Diffusion Type Problems on Streaming Architectures using the Flux Reconstruction Approach" introduces PyFR, a computational fluid dynamics (CFD) framework. Designed at Imperial College London, PyFR employs high-order Flux Reconstruction (FR) methods for simulating complex advection-diffusion phenomena over unstructured grids. This framework targets streaming architectures such as GPUs, harmonizing high-order numerical accuracy with computational efficiency.

Core Contributions and Features

PyFR is structured to solve the compressible Euler and Navier-Stokes equations on mixed unstructured grids, employing quadrilateral and triangular elements in 2D and hexahedral elements in 3D. Key features include:

• High-Order Flux Reconstruction: PyFR adopts FR methods that offer compact stencil sizes and high order accuracy, which are essential for complex geometries where classical methods fall short.
• Multi-Hardware Compatibility: The framework is designed for CPUs and NVIDIA GPUs, utilizing Python, CUDA, and C as implementation languages. The use of MPI and OpenMP ensures that PyFR is both multi-GPU and multi-CPU aware.
• Symbolic Manipulation: The software extensively uses symbolic manipulation for runtime code generation, making it adaptable and powerful, particularly when combined with Python's dynamic capabilities.

Numerical Performance and Validation

The paper describes the validation and performance evaluation of PyFR on benchmark problems ensuring its reliability and accuracy:

• Super Accuracy in Euler Vortex Simulations: For the Euler equations, PyFR achieves super accuracy, a characteristic of FR methods noted by other schemes such as DG and SD. This is critical in high-fidelity simulations where traditional methods may underperform.
• Couette Flow and Complex 3D Simulations: Validation against the Couette flow problem demonstrated PyFR's ability to predict steady-state solutions with expected orders of accuracy. In complex 3D scenarios like flow over a cylinder, the framework exhibits stability and accuracy in turbulence simulations.

Performance and Scalability

PyFR's performance on high-performance computing clusters has been tested, with scalability demonstrated up to 104 NVIDIA M2090 GPUs:

• GPU Utilization: Single GPU tests reveal an efficient utilization of resources, with compute-bound kernels capturing a significant portion of the available FLOP/s and memory bandwidth.
• Scalability: Weak scalability metrics indicate PyFR's capability to handle large problem sizes while maintaining runtime efficiency. In strong scalability tests, a consistent improvement in speedup is observed with increased GPU counts.

Implications and Future Directions

The implications of using PyFR extend beyond enhanced simulation precision. Its open-source nature and adaptability to various hardware architectures make it a significant tool for researchers exploring fluid dynamics problems in complex domains. The symbolic manipulation and runtime code generation pave the way for future improvements, including the potential development of bespoke kernels tailored to outperform general GEMM operations in memory bandwidth and arithmetic intensity.

Further developments may focus on expanding the range of physical models and incorporating new high-order schemes. The flexibility offered by Python and the ability to target different architectures may aid in adapting to evolving hardware trends, ensuring PyFR remains a cutting-edge tool for scale-resolving simulations in industrially relevant contexts.

In summary, PyFR represents a comprehensive effort to marry accuracy, adaptability, and efficiency in CFD simulations, solidifying its place in the landscape of modern computational methodologies for fluid dynamics. The framework exemplifies the power of high-order schemes and innovative computing architectures in advancing the fidelity of numerical simulations.