Optimizing System Memory Bandwidth with Micron CXL Memory Expansion Modules on Intel Xeon 6 Processors

Abstract: High-Performance Computing (HPC) and AI workloads typically demand substantial memory bandwidth and, to a degree, memory capacity. CXL memory expansion modules, also known as CXL "type-3" devices, enable enhancements in both memory capacity and bandwidth for server systems by utilizing the CXL protocol which runs over the PCIe interfaces of the processor. This paper discusses experimental findings on achieving increased memory bandwidth for HPC and AI workloads using Micron's CXL modules. This is the first study that presents real data experiments utilizing eight CXL E3.S (x8) Micron CZ122 devices on the Intel Xeon 6 processor 6900P (previously codenamed Granite Rapids AP) featuring 128 cores, alongside Micron DDR-5 memory operating at 6400 MT/s on each of the CPU's 12 DRAM channels. The eight CXL memories were set up as a unified NUMA configuration, employing software-based page level interleaving mechanism, available in Linux kernel v6.9+, between DDR5 and CXL memory nodes to improve overall system bandwidth. Memory expansion via CXL boosts read-only bandwidth by 24% and mixed read/write bandwidth by up to 39%. Across HPC and AI workloads, the geometric mean of performance speedups is 24%.