Challenges and Research Directions for Large Language Model Inference Hardware

Abstract: LLM inference is hard. The autoregressive Decode phase of the underlying Transformer model makes LLM inference fundamentally different from training. Exacerbated by recent AI trends, the primary challenges are memory and interconnect rather than compute. To address these challenges, we highlight four architecture research opportunities: High Bandwidth Flash for 10X memory capacity with HBM-like bandwidth; Processing-Near-Memory and 3D memory-logic stacking for high memory bandwidth; and low-latency interconnect to speedup communication. While our focus is datacenter AI, we also review their applicability for mobile devices.

• The paper establishes that memory bandwidth, capacity, and interconnect latency, rather than FLOPS, are the key bottlenecks in LLM inference.
• The paper proposes hardware innovations like High Bandwidth Flash, Processing-Near-Memory, and 3D stacking to overcome these limitations and reduce latency.
• The paper highlights the importance of low-latency interconnect enhancements in multi-chip systems to improve real-time performance and scalability in LLM deployments.

Architecting Efficient Hardware for LLM Inference: Bottlenecks and Prospective Solutions

Background and Current Inference Challenges

Contemporary LLM inference is fundamentally constrained by memory bandwidth, capacity, and interconnect latency—issues accentuated by the sequential Decode phase characteristic of Transformers. Unlike training, which is parallelizable and largely compute-bound, the decode step in autoregressive inference is inherently sequential and memory-intensive. Current accelerators (GPUs, TPUs) are largely repurposed training chips with no dedicated inference-centric design, resulting in inefficiency for handling the bottlenecks introduced by decode-centric workloads and emerging LLM trends such as Mixture-of-Experts (MoE), advanced reasoning, multimodality, and long-context scenarios. Notably, the cost-per-GB and cost-per-GBps of HBM continue to increase, with DRAM scaling slowing, exacerbating memory walls and inflating inference operating costs.

The decode phase is increasingly user-facing, necessitating strict latency constraints. Time-to-first-token and time-to-completion latency metrics are threatened by elongated KV-cache access and network overheads, especially as models scale in size and context lengths. Additionally, inference workloads now require multi-chip systems and frequent inter-chip communication due to massive weights and expansive context. Interconnect latency—rather than bandwidth—is often the bottleneck due to small-message, high-frequency communication patterns.

Research Opportunities: Proposed Hardware Innovations

High Bandwidth Flash (HBF)

HBF architectures offer up to a 10× increase in memory capacity while sustaining HBM-comparable bandwidth by vertically stacking flash dies akin to HBM packaging. Flash capacity scaling empirically outpaces DRAM, with density doubling every three years, providing a sustainable trajectory that DRAM struggles to match. While HBF suffers from limited write endurance and increased read latency (microseconds versus nanoseconds), these drawbacks are mitigated for inference workloads—model weights and seldom-updated context data suit flash characteristics. HBF could enable larger MoEs and context memory footprints, reduce the number of chips per system, and alleviate global supply constraints, but necessitates new software abstractions to address page-based access, optimal memory tiering, and variable endurance.

Processing-Near-Memory (PNM) Versus Processing-In-Memory (PIM)

PNM decouples logic and memory process nodes, enabling high bandwidth by proximity (separate stacked dies) and bypassing the logic area, power, and thermal restrictions that throttle PIM (processor embedded in DRAM die). For LLM inference, PNM facilitates software sharding on much larger granularity—1000× larger memory partitions—thereby minimizing data movement and communication. PIM’s advantages in bandwidth per watt and power efficiency are outweighed by restrictive area, performance, and software substrate constraints. In datacenter-scale inference, PNM provides scalability, cost efficiency, and flexibility, while PIM is more tenable for the restrictive scale and power budget of mobile devices.

3D Memory-Logic Stacking

By using TSVs and vertically stacking compute and memory dies, 3D stacking delivers significant improvements in bandwidth and bandwidth-per-watt relative to 2D designs. Compute-on-HBM-base-die models re-use existing interfaces and cut power consumption, while custom 3D integrations further amplify bandwidth via advanced packaging. The primary challenges are thermal management and interface standardization. Low arithmetic intensity for decode workloads complements low-clock, low-power logic placement. Research frontiers in 3D stacking encompass software mapping for heterogeneous memory hierarchies, communication strategies, and systematic tradeoff analysis for datacenter versus mobile device deployments.

Low-Latency Interconnects

Inference latency is increasingly dominated by interconnect delay rather than raw bandwidth. As decode workloads expand, system sizes and message frequency escalate, amplifying this sensitivity. Innovative topologies (tree, dragonfly, high-dimensional tori) reduce hop counts, directly improving latency at the potential expense of bandwidth. Processing-in-network accelerates common collectives (broadcast, all-reduce), and chip-level optimizations (on-chip SRAM buffering, network-near compute) further reduce end-to-end inference time. Reliability codesigning (standby spares, deterministic fallback on communication timeout) could mitigate node failures and enhance robustness without compromising latency.

Implications and Prospects

The push for specialized hardware for LLM inference necessitates rethinking architecting priorities—from maximizing FLOPS to optimizing memory bandwidth, capacity, and interconnect latency. HBF, PNM, 3D stacking, and purpose-designed low-latency interconnects collectively address the critical decode phase bottlenecks, fostering higher model capacity, lower latency, better energy efficiency, and reduced TCO/CO2e. Adoption of HBF could unlock model scales beyond current HBM-only architectures, while PNM and 3D stacking promise more scalable and cost-efficient LLM systems. Low-latency interconnects could render multi-chip deployments practical for real-time user-side AI. These approaches, if synergistically integrated, could fundamentally shift the hardware-software codesign paradigm for AI inference.

Mobile devices represent a distinct regime where energy and footprint constraints make PIM and constrained forms of the above techniques more relevant, especially as on-device LLMs proliferate.

Progress in these domains will require new simulation frameworks—roofline-based with detailed memory and sharding tracking—to inform better cost, power, and capacity tradeoffs. Moreover, algorithmic advances (e.g., non-autoregressive generation, improved batching, and software-hardware codesign) could further relax hardware constraints.

Conclusion

LLM inference is no longer fundamentally compute-bound; memory bandwidth, capacity, and network latency are now the decisive factors. The trajectory for scalable, cost-effective, and energy-conscious AI will be determined by innovations in HBF, near-memory processing, 3D stacking, and low-latency interconnects, superseding FLOPS-centric design prevalent in current datacenter accelerators. These research directions offer concrete pathways for resolving escalating economic and technical barriers to AI deployment at both datacenter and mobile scales, and invite a refocused collaborative effort from the computer architecture research community.