Memory-Centric Operator Fusion Scheme

• Memory-Centric Operator Fusion is a technique that fuses operator chains to maximize data locality and reduce off-chip (DRAM) traffic in deep learning, graph processing, and HPC.
• It systematically leverages DSM-based collectives, advanced thread mapping, and fine-grained tiling to achieve 50–90% reductions in memory traffic and optimize kernel launches.
• Cost models and resource-aware search strategies generate fused kernels, providing 2–4× speedup and significantly reducing redundant memory operations.

A memory-centric operator fusion scheme is an advanced compiler or runtime strategy designed to maximize data locality, minimize off-chip (DRAM) memory traffic, and exploit on-chip memory hierarchies for the efficient execution of operator sequences (chains) in deep learning, graph processing, and high-performance computing workloads. Such schemes systematically identify, validate, and generate fused kernels that retain intermediate results in the highest-bandwidth memories feasible—typically registers, SRAM scratchpad or shared memory, or, in modern hardware, distributed shared memory (DSM). They target the root cause of memory bandwidth bottlenecks: intermediate tensor materialization between operators, which dominates execution time and energy in memory-bound or bandwidth-constrained settings.

1. Memory-Centric Fusion: Objectives, Hardware Context, and Workload Fit

Memory-centric operator fusion directly addresses the growing gap between compute throughput and memory bandwidth in accelerator architectures, including GPUs (e.g., NVIDIA H100 with DSM), neural accelerators, and custom SSM hardware. While earlier fusion strategies were limited by local scratchpad (shared memory) capacity, recent work leverages new on-chip interconnects, DSM, or adaptive tiling strategies to break past this barrier, enabling fusion of much larger, more complex operator subgraphs—such as back-to-back GEMMs in LLMs, SSM updates, or broad elementwise-reduction chains in DNNs (Huang et al., 15 Dec 2025, Luo et al., 26 Aug 2025, Zhang et al., 27 Jun 2025, Geens et al., 24 Apr 2025).

Workloads that particularly benefit include:

• Long chains of compute- or memory-bound operators with moderate-to-large intermediate activation sizes (e.g., transformer attention, GNN message passing, SSM state updates).
• Models whose operational intensity falls below the roofline threshold due to axis size or shape (e.g., shallow reduction dimensions driving GEMM kernels memory-bound) (Zhang et al., 27 Jun 2025).
• Kernel launch- or context-switch-dominated regimes (e.g., inference with numerous small ops).
• Scenarios requiring strict peak memory usage reductions to fit deep or wide models on bandwidth-limited accelerators (edge, mobile, or tightly provisioned datacenter hardware) (Niu et al., 2021).

The central goal is to define a fusion plan that (i) minimizes peak and total global memory traffic, (ii) respects hardware resource constraints (local memory, registers, DSM, PE counts), and (iii) minimizes kernel launch overheads and improves end-to-end parallel efficiency.

2. Domain-Specific Communication and Fusion Abstractions

Recent memory-centric fusion frameworks systematically extend the abstraction set for on-chip communication and operator scheduling:

a) DSM-Based Collectives (FlashFuser, ClusterFusion):

• Define primitives such as dsm_all_exchange, dsm_shuffle, and dsm_reduce_scatter, which expose fine-grained, high-bandwidth communication between clusters of Streaming Multiprocessors via DSM (Huang et al., 15 Dec 2025).
• For LLM inference, primitives like ClusterReduce and ClusterGather enable tree- or butterfly-reduction/gather among thread blocks within an SM cluster, hiding global memory and kernel boundaries and reducing per-fusion group DRAM traffic by 50–80% (Luo et al., 26 Aug 2025).

b) Advanced Thread Mapping (GNN, DNNFusion):

• Unified thread-mapping functions allow fusing both vertex- and edge-centric operators in GNNs, ensuring all intermediate data can be kept in registers or shared memory and never spill to DRAM (Zhang et al., 2021).
• Mapping-type analysis enables legal and profitable fusion only for compatible combinations (one-to-one, many-to-one, one-to-many, reorganize, shuffle) and ensures correctness when fusing complex DNN operator DAGs (Niu et al., 2021).

c) Fine-Grained Tiling and DAG Analysis:

• Express arbitrary tiling and fusion opportunities as high-level expressions over nested and sequential loops, with exhaustive, but pruned, enumeration—allowing for memory-bound, compute-intensive operator chains to be fused at maximal tile size before exceeding shared or DSM capacity (Zhang et al., 27 Jun 2025).
• DAG-based schedule analysis tracks memory access and dependencies to automatically relocate loads/stores and eliminate redundant memory operations (Zhang et al., 27 Jun 2025).

3. Cost Models, Fusion Plan Search, and Capacity Constraints

All state-of-the-art schemes rely on multi-level cost models to drive the selection of fusion groups:

• Analytical Models:
  • Formulate total cost as $$\mathrm{Cost} = C_{\text{compute}} + C_{\text{mem}} + C_{\text{DSM}}$$, decomposing per-level data movement (register, SMEM, DSM, global) by bandwidth/latency, and explicitly modeling DSM/cluster bandwidth tradeoffs (Huang et al., 15 Dec 2025).
  • For SSM accelerators, quantify the minimal per-tile on-chip memory as a function of fusion granularity and tile count in each dimension, automatically splitting further (mem-aware) to not exceed area or SRAM constraints (Geens et al., 24 Apr 2025).
• Resource-Aware Search and Pruning:
  • Enumerate all candidate loop schedules, tile sizes, cluster shapes, and parallel mappings, aggressively prune plans violating tile divisibility, spilling, cluster/DSM limits, or unacceptable padding/occupancy (Zhang et al., 27 Jun 2025, Huang et al., 15 Dec 2025).
  • Search employs either analytical cost models and DSM-aware pruning (FlashFuser), beam or dynamic programming guided by delta-evaluators (FusionStitching), or evolutionary heuristics with runtime feedback (MCFuser).
  • For complex cases (Blockbuster), selection solves a knapsack-like dynamic program over block-program subgraphs, maximizing DRAM savings subject to local-memory fit (Dekel, 29 Apr 2025).
• Invalid Group Diagnosis:
  • Integrate graph-explanation techniques (GET: GNNExplainer/PGExplainer/RGExplainer) to diagnose and split invalid fusion groups (exceeding buffer cap), recursively partitioning the operator DAG at cut edges most responsible for violating memory constraints (Mills et al., 2024).

4. Fusion Code Generation, On-Chip Buffering, and Communication Scheduling

The generated kernels combine the following:

• Multi-level Composition Modes:
  • Thread-level: keep scalar intermediates in registers.
  • Warp-level: intra-warp shuffles pass reductions/broadcasts to consumers without global traffic (Zheng et al., 2020).
  • Block-level: temporaries or partial results in shared memory, with dominance-tree analysis for reuse and liveness-based deallocation (Long et al., 2019).
  • DSM/Cluster-level: buffer full or partial tiles in cluster DSM buffers, orchestrate collective exchange for reduction, shuffle, or gather to maintain all-tile locality (Huang et al., 15 Dec 2025, Luo et al., 26 Aug 2025).
• Communication Scheduling:
  • Overlap compute rounds with DSMEM transfers, using binary-tree reduction or gather algorithms to manage DSM buffer slices across blocks, synchronized by hardware-level primitives (Luo et al., 26 Aug 2025).
  • For SSMs, double-buffering states and overlapping compute/offload ensure no idle time during tile transitions (Geens et al., 24 Apr 2025).
• Buffer Allocation and Lifetime Tracking:
  • Shared-memory allocations are minimized by lifetime-overlap analysis; only tensors spanning intra-kernel group boundaries are stored (Long et al., 2019).
  • Recomputation strategies: for backward/gradient passes, store only O(|V|) summaries, reconstruct |E|-sized intermediates online, reducing peak training memory in GNNs by up to 7.7× (Zhang et al., 2021).
• Algebraic Correction for Fused Reductions:
  • For complex loop-carried dependencies (attention softmax + weighted sum), employ recurrence-based online softmax or normalization algorithms to merge otherwise separate reduction passes, avoiding materialization and ensuring bitwise or numerically stable equivalence (Zhao et al., 9 Oct 2025).

5. Practical Efficacy and Quantitative Outcomes

Empirical studies across state-of-the-art systems consistently demonstrate:

• Global memory traffic reduction:
  • FlashFuser: 58% reduction on NVIDIA H100 over unfused PyTorch (Huang et al., 15 Dec 2025);
  • ClusterFusion: 50–80% DRAM cut on LLM decoding (Luo et al., 26 Aug 2025);
  • DNNFusion: 60–70% average cut on intermediate result size in large models (Niu et al., 2021);
  • Blockbuster: 87–90% for multi-stage attention/norm kernels (Dekel, 29 Apr 2025).
• Kernel speedup and launch reduction:
  • FlashFuser delivers 3.3× kernel speedup (vs cuBLAS), 1.24× E2E throughput gain (Huang et al., 15 Dec 2025).
  • Neptune achieves 1.35× mean speedup and up to 2.1× in memory traffic over optimized Triton/TVM baselines, fusing attention in a single kernel (Zhao et al., 9 Oct 2025).
  • FusionStitching and DNNFusion show 2–4× speedup and 3–8× kernel count reduction (Zheng et al., 2020, Niu et al., 2021).
• Buffer capacity and area scaling:
  • SSM Mem-Aware fusion tracks on-chip SRAM use to within a programmable fraction of the total, enabling area reduction from 24 MiB to 10.5 MiB SRAM without loss in latency, with throughput scaling nearly 2× for iso-area accelerator designs (Geens et al., 24 Apr 2025).
• Resource-constraint handling:
  • The integration of GNN-based explainability ensures robust partitioning of fusion groups to fit within strict buffer or DSM/SRAM caps, with high rectify rates and Pareto-improved DRAM profile across search strategies (Mills et al., 2024).

6. Algorithmic Strategies for Comprehensive Fusion and Locality

Successful memory-centric fusion leverages a set of rigorous methodologies:

• Exhaustive but Pruned Search:
  • Rules for tile legality, memory cap, chunk size, and dependency correctness permit large fusion spaces, but cost models and access-pattern pruning collapse intractable search spaces to manageable candidate sets (Huang et al., 15 Dec 2025, Zhang et al., 27 Jun 2025, Geens et al., 24 Apr 2025).
• Property-Based DAG Rewriting:
  • Algebraic graph rewriting exploits distributivity, associativity, and commutativity in DNNFusion and Blockbuster to further simplify fusion subgraphs before plan enumeration (Niu et al., 2021, Dekel, 29 Apr 2025).
• Precomputed Fusion Legality Tables:
  • Mapping-type lookup tables enforce operator compatibility, avoiding illegal or unprofitable combinations during candidate expansion (Niu et al., 2021).
• Dynamic Programming and ILP Formulation:
  • Fusion group selection is globally optimal with respect to cost-benefit tradeoffs, with resource caps formalized as linear constraints or budget filters (Long et al., 2019).
• Pattern and Companion Rule Packs:
  • Blockbuster’s two-phase workflow applies memory-saving pattern substitutions, swaps, peels, and duplication alongside fusion, maximizing savings and exposing new fusion opportunities (Dekel, 29 Apr 2025).

7. Generalization, Limitations, and Future Directions

Memory-centric operator fusion has demonstrated broad generalizability:

• To reduction-intense blocks beyond classic MLPs: attention (softmax), SSM recurrent updates, GNN message-passing, normalization layers (LayerNorm, RMSNorm, GroupNorm), and wide elementwise or reduction chains (Zhao et al., 9 Oct 2025, Zhang et al., 2021, Geens et al., 24 Apr 2025).
• Across hardware generations: from classic GPUs (V100, A100) to DSM-enabled architectures (H100) and custom datacenter, edge, and SSM accelerators.
• Integration with graph-explanation methods for hybrid learned + rule-based fusion group selection (Mills et al., 2024).

Practical limitations remain:

• Fusion scope is typically upper-bounded by the available local memory, despite DSM advances.
• Fusion of arbitrary DAGs without algebraic recurrences can be blocked by non-commutative or non-distributive operators.
• Recomputation in the backward pass (GNNs/large DNNs) adds 5–10% compute overhead, though this is justified by the much larger memory savings.
• Fused kernels must still be tuned or searched over for hardware efficiency, as peak occupancy vs. memory tradeoffs vary per architecture and workload.

Future development will likely combine meta-learned search policies, further hardware-software DSM abstractions, and joint optimization of fusion and tiling or operator splitting, aiming to both minimize peak/total memory traffic and maximize capacity utilization in the context of ever-larger and deeper models.