BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal

Abstract: We present BaseRT, a native Metal inference runtime for LLMs on Apple Silicon, and report the highest inference throughput on this hardware to date. Existing runtimes, including llama.cpp and MLX-based frameworks, incur overhead from abstractions not designed for Metal's execution model or Apple Silicon's unified memory topology. By building natively on Metal with chip-specific kernel fusion, unified memory-aware optimisation, and custom dispatch logic, BaseRT recovers performance that framework-based approaches leave on the table. BaseRT supports a wide range of model families across eight quantisation formats (Q2 to FP16) on all Apple M-series devices. In this paper, we evaluate the Qwen3, Llama 3.2, and Gemma 4 families at Q4 and Q8 quantisation on M3 and M4 Pro devices. BaseRT achieves up to 1.56x higher decode throughput than llama.cpp and up to 1.35x higher than MLX, with substantially larger margins on prefill for mixture-of-experts models, delivering consistent best-in-class throughput from sub-1B to 30B parameter models. These results establish Apple Silicon as a more capable inference platform than previously reported, with direct implications for the emerging edge inference paradigm: as privacy requirements, latency constraints, and cloud cost pressures drive inference toward on-device deployment, performance-optimised local runtimes are a critical enabling layer for this transition. BaseRT is publicly available at https://github.com/basecompute/baseRT

• The paper introduces BaseRT, a C++ runtime that uses Apple’s Metal API with a data-driven architecture and zero-allocation decode loop to maximize LLM inference throughput.
• It demonstrates optimized kernel fusion and custom scheduling techniques that yield up to 1.56× higher token-generation throughput on M4 Pro compared to conventional frameworks.
• The work highlights the feasibility of private, high-performance edge LLM deployment on Apple Silicon, reducing reliance on cloud resources and mitigating latency and privacy risks.

BaseRT: Native Metal LLM Inference on Apple Silicon

Motivation and Edge Inference Paradigm

The acceleration of LLM deployment across industry is increasing inference demands at a rate outpacing the growth of centralized cloud resources. Capacity projections suggest that by 2030, AI inference will comprise 40% of global datacenter compute, making cloud-based scaling unsustainable for latency, privacy, and economic reasons. Concurrently, the hardware prerequisites for performant on-device LLMs are now met at scale, especially on Apple Silicon—whose unified memory architecture and high-bandwidth GPU present a uniquely favorable topology for running LLMs locally. The practical implications include eliminating cloud-jurisdiction data leakage risks, removing interactive latency inflation from network and queueing, and reducing cloud costs to the fixed amortization of local hardware.

Architectural Design of BaseRT

BaseRT introduces a C++ runtime targeting Apple's Metal GPU API natively—eschewing intermediate frameworks such as MLX, llama.cpp, or CoreML. Unlike prior approaches, BaseRT is architected around four principles that directly minimize sources of previously unaddressed overhead:

• Data-Driven Architecture Descriptors: Model architecture variations are abstracted as data, not control-flow, enabling a static, non-branching, and high-throughput decode path agnostic to model family.
• Zero-Allocation Decode Loop: All inference-phase buffers, including KV cache, are strictly pre-allocated at model load with layouts optimizing for GPU coalesced access. No runtime allocation or memory management occurs on the hot path.
• Chip-Adaptive Kernel Fusion: Performance-critical operator sequences (e.g., within the attention and feed-forward paths) are fused at the kernel level, tailored per M-series GPU generation, minimizing the frequency of kernel launches and memory round-trips. All quantization formats (Q2–Q8, BF16, FP16) use specialized fused kernels with on-the-fly dequantization.
• Custom Prefill and Decode Scheduling: Autoregressive decode leverages amortized CPU–GPU dispatch, reducing synchronization and command buffer overheads. Prefill implements GEMM using Metal simdgroup_intrinsics and tile-geometries tuned to sequence length and available scratch.

Empirical Evaluation and Numerical Results

Extensive benchmarking on Apple M3 and M4 Pro (covering both dense and mixture-of-experts models within the Llama 3.2, Qwen3, and Gemma 4 families) demonstrates state-of-the-art performance across quantization granularities:

• Decode Throughput: On M4 Pro, BaseRT achieves up to 1.56× higher token-generation throughput compared to llama.cpp and up to 1.35× versus MLX on the same model weights and quantization. Gains are most pronounced on small, dense models—Qwen3-0.6B and Llama-3.2-1B—where BaseRT delivers 35–56% higher throughput. With ~1.04–1.07× on 30B MoE models, results confirm the kernel fusion and custom dispatch minimize fixed per-token overheads that dominate small-model latency.
• Prefill Throughput: For mixture-of-experts models (Gemma-4-26B-A4B, Qwen3-30B-A3B), BaseRT outperforms llama.cpp by up to 1.81× at short prompts and maintains a substantial advantage across prompt lengths. On dense models, all engines reach near-theoretical GEMM throughput parity, verifying that prefill is compute-bound once framework overhead is removed.
• Cross-Generation Consistency: On M3 base, relative speedups over MLX and llama.cpp remain consistent (1.13–1.34×), showing robustness to hardware scaling.
• uzu Comparison: Against uzu, another native Metal runtime, BaseRT leads in token-generation throughput for 5 of 6 configurations (up to 1.19×), confirming the unique benefit of its decode scheduling and kernel fusion. On prefill, uzu is marginally superior, likely due to leveraging MPSGraph and ANE for high-concurrency GEMM, an avenue unexplored by BaseRT.

Technical and Practical Implications

BaseRT demonstrates the degree to which abstraction in mainstream Apple Silicon inference stacks leaves hardware capabilities unexploited—empirically establishing that previous throughput ceilings were not intrinsic to the platform but to software design. The implications for the edge inference paradigm are significant:

• Feasibility of Private, High-Performance LLM Deployment: With BaseRT, locally run 30B MoE models are viable on Apple hardware at practical throughputs, reducing reliance on remote APIs and mitigating both privacy and availability risks in enterprise and edge agentic scenarios.
• Unified Memory as a Performance Lever: The unified CPU–GPU memory and high-bandwidth Metal command buffer model enable optimizations (e.g., direct dequantization fusion, kernel sequence amalgamation) that are inaccessible to cross-platform or array-based runtimes. Exposing and exploiting these features is critical for future hardware-aware AI software stacks.

Limitations and Future Developments

Current limitations include lack of support for multi-device inference (no tensor parallelism, no continuous batching, no speculative decoding), and exclusive dependence on Metal, limiting deployments to Apple environments. Future work includes extending architectural coverage to multimodal and SSMs, adding CUDA and Vulkan backends for cross-platform deployment, and implementing advanced serving optimizations for latency-sensitive and multi-request workloads.

Conclusion

BaseRT sets a new benchmark for LLM inference on Apple Silicon by extracting maximal throughput via native Metal programming, kernel fusion, and architecture-neutral hot-path design. Its results reposition Apple Silicon as a top-tier edge inference target, and its architectural paradigm points toward the necessity of hardware-native, framework-free runtimes to realize the promised gains of local AI deployment. The public release of BaseRT invites further adoption and comparative evaluation against leading inference engines.