ProfInfer: An eBPF-based Fine-Grained LLM Inference Profiler

Abstract: As LLMs move from research to production, understanding how inference engines behave in real time has become both essential and elusive. Unlike general-purpose engines such as ONNX Runtime, today's LLM inference systems offer little operator-level visibility, leaving developers blind to where time and resources go. Even basic questions -- is this workload memory-bound or compute-bound? -- often remain unanswered. To close this gap, we develop a fine-grained, non-intrusive profiling framework for modern LLM inference engines, exemplified by llama-cpp but applicable to similar runtime architectures. Built on extended Berkeley Packet Filter (eBPF) technology, our system dynamically attaches probes to runtime functions across multiple layers -- without modifying or recompiling the source. It transforms collected traces into rich visualizations of operators, graphs, timelines, and hardware counter trends, exposing how dense inference, Mixture-of-Experts routing, and operator offloading behave in practice. With less than 4% runtime overhead and high profiling fidelity, our framework makes LLM inference both transparent and diagnosable, turning performance profiling into a practical tool for optimization, scheduling, and resource-aware deployment.

• The paper introduces a non-intrusive eBPF-based profiling tool that captures fine-grained operator details in LLM inferences.
• The methodology dynamically attaches kernel-space probes to trace token, graph, and operator levels, achieving less than 4% runtime overhead.
• The profiler reveals that MatMul operations dominate inference time and highlights memory bandwidth constraints that drive targeted system optimizations.

ProfInfer: An eBPF-based Fine-Grained LLM Inference Profiler

Motivation and Problem Statement

As LLM deployment shifts to mobile and edge scenarios, runtime observability and performance transparency have become critical, yet conventional inference frameworks and system profilers generally provide only coarse-grained metrics, require intrusive instrumentation, or lack semantic alignment with the operator-level execution dynamics relevant for LLMs. ProfInfer addresses this gap by leveraging eBPF for non-intrusive, fine-grained tracing and operator profiling specifically tailored for LLM inference engines such as llama.cpp, with extensibility to diverse hardware backends and dynamic execution paradigms.

Figure 1: High-level design of ProfInfer, illustrating eBPF-based tracers and trace analyzers.

ProfInfer Architecture and Implementation

ProfInfer operates by dynamically attaching eBPF-based probes at multiple layers of the LLM inference pipeline without source-level modifications or recompilation. The framework is divided into user-space and kernel-space components:

• User-space: Responsible for probe attachment, buffer management, conditional flagging for granular tracing, and asynchronous event processing. Enables selective activation of tracing features (e.g., tensor dimension collection, PMC sampling) to balance overhead and insight depth.
• Kernel-space: Implements probe handlers for targeted runtime functions, extracting thread, CPU, timestamp, and operator arguments for comprehensive execution context reconstruction. Ring and perf buffers are used for efficient data collection, with negligible impact on system load.

  Figure 2: Operational overview of ProfInfer showing probe distribution and metric flows across layers.

ProfInfer targets multiple levels of analysis:

• Token-level: Captures per-token timings (TTFT, TPOT) to characterize throughput and dynamically adjust tracing resolution or resource allocation.
• Graph-level: Traces computational graphs, exploiting backend scheduling logic in frameworks such as GGML (layer partitioning, heterogeneous backend selection).
• Operator-level: Parses run-time operator invocations with detailed tensor metadata and topological context, enabling profiling of matrix multiplications, attention mechanisms, softmax, and MoE routing.
• Scheduler-level: Hooks into Linux scheduler tracepoints (sched_switch, sched_wakeup) to correlate thread states and identify performance degradation due to workload interference.

Integration with hardware PMCs provides metrics on cache refills, memory access patterns, CPU cycles, and disk I/O efficiency, facilitating diagnosis of compute- or memory-bound bottlenecks at operator granularity.

Visualization and Analytics

ProfInfer generates three primary analytic views:

Profiled DAG (ProfDAG)

By reconstructing the computational DAG from traced operator events, ProfDAG exposes architecture-specific workflows, execution order, and metric overlays (time, memory bandwidth, etc.) at operator resolution.

Figure 3: Graphical view of an FFN for LLaMA-3.2-1B, with profiling annotated by memory bandwidth.

Further, the tool can compare self-attention implementations across model families (LLaMA, Qwen, Gemma), revealing critical differences in operator composition and performance characteristics.

Figure 4: Architecture differences in the self-attention of different LLMs visualized via ProfDAG.

Timeline (ProfTime)

Temporal profiling uses Chrome Trace-compatible timeline views to represent sequential and parallel execution of operators, backend partitioning, and multi-thread scheduling patterns.

Figure 5: Timeline view in Perfetto, emphasizing intra-operator parallelism for LLaMA3.2-1B-F16.

ProfInfer demonstrates that operator execution (notably MatMul) dominates total inference time, and that load balancing across threads varies by operator type.

Figure 6: (Un)balanced operator load across threads between activation functions and MatMul operations.

Backend partitioning is rendered for heterogeneous hardware execution, including NPU, GPU, and CPU offload patterns.

Figure 7: Timeline view of graph and operator execution across CPU, NPU, and GPU backends.

ProfTime also assists in diagnosing performance interference due to competing workloads.

Figure 8: Timeline trace of LLaMA3.2-1B showing operator delays with an interfering task on one core.

Statistical (ProfStat)

The ProfStat view supports targeted analyses of performance determinants:

• Token evolution: Profiling exposes trends in iteration time and identifies causative operators (KQ and KQV in decoding).

  Figure 9: Elapsed time for token generation, showing KQ/KQV contribution to iteration time increase.

• Operator dimension scaling: Execution times scale linearly with $M \times N \times K \times H$, revealing hidden-dimension-dependent bottlenecks.

  Figure 10: Execution time vs. matrix multiplication complexity.

• PMC-backed insights: ProfInfer quantifies thread scaling, showing that multi-threaded performance gains saturate with memory bandwidth constraints, and excess threads induce high backend stall cycles.

Figure 11: Memory access and stalled cycle ratios for MatMul with varying thread counts.

• Backend comparisons: Empirical data show that GPUs outperform CPUs only within certain tensor dimension ranges; selective operator offloading based on profiling can optimize throughput.

  Figure 12: CPU vs. GPU MatMul operator comparison at peak decoding speed.

• MoE expert activation: Fine-grained expert tracking identifies disk I/O as the principal bottleneck when activated experts are not memory-resident, with activation locality directly impacting runtime via major page faults.

  Figure 13: Activation patterns for MoE experts in Qwen1.5-MoE-A2.7B-Q4, showing density and latency impacts.

Numerical Results and Claims

ProfInfer achieves fine-grained, operator-level tracing with less than 4% runtime overhead when evaluated on a range of ARM-based edge devices, including Orange Pi 5 Ultra and Rubik Pi 3. Overhead is further reduced to as low as 1.7% under libbpf, with negligible average CPU load (<1%). Token and graph-level tracing can be confined to 0.1% overhead. These empirical results highlight the practical deployability of ProfInfer in resource-constrained environments.

Strong claims substantiated by the data:

• MatMul operations account for >97% of total inference time across models and configurations.
• Parallelism is intra-operator only; no inter-operator parallelization is realized in prevailing inference engines (e.g., llama.cpp), potentially limiting throughput.
• PMC-guided profiling demonstrates memory bandwidth saturation at roughly two threads on Cortex-A76 cores, and increasing thread counts beyond this point induces excessive cache misses and backend stalls.
• MoE inference is disk I/O bound on devices with insufficient DRAM for expert weights; major page faults correlate directly with performance degradation.

Practical and Theoretical Implications

Deployment

ProfInfer offers a practical profiling solution for developers targeting real-time, resource-aware deployment of LLMs on mobile and edge devices. The tool’s non-intrusive nature and broad hardware compatibility lower barriers to systematic performance debugging and operator-level optimization. Insights from ProfInfer can inform thread scheduling, backend selection, quantization strategies, and model partitioning to maximize efficiency under tight compute/memory constraints.

Systems Optimization

The architecture of ProfInfer enables empirical exploration of bottlenecks inherent to transformer decoding (e.g., memory-bound MatMuls, KV-cache effects, multi-expert paging), suggesting directions for backend-specific optimization, speculative decoding, or dynamic workload scheduling. Data-driven operator offloading (CPU vs. GPU/NPU) can be aligned with tensor dimensionality and hardware PMC limits, supporting adaptive heterogeneous inference.

Future Developments

The authors propose extending ProfInfer’s capabilities to partition graphs for inter-operator concurrency, integrate external memory bandwidth benchmarks, and further refine profiling on emerging NPU architectures. Enhanced expert selection and caching could alleviate MoE I/O bottlenecks. The framework’s modularity paves the way for targeting broader classes of LLM inference engines and custom hardware accelerators.

Conclusion

ProfInfer codifies a rigorous approach to LLM inference profiling by coupling eBPF-based non-intrusive system tracing with operator semantic analysis and hardware performance counter integration. It supplies actionable insights for optimizing LLM deployments on mobile/edge hardware, revealing bottlenecks and resource utilization patterns across models and inference engines. The methodology and empirical findings provide a solid foundation for future research in efficient LLM system design, adaptive backend orchestration, and profiling-driven systems optimization.