BPF-oF: Storage Function Pushdown Over the Network

Abstract: Storage disaggregation, wherein storage is accessed over the network, is popular because it allows applications to independently scale storage capacity and bandwidth based on dynamic application demand. However, the added network processing introduced by disaggregation can consume significant CPU resources. In many storage systems, logical storage operations (e.g., lookups, aggregations) involve a series of simple but dependent I/O access patterns. Therefore, one way to reduce the network processing overhead is to execute dependent series of I/O accesses at the remote storage server, reducing the back-and-forth communication between the storage layer and the application. We refer to this approach as \emph{remote-storage pushdown}. We present BPF-oF, a new remote-storage pushdown protocol built on top of NVMe-oF, which enables applications to safely push custom eBPF storage functions to a remote storage server. The main challenge in integrating BPF-oF with storage systems is preserving the benefits of their client-based in-memory caches. We address this challenge by designing novel caching techniques for storage pushdown, including splitting queries into separate in-memory and remote-storage phases and periodically refreshing the client cache with sampled accesses from the remote storage device. We demonstrate the utility of BPF-oF by integrating it with three storage systems, including RocksDB, a popular persistent key-value store that has no existing storage pushdown capability. We show BPF-oF provides significant speedups in all three systems when accessed over the network, for example improving RocksDB's throughput by up to 2.8$\times$ and tail latency by up to 2.6$\times$.

• The paper introduces a protocol that uses eBPF to execute custom storage functions on remote servers, mitigating network overhead and reducing CPU load.
• The methodology splits I/O operations into in-memory cache lookups and disk transactions while ensuring metadata synchronization for safe remote execution.
• Performance tests demonstrate up to 2.8× throughput improvement, 37% fewer CPU cycles, and a 23% reduction in network traffic across storage systems.

BPF-oF: Storage Function Pushdown Over the Network

Introduction

"BPF-oF: Storage Function Pushdown Over the Network" introduces a novel approach to reduce CPU resource usage in storage disaggregation environments via a new protocol, BPF-oF, that leverages eBPF for executing custom storage functions on remote storage servers. The core idea is to mitigate the network processing overhead caused by disaggregation by shifting complex, series-dependent I/O operations closer to the storage layer. This strategy not only lessens the back-and-forth communication but also aligns with the increasing deployment of NVMe-oF (NVMe over Fabrics) in data centers.

Figure 1: NVMe-oF overview.

Motivation and Challenges

Storage disaggregation allows applications to separate scaling of compute and storage resources, promoting flexibility and efficiency. However, this setup burdens systems with additional network latency and CPU processing costs, especially notable in NVMe-oF environments using TCP, which can halve throughput compared to local disk access.

BPF-oF addresses the problem by enabling safe execution of eBPF functions on remote servers, effectively grouping multiple I/O operations into single transactions executed closer to the data. This requires overcoming several challenges:

• Versioning and Metadata Synchronization: Ensuring that file-to-block mappings remain consistent between the client and server, particularly when inodes are updated.
• Integration with In-Memory Caching: Striking a balance between leveraging in-memory data structures and executing storage operations remotely without losing the performance benefits those caches provide.

System Design

BPF-oF operates on three key components:

1. Metadata Synchronization: An asynchronous mechanism using versioning to maintain up-to-date inode mappings which helps in safe node-read operations.
2. Query Splitting: Dividing operations into in-memory cache lookups and disk operations to optimize performance. This effectively reduces the disk I/O operations required on remote execution.
3. eBPF Execution Framework: Incorporating a programmable layer for running eBPF functions that interpret and act on received data, effectively filtering and aggregating it before completing the query response.

   Figure 2: BPF-oF architecture.

Performance Analysis

BPF-oF’s implementation within RocksDB, WiredTiger, and BPF-KV demonstrates significant performance boosts. For instance, the integration with RocksDB improves throughput by up to 2.8× and lowers tail latency by 2.6× across various read-intensive workloads. These improvements are largely attributed to effective elimination of redundant network traversal and CPU processing reduction.

Additionally, comparative analysis with NVMe-oF setups affirms BPF-oF's efficacy across network protocols (TCP/RDMA) and storage types (NAND/Optane SSDs):

• NAND SSDs: BPF-oF provided 1.4-2.2× improvement over NVMe/TCP.
• Optane SSDs: Enhanced throughput by 2-2.8×, underscoring the application’s capability to leverage high IOPS devices more efficiently.

Figure 3: RocksDB (a) uniform read throughput-latency and throughput (b) without and (c) with data blocks cached of BPF-oF vs. NVMe/TCP on NAND SSD.

CPU and Network Considerations

One of BPF-oF's key advantages is reducing CPU load on both client and server sides, thus lowering overall power consumption. Studies show up to 37% fewer CPU cycles and 23% less network traffic per request under TCP setups. Energy savings further substantiate BPF-oF's efficiency by showing a 36% decrease in energy consumption per operation.

Figure 4: RocksDB throughput with different sampling rates with cache under BPF-oF using TCP and NAND SSD.

Future Work

Current limitations of BPF-oF include its compatibility with ext4 and reliance on predetermined file accesses. Addressing support for additional file systems, integrating advanced storage configurations (like RAID), and refining support for encrypted storage remain key areas for future research. Additionally, further optimizing the deployment within SmartNICs presents an opportunity to harness hardware acceleration further, advancing the utility of storage disaggregation models in complex datacenter environments.

Conclusion

BPF-oF's architecture and implementation realizes substantial efficiency improvements in storage-disaggregated environments, specifically through network and CPU optimizations. The protocol's design and results advocate for broader adoption of function pushdowns in storage systems, offering promising avenues for future advancements in both software methodologies and hardware enhancements.