Faster Offloads by Unloading them -- The RDMA Case

Abstract: From hardware offloads like RDMA to software ones like eBPF, offloads are everywhere and their value is in performance. However, there is evidence that fully offloading -- even when feasible -- does not always give the expected speedups. Starting from the observation that this is due to changes the offloads make -- by moving tasks from the application/CPU closer to the network/link layer -- we argue that to further accelerate offloads, we need to make offloads reversible by unloading them -- moving back part of the offloaded tasks. Unloading comes with a set of challenges that we start answering in this paper by focusing on (offloaded) RDMA writes: which part of the write operation does it make sense to unload? how do we dynamically decide which writes to execute on the unload or offload path to improve performance? how do we maintain compatibility between the two paths? Our current prototype shows the potential of unloading by accelerating RDMA writes by up to 31%.