RoCEv2 RDMA Offload Engine

• RoCEv2-compliant RDMA offload engines are specialized hardware/firmware platforms designed to enable zero-copy, low-latency, high-throughput data movement over Ethernet.
• They use pipelined architectures with integrated modules for protocol parsing, queue pair management, DMA operations, and memory translation to offload the RDMA stack from host CPUs.
• Engine designs optimize performance with features like precise congestion control, FPGA-driven reconfigurability, and support for diverse applications in data centers, scientific instrumentation, and distributed ML.

A RoCEv2-compliant RDMA offload engine is a hardware or tightly integrated firmware-hardware platform that implements the RDMA over Converged Ethernet version 2 (RoCEv2) transport protocol, providing zero-copy, low-latency, high-throughput data movement directly between endpoints via lossless or near-lossless Ethernet fabrics. These engines are implemented in FPGAs, SmartNICs, DPUs, or ASICs to offload the entire RoCEv2 stack from host CPUs, enabling high-performance scientific instrumentation, data center networking, and accelerator-based architectures. RoCEv2-compliant engines typically provide reliable connection (RC) transport, direct integration with host or accelerator memory, and support for standard queue pairs (QPs), work queue entries (WQEs), completion queues (CQs), and memory registration as defined by the Infiniband and RoCE standards.

1. Core Architecture and Data Path

Fundamental to a RoCEv2-compliant RDMA offload engine is a pipelined architecture comprising several tightly coupled modules: protocol header parsers, queue pair (QP) managers, DMA engines, memory translators, flow- and congestion-control units, and completion queue logic. High-speed AXI-Stream or memory-mapped interconnects are used for data and control across internal blocks.

For example, the BALBOA engine uses a deep pipeline (at 250 MHz, 512 bits/cycle, 128 Gb/s) with modules for Ethernet/IP/UDP/IB header parsing, QP/PSN state lookups, BTH/RETH handling, retransmission/hbm stream mux, flow control (ACK-clocked, per-QP), and fast ICRC offload, culminating in direct PCIe DMA to host or GPUs (Heer et al., 27 Jul 2025). Similarly, RecoNIC instantiates AMD's ERNIC IP as the RDMA offload block, supporting 100 Gb/s per port and routing packets between MACs, compute blocks, and the host through AXI4-Stream/crossbar fabric (Zhong et al., 2023).

The CTAO engine targets a 10 GbE link, with a front-end JESD204C receiver, in-FPGA packetizer and trigger logic, and a back-end RDMA WRITE controller that injects encoded packets on fully managed 10 GbE links (Marini et al., 2 Sep 2025).

The following table summarizes high-level architecture features from representative implementations:

Engine	Fast Path Width/Rate	Host Interface	Supported Protocols
BALBOA	512b@250 MHz/100 G	PCIe XDMA/QDMA, GPU	RC: WRITE, READ, SEND
RecoNIC	Up to 100 G	PCIe AXI4-MM, DDR4	RC & UD: WRITE, READ, SEND
CTAO-LST	10 G AXI-Stream	SURF MAC, FPGA DMA	RC: RDMA WRITE only

2. RoCEv2 Protocol Compliance and Stack Handling

RoCEv2-compliant engines must implement the protocol stack: Ethernet → IPv4 → UDP → Infiniband Transport Headers (BTH, RETH) with precise conformance to header layouts, in-order delivery, correctness in PSN state, and mandatory iCRC coverage.

Each packet processed by the offload engine is wrapped/parsed with the required protocol headers: Ethernet (including VLAN/Priority fields as needed), IPv4 (RFC 791, ECN bits programmable), UDP (length, zero checksum, as specified by RoCEv2), and Infiniband BTH/RETH/AETH fields. The engines insert PSN, QPN, and opcode, and compute or verify the invariant CRC (iCRC, e.g., in a 40 ns/512b pipeline in BALBOA) (Heer et al., 27 Jul 2025).

Reliable Connection (RC) semantics are standard: every outgoing packet (SEND/WRITE/READ) must be acknowledged at the protocol level; missing ACKs trigger retransmissions managed by replay buffers and protocol state machines. Flow- and congestion-control mechanisms (e.g., Priority Flow Control, DCQCN) are implemented in both NIC hardware and FPGA-based engines, with control parameters programmable per QP or per priority (Qi et al., 16 May 2025, Marini et al., 2 Sep 2025).

For large payloads exceeding MTU (e.g., 4 KB), payload segmentation into RoCEv2 datagrams with correct PSN and ordering is handled in hardware, and reassembly occurs in FPGA-resident BRAM or device-side memory (Zhong et al., 2023).

Table—Protocol Compliance Features (selected implementations):

Feature	BALBOA	RecoNIC	CTAO-LST
Ethernet encapsulation	Yes	Yes	Yes
Supported opcodes	RC: W,R,S	RC&UD: W,R,S	RC: W only
Packet iCRC/iFCS	Yes	Yes	Yes
Congestion marking/ECN	PFC/DCQCN	DCQCN planned	ECN bits

3. Memory Registration, Queue Pair, and Work Request Management

Memory registration and virtual-to-physical translation are supported as per the RoCE/IB verbs API. Engines track Memory Region (MR) descriptors (rkey, base VA, protection domain) for both local and remote buffers.

Queue Pairs (QPs) represent connection state and are indexed in dedicated tables (up to 500 QPs at 100 Gbps in BALBOA) (Heer et al., 27 Jul 2025). Each QP maintains send and receive queues as ring buffers of WQEs; doorbell registers are used for notification, typically batched for latency amortization (Zhong et al., 2023).

Completion queues (CQs) are realized as FIFO structures, updated on ACK/operation completion events and polled/consumed either by the host (via MMIO or PCIe-DMA) or by accelerators directly. Zero-CPU offload is central: all per-transfer packet assembly, submission, and completion handling are hardware-driven, with the host/CPU only managing setup and teardown (Marini et al., 2 Sep 2025).

In multi-tenant or distributed scenarios (e.g., Palladium (Qi et al., 16 May 2025)), QPs and MR keys are virtualized and pooled, with fairness enforced via deficit-weighted round robin (DWRR) or credit-based scheduling to prevent cross-tenant interference.

4. Performance Characteristics and Resource Utilization

Offload engines deliver bandwidth, latency, and resource utilization metrics competitive with commercial ASIC NICs:

• Throughput: Engines such as BALBOA saturate 100 Gbps at 32 KB buffers for both WRITEs and READs; RecoNIC achieves near line-rate (>92 Gbps) for batched RDMA operations (Heer et al., 27 Jul 2025, Zhong et al., 2023).
• Latency: Pipeline latencies are typically <2 µs (488 ns pure pipeline at 64 B in BALBOA, 600–960 ns FPGA→host in RecoNIC, sub-microsecond hardware path in CTAO). End-to-end 4 kB RDMA WRITE over 10 G is typically ~1.5–2.5 µs (Marini et al., 2 Sep 2025).
• Scalability: Support for hundreds of concurrent QPs with linear bandwidth sharing; engineered to saturate available physical link speeds.
• Resource usage: FPGA resource footprints are ≈4–13% of LUTs and 1–5% of BRAM on modern FPGAs for the full engine, with streaming datapaths replacing DSP use for most core functions (Heer et al., 27 Jul 2025Marini et al., 2 Sep 2025).

Performance equations provided in the literature, e.g.,

$$\mathrm{Latency}_{\mathrm{BALBOA}}(s) = \alpha + \beta \frac{s}{R}$$

with $\alpha \approx 1.2\,\mu\mathrm{s}$, $\beta \approx 0.0005\,\mu\mathrm{s}/\mathrm{B}$, $R=12.5\,\mathrm{GB/s}$ (Heer et al., 27 Jul 2025).

5. Integration, Portability, and Extensibility

Many engines are explicitly designed for portability and in-field extensibility. For example, BALBOA features AXI-Stream modularity, supporting rapid insertion of protocol services (encryption, ML-based DPI) or application-specific compute offloads (e.g., ML preprocessing for recommender systems) directly on the RoCEv2 pipeline. At 100 G, these accelerations add negligible (<100 ns) incremental latency and do not limit line rate (Heer et al., 27 Jul 2025).

Firmware-level reconfiguration (e.g., new collective algorithms in ACCL+ (He et al., 2023)) and runtime parameter tuning (MTU, QP, thresholds) enable deployment without re-synthesis. Engines written in portable HDL (e.g., Bluespec SystemVerilog in CTAO, plain Verilog MACs) can be retargeted across FPGA vendors or silicon process nodes (Marini et al., 2 Sep 2025).

Multi-lane, multi-engine instantiations support highly parallel applications (e.g., readout for tens of detector channels per device), with hardware scheduler logic managing parallel flows for bandwidth scaling (Marini et al., 2 Sep 2025).

6. Application Domains and Measured Use Cases

RoCEv2-compliant RDMA offload engines have seen deployment in a range of high-throughput, low-latency domains:

• Scientific instrumentation: Instruments such as CTAO-LST use FPGA-based engines for direct, zero-copy streaming from JESD204C digitizers to event builders, handling 12 × 12 Gb/s front-end channels with hardware triggers and cyclic buffers (Marini et al., 2 Sep 2025).
• Cloud and Datacenter: Platforms like RecoNIC (Zhong et al., 2023) and BALBOA (Heer et al., 27 Jul 2025) act as SmartNICs for heterogeneous compute and accelerator offload; DPUs in Palladium (Qi et al., 16 May 2025) drive multi-tenant, zero-copy serverless data planes, freeing up CPU resources while maintaining strict tenant fairness.
• Distributed ML: ACCL+’s CCLO engine provides FPGA-resident collectives at near line-rate (95 Gb/s+), supporting both in-kernel and host-driven MPI primitives with competitive MPI-style APIs (He et al., 2023).
• Service insertion: BALBOA demonstrates in-pipeline AES encryption and machine-learning packet inspection, as well as application-level preprocessing for ML, with up to 12 × throughput and 5 × reduced latency compared to CPU-based preprocessing (Heer et al., 27 Jul 2025).

Measured microbenchmarks confirm that state-of-the-art FPGA engines can match or approach the performance of high-end ASIC RDMA NICs (e.g., ConnectX 5/6), with <0.3 µs added latency and comparable bandwidth saturation points (Heer et al., 27 Jul 2025, Zhong et al., 2023).

7. Design Trade-offs, Limitations, and Trends

While fully offloaded engines eliminate CPU cost for data movement and offer deterministic low latency, there are design and implementation trade-offs:

• Verb and Transport Support: Some engines (e.g., CTAO) restrict verb support to RDMA WRITE only for resource optimization, removing SEND/READ paths as needed (Marini et al., 2 Sep 2025).
• Host Integration: Access to host memory requires robust memory registration and address translation infrastructure, with PCIe bandwidth and doorbell batching as bottlenecks for small message rates (Zhong et al., 2023).
• Congestion and Flow Control: End-to-end lossless operation relies on comprehensive support for ECN, PFC, and DCQCN, which must be coordinated between hardware and network switches; partial or under-developed support may limit performance in oversubscribed fabrics (Marini et al., 2 Sep 2025, Qi et al., 16 May 2025).
• Extensibility: AXI-Stream modularity and firmware reconfigurability are not universal; some commercial IP blocks may restrict deep customization (Zhong et al., 2023).

A plausible implication is that, as FPGA and DPU fabric integration advances, RoCEv2-compliant offload engines will increasingly serve as in-network compute devices, supporting dynamic service insertion, large-scale multi-tenancy, and seamless accelerator–network interconnects with minimal host intervention.

---

References:

(Marini et al., 2 Sep 2025, Zhong et al., 2023, He et al., 2023, Qi et al., 16 May 2025, Heer et al., 27 Jul 2025)