Choosing the Right NVMe over Fabrics Transport Protocol Option for Data Center Storage
With dramatic growth in data, the need to move faster, store more, and process more data is essential to grow your business and remain competitive. To support fast data transfer for high performance storage, the new Intel Ethernet 800 Series supports all three Ethernet-based transfer protocols over network fabric: Non-Volatile-Memory-Express (NVMe) over iWARP RDMA, NVMe over RoCE v2 RDMA, and NVME over TCP. But how do you decide which approach is right for your organization?
Different Protocols Options
There are several different data transport protocol options supported by NVMe-oF, each of which has different advantages.
Connectivity Data Center
As I mentioned above, the Intel Ethernet 800 Series supports all three of the Ethernet-based options: NVMe over iWARP RDMA, NVMe over RoCE v2 RDMA, and NVME over TCP (Transmission Control Protocol). Furthermore, NVMe over TCP can be accelerated with Application Device Queues (ADQ) technology, also available in the Intel Ethernet 800 Series. Let’s take a look at the advantages of each.
To start, both iWARP and RoCE v2 (RDMA over Converged Ethernet) are supported by Remote Direct Memory Access (RDMA). RDMA enables two machines on the same network to exchange memory without involving the operating system or the processor, further cutting latency. We have found that congestion management is essential for RDMA deployment because it can minimize packet loss and improve performance and scalability. Intel Ethernet 800 Series therefore supports both Congestion Notification (ECN) and Roundtrip Time (RTT) based congestion control technologies.
RDMA using iWARP
iWARP sits on top of TCP/IP to manage traffic congestion with the reliability of TCP transport. Organizations may want to consider iWARP because it is highly scalable and works across standard Ethernet without the complexity of DCB (data center bridging) and PFC (priority flow control), making it easy to deploy and maintain. Plus, iWARP still uses legacy infrastructure, like existing switches and routers, while taking advantage of RDMA’s high performance and low CPU utilization.
RDMA using RoCE v2
RoCE sits on top of UDP/IPv4 or IPv6 and runs on Converged Ethernet, which uses a set of DCB enhancements to create a lossless Ethernet network and is configured via PFC. RoCE v2 has been updated to take advantage of User Datagram Protocol (UDP) and can be routed. Though generally less scalable than iWARP, RoCE v2 takes advantage of RDMA’s high performance and low CPU utilization, while also using the popular Ethernet standard with more defined controls for a more efficient network. As more converged workloads, such as high performance computing (HPC), are moving to the Ethernet standard, organizations looking to prepare for cutting-edge processes may want to consider RoCE v2.
NVMe over TCP with ADQ Acceleration
While both iWARP and RoCE v2 offer significant speed advantages, not all organizations have the infrastructure needed to support RDMA immediately. NVMe/TCP enables customers to run NVMe-oF over existing Ethernet infrastructure with the use of TCP to deliver data across existing IP networks. This can be an attractive option for organizations who have made substantial investments in their legacy infrastructure. Because TCP is common and well understood through the use of out-of-the-box Linux, it is easy to implement and inherently scalable. However, it typically lags RDMA-based solutions in terms of performance. Accelerating NVMe/TCP with new Application Device Queues (ADQ) found in the Intel Ethernet 800 Series, builds on the positive attributes of NVMe/TCP while lessening the performance gaps with NVMe-oF with RDMA to offer customers an even more attractive choice.
Intel is working on proposed updates (“patches” in Linux-speak) to the open source Linux NVMe/TCP host and target necessary to enable ADQ acceleration. At the Storage Developer Conference being held this week in Santa Clara, my colleague Rebecca Weekly from Intel’s Cloud Platforms Group will announce that Intel is making the proposed open source Linux NVMe/TCP updates necessary to enable ADQ acceleration available as Request for Comments (RFC) to the Linux community, starting with RFC for the target updates this week and moving to RFC for the host updates later this year. This is a first step in making ADQ acceleration of NVMe/TCP open to all in the industry.
As Muli Ben-Yehuda, co-founder and chief technology officer at Lightbits Labs, notes,
“Lightbits LightOS, based on standard NVMe/TCP and running on standard servers, is a disaggregated storage solution that is inherently scalable, easy to deploy, and low-latency. We anticipate that adding the Intel Ethernet 800 Series with Application Device Queues (ADQ) NICs to servers running LightOS will provide even higher performance and even lower latency, making LightOS an even more attractive choice for building cloud-native disaggregated storage stacks.”
The Necessity of a Data-Centric Approach
Choosing the most appropriate NVMe-oF for your organization is only one piece of a larger puzzle. To fully realize the advancements being made in data center connectivity requires a holistic view of operations. Intel Ethernet 800 Series combined with 2nd Generation Intel® Xeon® Scalable processors, Intel® Optane™ DC persistent memory, and Intel® SSDs, provide an excellent foundation for our customer’s evolving data-centric workloads. For more information, visit intel.com/ethernet.
Article originally published via Intel IT Peer Network
Learn more about NVMe Storage and Why NVMe/TCP Is The Better Choice
