Modeling hyperscaler cloud architecture is gaining significant momentum in enterprise data centers as many IT teams are repatriating their public cloud workloads back on-premises, modernizing their data center for cloud-native workloads, or building their own specialized public cloud services. They want to integrate the best capability and efficiency aspects of the public cloud with on-premises control. Technological innovations have emerged as key enablers of best-of-breed cloud architecture to achieve the benefits promised by the public cloud, which are software-defined NVMe storage and open-source orchestrators such as Kubernetes. All are gaining popularity as foundational components of modern cloud architecture.
What is NVMe Storage?
NVMe (Non-Volatile Memory Express) is a storage protocol specifically designed for high-performance storage devices like solid-state drives (SSDs). It’s like a super-fast lane for data to travel between your storage and your computer’s processor. The first specifications for NVMe 1.0 were released in January 2013.
Why is NVMe storage gaining in popularity? NVMe leverages the PCIe (Peripheral Component Interconnect Express) interface, which offers much higher bandwidth than traditional SATA connections. This means data can be read and written at incredibly fast speeds, leading to quicker boot times, faster application loading, and smoother overall system performance.
NVMe flash storage reduces latency and overhead, allowing for more efficient communication between the storage device and the system. And because it’s highly parallel, it can handle multiple data requests simultaneously, making it ideal for IOPs intensive operations.
Source: Lightbits Labs
What is NVMe-of?
The NVMe-oF (NVMe over Fabrics) v1.0 specification was released in June 2016. NVMe-oF takes NVMe to the next level by extending the parallel access and low latency features of NVMe protocol across networked storage. Originally designed for local storage and common in direct-attached storage (DAS) architectures, NVMe delivers high-speed data access and low latency by directly interfacing with solid-state disks (SSD). NVMe-oF allows these same advantages to be achieved in distributed and clustered environments by enabling external storage to perform as if it were local. Think of NVMe as a high-speed highway for data within your computer. NVMe-oF extends that highway across a network, allowing multiple computers to access super-fast storage as if it were right next to them. Imagine being able to access storage as if it were directly connected to your computer, even if it’s located in a different NVMe storage server or data center.
NVMe-oF uses a network fabric allowing multiple computers to share access to a pool of high-performance NVMe storage. It can support various transport protocols, including Fibre Channel, InfiniBand, remote direct memory access (RDMA), Ethernet with RoCE v2, iWARP, and Transmission Control Protocol (TCP).
NVMe-oF improves resource utilization and simplifies storage management, enabling you to scale your storage without compromising performance and making it ideal for modern data-intensive workloads. It addresses use cases where high performance and efficiency at scale are critical requirements.
Source: Lightbits Labs
Market adoption of Enterprise NVMe Storage
While still relatively adolescent in its market penetration (5-20% according to Gartner’s Hype Cycle for Storage Technologies, 2024), given all of the technological and business benefits it offers, I expect that it will gain momentum as the main Tier-1 cloud storage connectivity. Early adopters of NVMe-oF are already reaping its benefits to their competitive advantage.
Source: Gartner Hype Cycle for Storage Technologies, 2024.
NVMe-oF substantially reduces data access latency while ensuring more efficient connectivity between storage and servers. Storage targets can be dynamically shared among workloads, thus providing composable storage resources that provide flexibility, agility and greater resource efficiency.
The adoption of NVMe-oF is evident across industries where high performance, efficiency and low latency at scale are critical. Notable market sectors include: financial services, e-commerce, AI and machine learning, and specialty cloud service providers (CSPs). Legacy VM migration, real-time analytics, high-frequency trading, online transaction processing (OLTP), and the rapid development of cloud native, performance-intensive workloads at scale are use cases that have compelled organizations to modernize their data platforms with NVMe-oF solutions. Its ability to handle massive data flows with efficiency and high-performance makes it indispensable for I/O-intensive workloads.
What is NVMe over TCP?
Technological advancements like NVMe over TCP (NVMe/TCP) can induce a paradigm shift in data center architecture that enables fast data access at scale while at the same time offering the best cost-efficiency. NVMe/TCP, as a subset of NVMe-oF, combines the high performance of NVMe with the ubiquity and cost-efficiency of standard Ethernet networks using TCP as its transport layer. Unlike alternatives like Fiber Channel (FC) or RDMA-based protocols, NVMe/TCP operates over ubiquitous and cost-effective Ethernet infrastructure. This architecture model ensures ultra-low latency and high throughput without custom hardware or proprietary drivers.
Developed by Lightbits Labs and donated to the Linux community, it is the next evolution of using NVMe storage over TCP Fabric. Lightbits software-defined storage is easy to manage with latency as low as 170 microseconds while supporting up to 6 million IOPS with just two nodes. The ability to deploy the storage software on hardware of your choosing and use your existing Ethernet infrastructure makes it accessible and cost-effective for organizations looking to modernize their data center systems or to replace their legacy FC infrastructure. In comparison to DAS, Lightbits can deliver higher utilization for lower TCO and better utilization of NVMe flash for longer endurance. Compared to DAS [and SAN], Lightbits NVMe/TCP storage delivers high performance without the proprietary hardware stack.
Technological innovations like NVMe/TCP exemplify the best of NVMe storage and have accelerated its adoption by reducing the cost and complexity of deploying NVMe-oF. NVMe/TCP is ideally positioned to replace iSCSI (Internet Small Computer Systems Interface) and FC and become the de facto standard for data center modernization and the underlying storage access protocol to support cloud native applications with a thirst for higher performance.
Best NVMe Storage
While NVMe over TCP is gaining traction across the industry, it’s important to note that not all storage vendors who have adopted this technology are the same. Many have simply retrofitted their traditional storage products to utilize NVMe/TCP, which can result in suboptimal performance and limited scalability.
Lightbits Labs, however, took a different approach, they designed their platform from the ground up to maximize the benefits of this technology. This contrasts sharply with vendors like Pure Storage and Dell PowerFlex, who have added NVMe/TCP as a feature to their existing architectures rather than developing a purpose-built solution.
3 Significant Business and Technological Benefits of NVMe-oF
NVMe-oF can transform the way modern storage systems are architected and used.
Here are three business benefits:
1. Consolidation of storage resources for better efficiency
With NVMe-oF, you can consolidate and share high-speed storage across multiple servers, reducing storage silos and maximizing resource utilization. Doing so can reduce CapEx and OpEx by eliminating the need to overprovision storage and infrastructure management resources. By using ubiquitous Ethernet, NVMe/TCP delivers the best cost-efficiencies of all the network protocol options.
This is a particularly ideal scenario for cloud service providers who can pass savings on to customers. Offering faster storage services at competitive prices can be an advantage. It’s also an ideal scenario for companies looking to replace a virtual storage area network (vSAN). By eliminating overprovisioned storage, you can reduce your storage footprint by as much as 25%, cut energy costs and free up rack space in your data center.
2. Low latency and high throughput for accelerated application performance at scale
The ultra-low latency and high throughput benefits of NVMe-oF are unequaled. Fast access to big data accelerates application performance, time to market and time to answers, which are critical for real-time analytics, AI/machine learning workloads and high-frequency trading.
For this reason, early adopters of NVMe-oF are in the financial services and e-commerce sectors. Real-time fraud detection and online retail platforms can experience significant application performance degradation or timouts using traditional direct-attached storage (DAS) and storage area network (SAN) storage due to high latency. By deploying NVMe-oF, you will reduce latency from milliseconds to microseconds, enabling millions of transactions per second and tens of thousands of simultaneous user requests without bottlenecks or application timeouts. Such a highly performant data platform can improve customer satisfaction, trust and confidence while boosting revenue and operational efficiency.
3. Enhanced scalability and flexibility align with data center modernization initiatives
NVMe-oF supports seamless scaling by enabling storage to be dynamically added or reallocated without disrupting operations. NVMe/TCP storage enables organizations to dynamically scale or reallocate storage resources between applications, and helps ensure uptime and optimize storage costs while supporting new projects without disrupting operations.
This is particularly prevalent in AI/ML. The more data that can be run through inference and training models, the better the AI and ML outcomes. The workloads are inherently data-intensive, requiring storage solutions that can handle massive data sets with high performance and efficiency. NVMe/TCP’s high throughput allows AI/ML training and inference processes to access vast amounts of data efficiently, accelerating model training. As the scale of these workloads increases, the flexibility to scale storage and compute resources ensures that infrastructure remains agile and cost-efficient.
The Future of NVMe
NVMe is a game-changer for storage performance, and NVMe-oF takes it even further by enabling high-speed access to NVMe storage over a network. Together, they represent the future of storage technology, offering unparalleled speed, efficiency, and scalability.
The adoption rate of NVMe-oF is poised to accelerate as organizations continue to model hyperscale-like cloud architectures in their data centers. Key trends driving its adoption include enhanced protocol standardization and widespread vendor support, further simplifying deployment and interoperability. The industry has widely accepted that the NVMe-oF model will replace DAS and become the default protocol for disaggregated storage in cloud-modeled infrastructure.
The development of more cost-effective, high-capacity NVMe drives will complement NVMe-oF’s scalability, making it even more accessible to more organizations. And its ability to reduce power consumption and optimize resource utilization aligns with organizations’ sustainability initiatives.
The Rise of NVMe over TCP
Industry tech leaders like Microsoft have recognized the convergence of cloud native computing and modern data centers by jumping into the mix and democratizing the NVMe protocol with its announcement in 2023 at MSFT Ignite to support inbox NVMe/TCP, making it available now on all data center operating systems.
Ultimately, NVMe/TCP’s role as a foundational technology for modern data centers ensures its relevance well into the future. It’s already seen in new use cases, such as edge computing, where high-speed storage is critical to processing data closer to its source. Its capabilities will continue to empower businesses to meet the challenges of the cloud era with flexibility, performance and efficiency.
If you want to learn more about NVMe/TCP and its transformative benefits for data center modernization, download the white paper we produced with IDC “NVMe over TCP Enables the Democratization of Disaggregated, NVMe Storage.”
