In collaboration with Intel, we recently commissioned the Evaluator Group, a technology analyst firm, to run performance benchmarks comparing Ceph Storage with Lightbits. Both are storage systems suited for cloud-native application environments. The requirement was to run the benchmarks using identical hardware and storage media, running workloads as containers in an OpenShift Kubernetes environment.
Software-Defined Storage
Ceph and Lightbits are both software-defined storage (SDS) solutions, which is storage software that handles policy-based provisioning and storage management independent of underlying hardware. SDS solutions usually include some sort of storage virtualization, which separates the storage hardware from the software that manages it. An SDS solution also typically provides functions like compression, replication, thin provisioning, snapshots, and backup.
Why This Comparison Matters
Comparing the performance of Ceph to Lightbits is relevant because of two trends that are driving significant change in the IT landscape. These are the shift toward cloud computing and the rise of container-based applications. Both technologies offer the ability to run and operate applications with greater flexibility, and, in some cases, lower costs than were traditionally possible.
Realizing the benefits of the cloud and containers does involve some challenges, however. For example, storage for container-based applications in the cloud needs to be highly scalable. It has to provide performance without requiring extensive administration. In the case of Kubernetes, the application can transparently utilize multiple compute and storage resources. This ability to retain access to persistent data, however, means that the underlying storage infrastructure has to be resilient—while also meeting the capacity and performance requirements of potentially thousands of applications and microservices.
About Ceph Storage
Ceph Storage is open-source SDS that is often used in cloud-native environments. It was originally designed 15 years ago in the era of spinning hard disk drives (HDDs). At the time, NAND and solid-state drives (SSDs) were in their infancy for mainstream storage systems. Since then, Ceph’s design has adapted to solid-state flash.
Ceph comes with interfaces that enable block, filesystem, and object storage access. However, Ceph has not been optimized for modern Persistent Memory and does not yet support NVMe over fabrics, including NVMe over TCP. Ceph also doesn’t contain optimizations for new solid-state storage, such as Quad-Level Cell (QLC) and Triple-Level Cell (TLC) storage.
About Lightbits
The Lightbits Cloud Data Platform is SDS designed for private, public, and edge clouds. It was built to utilize recent solid state storage technologies such as NAND flash and Intel Optane persistent memory (PMem). Lightbits also supports NVMe® over TCP (NVMe/TCP) and features a disaggregated architecture. This enables it to scale CPU, memory, PMem, or NVMe devices independently as needed.
Lightbits Intelligent Flash Management™ (IFM) is a set of features that maximize the performance and extend the endurance of SSDs—delivering 20X more endurance for the storage media. Key features of IFM are write striping, IOP-less metadata access, Smart Garbage Collection, Append Write Strategy, and Parallel Read/Write Pipelines.
For this comparison benchmark, Lightbits was configured to work with Intel technologies: components included Intel® Xeon® processors, Intel network adapters and Intel Optane™ technology delivered both as NVMe solid-state drives and as memory form-factor Optane Persistent Memory (Pmem) devices. Lightbits further added Intel Ethernet 800 Series with Application Device Queues (ADQ) that provide high performance, low-latency NVMe/TCP network interface cards.
The Comparison
Evaluator Group compared Ceph with Lightbits with each running as containers in a Red Hat® OpenShift® Kubernetes applications environment. The test environment comprised 12 Kubernetes nodes running container workloads against either a dedicated three-node Lightbits cluster or a three-node Ceph cluster, running on equivalent hardware. The comparison measured storage performance using the well-known ‘vdbench’ tool to create workloads. The test used eight container instances running vdbench per node, for a total of 96 instances of vdbench.
Ceph was configured to create three data copies for data protection. The Ceph instance consisted of Ceph version 16.2.6 aka “Pacific.” The three Ceph nodes ran on CentOS version 8.4. Each NVMe SSD was configured as six logical object storage devices (OSDs) with six OSDs per device.
The testing covered five different types of access patterns and block sizes. The idea was to mimic the common workloads in performance-sensitive applications:
- 4KB, 100% read with 100% random access
- 4KB, 100% write with 100% random access
- 8KB, 80% read/20% write with 80% random access
- 16KB, 70% read/30% write with 80% random access
- 32KB, 50% read/50% write with 80% random access
These workloads were used to compare the performance of Lightbits and Ceph using Quad-level cell (QLC) solid-state media as the primary storage media. Higher speed persistent media, such as Intel Optane™, was deployed where appropriate for each storage system. In all cases, Evaluator Group ran each of the five workloads multiple times, with the objective of providing an average result for comparison.
The Results
Lightbits outperformed Ceph with each workload. In terms of inputs/outputs per second (IOPs), Lightbits had 3.94X the performance in the 4k-100% read workload, 16.78X in the 4k-100% write, 12.5X in 8k-80% read, 10.41X in 16k-70% read, and 5.65X in 32k-50% read. The chart below summarizes the results. The report contains more detailed findings.
| Ceph IOPs | Lightbits IOPs | Lightbits Advantage | |
|---|---|---|---|
| 4K-100% Read | 1,032,428 | 4,068,462 | 3.94 X |
| 4K-100% Write | 30,728 | 515,697 | 16.78 X |
| 8K-80% Read | 90,363 | 1,129,335 | 12.50 X |
| 16K-70% Read | 35,740 | 372,009 | 10.41 X |
| 32K-50% Read | 19,797 | 111,852 | 5.65 X |
Conclusion
Using the same hardware configuration, Lightbits outperformed Ceph by a margin ranging from 4X to 16X. The results demonstrate that leveraging new technologies like Persistent Memory and end-to-end NVMe enabled Lightbits to provide significantly better performance with lower latency than Ceph. The comparison between Ceph and Lightbits is relevant because storage performance is always an essential consideration for IT managers, especially when dealing with cloud-native applications. Not every application or micro service requires ultra-high I/O, but when you need storage that can support thousands of microservices, it is a good practice to go with the higher-performing storage option when it is available.
Additional Resources:
