Apache Kafka has been transformative in use cases the require reliable, high-performing real-time data feeds. Deploying storage for Kafka can be challenging, however. Issues include sub-optimal storage performance, underutilization of drives, synchronization problems, and drives dictating data placement. New solutions, such as NVMe® over TCP (NVMe®/TCP) for Kafka Storage, address these difficulties—enabling the kind of flexibility and efficiency that owners of Kafka-based systems tend to want.
Overview of Kafka Storage
Kafka, which was developed by LinkedIn and open-sourced to Apache in 2011, is designed for low-latency, high-throughput handling of real-time data feeds. Kafka typically connects to external systems for the import and export of data via Kafka Connect. From this setup, Kafka is able to deliver Kafka Streams, which comprise a Java stream processing library.
Kafka works on the classic publish/subscribe (pub/sub) model, with processes known as “producers” set up to send key value messages to the Kafka server or “broker.” Kafka almost always runs as a cluster of one or more brokers. The broker can split the data streams into “partitions” that consist of a “topic” or multiple topics.
Partitions are able to be replicated onto multiple brokers, if necessary. With this architecture, a Kafka cluster can deliver a huge number of messages in streams on a fault-tolerant basis. Traditional messaging systems like Java Message Service (JMS) and Advanced Message Queuing Protocol (AMQP) are no longer necessary.
With pub/sub and massive data streams comes storage, of course. One of the most common approaches to Kafka storage is to deploy a local Solid-State Drive (SSD), such as a Non-Volatile Memory Express® (NVMe) drive, on each server running a Kafka broker, or a Direct Attached Storage (DAS) architecture.
Kafka Persistent Storage
Kakfa is architected for persistent storage. This is due to the fact that a Kakfa message log is always persistent, unlike that of most other messaging systems. Thus, a storage device attached to a Kakfa broker has to be persistent, too, retaining its data after it has been shut off. The persistent storage approach makes it possible for a Kafka system to keep data on one platform, but available for reference and access by Application Programming Interfaces (APIs) and query languages like SQL and CQL. Architects like this model because they can leverage Kafka to use with combinations of database platforms, such as Amazon Redshift or IBM DB2.
Persistent storage with Kakfa also ensures reliability. Kafka’s designers made this simple by relying on the Linux file system, which caches up to the limits of available memory. Inside a Kafka broker, all stream data gets instantly written onto a persistent log on the filesystem, where it is cached before writing it to disk.
The use of persistent storage, with its reliability, makes sense in the context of typical Kafka workloads. Kafka is often serving as the heart of a critical system. It cannot be unreliable. It cannot go down and leave the rest of the system lacking for a low-latency event streaming component. Workloads like fraud detection require constant logging and analytics. Website activity tracking, stream processing and log aggregation—essential for cybersecurity and high availability—are also common uses of Kakfa where persistent storage is needed for dependability.
Challenges with Direct Attached Storage for Modern Applications Using Kafka
Using local flash offers high performance for Kafka systems, but the architecture does come with its challenges. One issue emerges from the fact that each Kakfa “topic” is limited to a single drive. Then, with a drive on each server, there is bound to be underutilization. Topics are never the same size; it’s possible to have DAS for Kafka with some drives nearly full while others are perhaps 20% utilized.
From there, performance can suffer due to parallel input/output (I/O). For example, if multiple producers have to write to a single topic, the speed of the process will be limited by that single driver’s write performance.
There can also be difficulties with synchronization between drives, which result in cost and efficiency problems. Take synchronization, for instance. A Kafka “leader” drive is responsible for taking a message from message sources, known as “producers,” and replicating it to “follower” volumes. The followers have to keep up. If they fall behind, the followers may be declared out-of-sync. This can create problems for replication policy and degrade overall system performance.
Conventional DAS architecture with Kafka can lead to a scenario where drives dictate data placement. Given that topic partitions are limited by a drive’s capacity, the drive can start to determine where data is written. This can lead to a load imbalance between the various SSDs.
What happens if a drive fails, or if a Kafka server fails or migrates? Such an event causes a problem for DAS with Kafka. A failure will necessitate a full rebuild of data from the attached SSDs. It is time-consuming and a drag on the performance of the cluster.
How to Overcome Kafka Storage Challenges by Using a Shared Storage Environment
As with almost every area of IT, problems have solutions. The solution may not manifest right away, but if enough people are having a problem, a solution will appear. For Kafka storage, a solution that addresses the parallel I/O issue comes in the form of NVMe over Transfer Control Protocol (TCP). NVMe/TCP is the latest evolution in network protocols. It is a further iteration of the already powerful NVMe over Fabrics (NVMe-oF), which was designed to replace DAS by efficiently supporting hyperscale SSD pools.
NVMe/TCP goes even further, making NVMe-oF even higher performing. It lowers deployment costs while simplifying storage architecture. It achieves these goals by extending NVMe across an entire storage landscape using a TCP/IP fabric.
NVMe/TCP based storage is ideal for Kafka because it flattens the tradeoff between reliability and performance. Then, when coupled with a shared storage environment, as exemplified by the LightOS Logical Volumes virtual drive architecture, system owners can overcome issues with synchronization and more. Regarding synchronization, LightOS with NVMe/TCP offers an intelligent flash management layer that separates disk read and write paths. The results include consistent read/write response times along with lower tail latencies than are possible with local DAS/NVMe SSDs.
The LightOS with NVMe/TCP virtual drives also gets rid of the phenomenon of drives dictating data placement. Because they are virtual, the drives can expand beyond the limits of physical drives. This allows for data placement that makes sense for the system, not the drive. For drive rebuilds, LightOS with NVMe/TCP virtual drives offers virtual rebuilds and scaling of capacity independent of Kafka compute. It is possible to add new Kafka nodes on demand without affecting storage.
Kafka has amazing capabilities. Now, Kafka storage can perform at the same level. With NVMe/TCP and virtual drives, Kakfa systems become more flexible and high performing.
Additional Resources
SAN Replacement: Why it Might be the Time
Direct Attached Storage (DAS) Disadvantages & Alternatives
Cloud-Native Storage for Kubernetes
Disaggregated Storage
Ceph Storage
Persistent Storage
Kubernetes Storage
Edge Cloud Storage
NVMe® over TCP
