Clustered Realities: Migrating VMs to Red Hat OpenShift Virtualization

Intro:

The journey that containers have had since around the mid 2010s is very impressive. From a developer’s “hack” to run a different “OS” with the same kernel, to something users can package an application into, to combining containers for a specific task (pods) and finally managing these pods in a manner that is sustainable, upgradeable, version trackable and easily deployed (Kubernetes). Today, Kubernetes is at the forefront of cloud infrastructures and platforms. For example, Red Hat OpenShift is recognized as the industry’s leading hybrid cloud application platform powered by Kubernetes.

In 2016, as Kubernetes’ popularity and adoption surged, Red Hat launched an open source project called KubeVirt with the aim of enabling virtual machines (VMs) to run alongside containers using Kubernetes. While the use cases for implementing Kubernetes continued to focus on containers, KubeVirt was positioned as an ancillary project to address specific corner cases during which time Kubernetes core capabilities continued to be developed. 

In 2024, KubeVirt grabbed the headlines again thanks not only to the massive popularity of Kubernetes, but also because enterprise customers began investigating an alternative to current hypervisor standards and virtualization platforms, as well as modernize their infrastructure with the mixture of pods and VMs running side by side. 

This brings us to the critical question that we’re now faced with: How do we modernize a virtualization platform without modernizing the critical underlying storage subsystem? Why is storage so important for running VMs in Red Hat OpenShift (or any Kubernetes)? How does Lightbits block storage fit the virtualization world in Kubernetes? 

Well, let’s dive into that.

Red Hat OpenShift Virtualization and KubeVirt:

At its core, KubeVirt is utilizing Linux’s ability to run VMs using KVM, a long battle-tested open source project. KubeVirt is an open source project that enables users to run, deploy and manage VMs with Kubernetes as the underlying orchestration platform—making it remarkably straightforward. KubeVirt is also the foundation for Red Hat OpenShift Virtualization, a feature of Red Hat OpenShift that supports the migration and management of VMs and containers on a consistent hybrid cloud platform. 

There is, of course, more to Red Hat OpenShift Virtualization than just pods running virtual machines. Red Hat OpenShift Virtualization uses CRDs (Custom Resource Definition) to make a VM look like any other Kubernetes object, and together with controllers and agents on the worker nodes, users get to control and run tasks like stop, start, pause and more on the VMs. This makes it very easy for customers who are either migrating VMs (even VMs from a non-Linux environment) or have certain tasks that require a VM but want to manage them together in the same Kubernetes environment where the rest of their applications are running.

Creating a virtual machine in Red Hat OpenShift:

There are several ways to create a brand new VM in Red Hat OpenShift. Users can install a new operating system (OS) from an ISO, import an existing qcow2 image, or simply use the OS images that Red Hat provides out-of-the-box as part of the Red Hat OpenShift Virtualization Operator (see image below).

The catalog has Red Hat-certified images that are downloaded along with the Red Hat OpenShift Virtualization Operator. 

Storage for Red Hat OpenShift Virtualization:

VMs in Kubernetes support multiple types of devices that can be attached as storage, LUN, disk, CDROM and filesystems. In this article we will concentrate on the disk type of storage.

One of the key benefits of embedding a VM into the container world of Kubernetes is that the VM is able to somewhat mimic the behaviour of a pod. For instance, being able to migrate VMs from one worker node to another, just like any other pod. The key distinction is that moving a traditional pod from worker node to worker node is different from migrating a VM from worker node to worker node, as the connections are expected to be continuous and uninterrupted.  

Live migration is a capability that needs to be enabled at the VM level when it’s created, and the recommendation is to use a raw block device (vs a filesystem) as the PVC that runs the VMs. The other feature that is required of the storage class and inherently from the storage vendor, is to support ReadWriteMany (RWX). Lightbits storage supports RWX at raw block device level (“volumeMode: Block” setting in the storage class).

Another storage capability that many need is the ability to sustain high throughput of IOs consistently during standard operations as well as during a live migration. Workloads and users require storage that uses advanced technology to provide a consistent stream of IOs to your VMs (and, of course, to your pods as well) – and this is where Lightbits storage comes into play.

Lightbits Storage

Lightbits architecture is based on NVMe/TCP (NVMe® Over TCP) as the transport layer. In fact, the founders of Lightbits are the people that invented NVMe/TCP. The ability to handle multiple streams of IOs with dedicated queues is one of the protocol advantages, and the Lightbits targets (the servers that create the Lightbits cluster), can each provide north of 4.2M IOPS. This massive performance (a minimal cluster of 3 targets can provide 12M+ IOPS) is one of the key advantages of Lightbits, minimal footprint/tile space in the datacenter to match any application or cluster of applications performance needs.

This comes into play when looking at live migration of VMs using Red Hat OpenShift Virtualization, especially those that run demanding applications that can not be disturbed during the migration.

Demo of live migration while IOs are running:

Let’s demo how live migration works using Lightbits storage while under constant IOs.

To emulate a VM that runs a heavy IO application, I’ve created a VM using the following yaml file:

As you can see there are 3 volumes (PVCs) that are being created, the first to hold the Fedora operating system and the other two volumes to hold the application data (think a PostgreSQL database, so one volume for data and one volume for logs).

As explained earlier, the important part here is that we’re using a storage class that supports RWX and block mode for the volume.

As you can see the VM has the following block devices setup:

Now that we have a running VM, I’m going to run a simple fio script that will create IOs (70r/30w) to both vdc and vdd (our data disks), and while the two jobs are running on both disks in parallel, I’ll migrate the VM.

First, lets see where the VM currently resides:

To initiate the VM migration, I’ll just create a VirtualMachineInstanceMigration object so we can better monitor the states and time, the yaml file looks like this:

When the migration is done, let’s check the location of the VM:

Now let’s take a look first at how long it took the VM to move:

We can see that it took around 4 seconds to migrate the VM from the “Scheduling” state to “Running” state and 16 seconds to the “Succeeded” state.

Let’s look now at the fio numbers, and compare the same run of 120 seconds, using 70%read / 30%write random IOs:

We can see (as expected), a tiny decrease in IOPS and a tiny bump in latency, between a regular run of fio (VM is not migrating) and a run that continuously sends IOs to the Lightbits storage during the live migration.

Summary:

VMs are a major component of the modern data center and are not going away anytime soon. Red hat OpenShift Virtualization makes it easier for organizations to manage both VMs and containers. 

Lightbits’ performance and cost-effectiveness make it an ideal storage layer for handling any kind of application IO traffic, whether for pods or VMs. 

We showed here how easy, it can be to migrate a VM with several disk drives using Red Hat OpenShift Virtualization and Lightbits storage and the minimized impact of running IOs while the VM is migrating from one worker node to another.

* Demo used in a small development environment to show the relative impact of IOs and Latency during migration.