Introduction

I migrated >120 VMware virtual machines (Linux and Windows) from VMware ESXi to OpenStack. In a lab environment I also migrated from Hyper-V with these steps. Unfortunately I am not allowed to publish the script files I used for this migration, but I can publish the steps and commands that I used to migrate the virtual machines. With the steps and commands, it should be easy to create scripts that do the migration automatically.

Just to make it clear, these steps do not convert traditional (non-cloud) applications to cloud ready applications. In this case we started to use OpenStack as a traditional hypervisor infrastructure.

Update 9 September 2015: The newer versions of libguestfs-tools and qemu-img convert handle VMDK files very well (I had some issues with older versions of the tools), so the migration can be more efficient. I removed the conversion steps from VMDK to VMDK (single file) and from VMDK to RAW. The migration speed will be doubled by reducing these steps.

Disclaimer: This information is provided as-is. I will decline any responsibility caused by or with these steps and/or commands. I suggest you don’t try and/or test these commands in a production environment. Some commands are very powerful and can destroy configurations and data in Ceph and OpenStack. So always use this information with care and great responsibility.

Global steps 

1. Inject VirtIO drivers
2. Expand partitions (optional)
3. Customize the virtual machine (optional)
4. Create Cinder volumes
5. Convert VMDK to Ceph
6. Create Neutron port (optional)
7. Create and boot instance in OpenStack

Specifications

Here are the specifications of the infrastructure I used for the migration:

• Cloud platform: OpenStack Icehouse
• Cloud storage: Ceph
• Windows instances: Windows Server 2003 to 2012R2 (all versions, except Itanium)
• Linux instances: RHEL5/6/7, SLES, Debian and Ubuntu
• Only VMDK files from ESXi can be converted, I was not able to convert VMDK files from VMware Player with qemu-img
• I have no migration experience with encrypted source disks
• OpenStack provides VirtIO paravirtual hardware to instances

Requirements

A Linux ‘migration node’ (tested with Ubuntu 14.04/15.04, RHEL6, Fedora 19-21) with:

• Operating system (successfully tested with the following):

• RHEL6 (RHEL7 did not have the “libguestfs-winsupport” -necessary for NTFS formatted disks- package available at the time of writing)
• Fedora 19, 20 and 21
• Ubuntu 14.04 and 15.04

• Network connections to a running OpenStack environment (duh). Preferable not over the internet, as we need ‘super admin’ permissions. Local network connections are usually faster than connections over the internet.
• Enough hardware power to convert disks and run instances in KVM (sizing depends on the instances you want to migrate in a certain amount of time).

We used a server with 8x Intel Xeon E3-1230 @ 3.3GHz, 32GB RAM, 8x 1TB SSD and we managed to migrate >500GB per hour. However, it really depends on the usage of the disk space of the instances. But also my old company laptop (Core i5 and 4GB of RAM and an old 4500rmp HDD) worked, but obviously the performance was very poor.

• Local sudo (root) permissions on the Linux migration node
• QEMU/KVM host
• Permissions to OpenStack (via Keystone)
• Permissions to Ceph

• Unlimited network access to the OpenStack API and Ceph (I have not figured out the network ports that are necessary)
• VirtIO drivers (downloadable from Red Hat, Fedora, and more)
• Packages (all packages should be in the default distributions repository):

“python-cinderclient” (to control volumes)

“python-keystoneclient” (for authentication to OpenStack)

“python-novaclient” (to control instances)

“python-neutronclient” (to control networks)

“python-httplib2” (to be able to communicate with web service)

“libguestfs-tools” (to access the disk files)

“libguestfs-winsupport” (should be separately installed on RHEL based systems only)

“libvirt-client” (to control KVM)

“qemu-img” (to convert disk files)

“ceph” (to import virtual disk into Ceph)

“vmware-vdiskmanager” (to expand VMDK disks, downloadable from VMware)

Steps

1. Inject VirtIO drivers

1.1 Windows Server 2012

Since Windows Server 2012 and Windows 8.0, the driver store is protected by Windows. It is very hard to inject drivers in an offline Windows disk. Windows Server 2012 does not boot from VirtIO hardware by default. So, I took these next steps to install the VirtIO drivers into Windows. Note that these steps should work for all tested Windows versions (2003/2008/2012).

1. Create a new KVM instance. Make sure the Windows vmdk disk is created as IDE disk! The network card should be a VirtIO device.
2. Add an extra VirtIO disk, so Windows can install the VirtIO drivers.
3. Off course you should add a VirtIO ISO or floppy drive which contains the drivers. You could also inject the driver files with virt-copy-in and inject the necessary registry settings (see  paragraph 4.4) for automatic installation of the drivers.
4. Start the virtual machine and give Windows about two minutes to find the new VirtIO hardware. Install the drivers for all newly found hardware. Verify that there are no devices that have no driver installed.
5. Shutdown the system and remove the extra VirtIO disk.
6. Redefine the Windows vmdk disk as VirtIO disk (this was IDE) and start the instance. It should now boot without problems. Shut down the virtual machine.

1.2 Linux (kernel 2.6.25 and above)

Linux kernels 2.6.25 and above have already built-in support for VirtIO hardware. So there is no need to inject VirtIO drivers. Create and start a new KVM virtual machine with VirtIO hardware. When LVM partitions do not mount automatically, run this to fix:

(log in)

mount -o remount,rw /

pvscan

vgscan

reboot

(after the reboot all LVM partitions should be mounted and Linux should boot fine)

Shut down the virtual machine when done.

1.3 Linux (kernel older than 2.6.25)

Some Linux distributions provide VirtIO modules for older kernel versions. Some examples:

• Red Hat provides VirtIO support for RHEL 3.9 and up
• SuSe provides VirtIO support for SLES 10 SP3 and up

The steps for older kernels are:

1. Create KVM instance:
2. Linux (prior to kernel 2.6.25): Create and boot KVM instance with IDE hardware (this is limited to 4 disks in KVM, as only one IDE controller can be configured which results in 4 disks!). I have not tried SCSI or SATA as I only had old Linux machines with no more than 4 disks. Linux should start without issues.
3. Load the virtio modules (this is distribution specific): RHEL (older versions): https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/ch10s04.html) and for SLES 10 SP3 systems: https://www.suse.com/documentation/opensuse121/book_kvm/data/app_kvm_virtio_install.htm
4. Shutdown the instance.
5. Change all disks to VirtIO disks and boot the instance. It should now boot without problems.
6. Shut down the virtual machine when done.

For Red Hat, see: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/ch10s04.html

For SuSe, see: https://www.suse.com/documentation/opensuse121/book_kvm/data/app_kvm_virtio_install.htm

1.4 Windows Server 2008 (and older versions); deprecated

For Windows versions prior to 2012 you could also use these steps to insert the drivers (the steps in 4.1 should also work for Windows 2003/2008).

1. Copy all VirtIO driver files (from the downloaded VirtIO drivers) of the corresponding Windows version and architecture to C:\Drivers\. You can use the tool virt-copy-in to copy files and folders into the virtual disk.
2. Copy *.sys files to %WINDIR%\system32\drivers\ (you may want to use virt-ls to look for the correct directory. Note that Windows is not very consistent with lower and upper case characters). You can use the tool virt-copy-in to copy files and folders into the virtual disk.
3. The Windows registry should combine the hardware ID’s and drivers, but there are no VirtIO drivers installed in Windows by default. So we need to do this by ourselves. You could inject the registry file with virt-win-reg. If you choose to copy all VirtIO drivers to an other location than C:\Drivers, you must change the “DevicePath” variable in the last line (the most easy way is to change it in some Windows machine and then export the registry file, and use that line).

Registry file (I called the file mergeviostor.reg, as it holds the VirtIO storage information only):

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\CriticalDeviceDatabase\pci#ven_1af4&dev_1001&subsys_00000000]
 "ClassGUID"="{4D36E97B-E325-11CE-BFC1-08002BE10318}"
 "Service"="viostor"

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\CriticalDeviceDatabase\pci#ven_1af4&dev_1001&subsys_00020000]
 "ClassGUID"="{4D36E97B-E325-11CE-BFC1-08002BE10318}"
 "Service"="viostor"

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\CriticalDeviceDatabase\pci#ven_1af4&dev_1001&subsys_00021AF4]
 "ClassGUID"="{4D36E97B-E325-11CE-BFC1-08002BE10318}"
 "Service"="viostor"

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\CriticalDeviceDatabase\pci#ven_1af4&dev_1001&subsys_00021AF4&rev_00]
 "ClassGUID"="{4D36E97B-E325-11CE-BFC1-08002BE10318}"
 "Service"="viostor"

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Control\CriticalDeviceDatabase\pci#ven_1af4&dev_1004&subsys_00081af&rev_00]
 "ClassGUID"="{4D36E97B-E325-11CE-BFC1-08002BE10318}"
 "Service"="viostor"

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\Services\viostor]
 "ErrorControl"=dword:00000001
 "Group"="SCSI miniport"
 "Start"=dword:00000000
 "Tag"=dword:00000021
 "Type"=dword:00000001
 "ImagePath"="system32\\drivers\\viostor.sys"

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion]
 "DevicePath"=hex(2):25,00,53,00,79,00,73,00,74,00,65,00,6d,00,52,00,6f,00,6f,00,74,00,25,00,5c,00,69,00,6e,00,66,00,3b,00,63,00,3a,00,5c,00,44,00,72,00,69,00,76,00,65,00,72,00,73,00,00,00

When these steps have been executed, Windows should boot from VirtIO disks without BSOD. Also all other drivers (network, balloon etc.) should install automatically when Windows boots.

See: https://support.microsoft.com/en-us/kb/314082  (written for Windows XP, but it is still usable for Windows 2003 and 2008).

See also: http://libguestfs.org/virt-copy-in.1.html and http://libguestfs.org/virt-win-reg.1.html

2. Expand partitions (optional) 

Some Windows servers I migrated had limited free disk space on the Windows partition. There was not enough space to install new management applications. So, I used the vmware-vdiskmanager tool with the ‘-x’ argument (available from VMware.com) to increase the disk size. You then still need to expand the partition from the operating system. You can do that while customizing the virtual machine in the next step.

3. Customize the virtual machine (optional) 

To prepare the operating system to run in OpenStack, you probably would like to uninstall some software (like VMware Tools and drivers), change passwords and install new management tooling etc.. You can automate this by writing a script that does this for you (those scripts are beyond the scope of this article). You should be able to inject the script and files with the virt-copy-in command into the virtual disk.

3.1 Automatically start scripts in Linux

I started the scripts within Linux manually as I only had a few Linux servers to migrate. I guess Linux engineers should be able to completely automate this.

3.2 Automatically start scripts in Windows

I choose the RunOnce method to start scripts at Windows boot as it works on all versions of Windows that I had to migrate. You can put a script in the RunOnce by injecting a registry file. RunOnce scripts are only run when a user has logged in. So, you should also inject a Windows administrator UserName, Password and set AutoAdminLogon to ‘1’. When Windows starts, it will automatically log in as the defined user. Make sure to shut down the virtual machine when done.

Example registry file to auto login into Windows (with user ‘Administrator’ and password ‘Password’) and start the C:\StartupWinScript.vbs.:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnce]
 "Script"="cscript C:\\StartupWinScript.vbs"
 "Parameters"=""

[HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon]
 "AutoAdminLogon"="1"
 "UserName"="Administrator"
 "Password"="Password"

4. Create Cinder volumes

For every disk you want to import, you need to create a Cinder volume. The volume size in the Cinder command does not really matter, as we remove (and recreate with the import) the Ceph device in the next step. We create the cinder volume only to create the link between Cinder and Ceph.

Nevertheless, you should keep the volume size the same as the disk you are planning to import. This is useful for the overview in the OpenStack dashboard (Horizon).

You create a cinder volume with the following command (the size is in GB, you can check the available volume types by cinder type-list):

cinder create --display-name <name_of_disk> <size> --volume-type <volumetype>

Note the volume id (you can also find the volume id with the following command) as we need the ids in the next step.

cinder list | grep <name_of_disk>

Cinder command information: http://docs.openstack.org/cli-reference/content/cinderclient_commands.html

5. Convert  VMDK to Ceph

As soon as the Cinder volumes are created, we can convert the VMDK disk files to RBD blocks (Ceph). But first we need to remove the actual Ceph disk. Make sure you remove the correct Ceph block device!

In the first place you should know in which Ceph pool the disk resides. Then remove the volume from Ceph (the volume-id is the volume id that you noted in the previous step ‘Create Cinder volumes’):

rbd -p <ceph_pool> rm volume-<volume-id>

Next step is to convert the VMDK file into the volume on Ceph (all ceph* arguments will result in better performance. The vmdk_disk_file variable is the complete path to the vmdk file. The volume-id is the ID that you noted before).

qemu-img convert -p <vmdk_disk_file> -O rbd rbd:<ceph_pool>/volume-<volume-id>

Do this for all virtual disks of the virtual machine.

Be careful! The rbd command is VERY powerful (you could destroy more data on Ceph than intended)!

6. Create Neutron port (optional) 

In some cases you might want to set a fixed IP-address or a MAC-address. You can do that by create a port with neutron and use that port in the next step (create and boot instance in OpenStack).

You should first know what the network_name is (nova net-list), you need the ‘Label’. Only the network_name is mandatory. You could also add security groups by adding

 --security-group <security_group_name>

Add this parameter for each security group, so if you want to add i.e. 6 security-groups, you should add this parameter 6 times.

neutron port-create --fixed-ip ip_address=<ip_address> --mac-address <mac_address> <network_name> --name <port_name>

Note the id of the neutron port, you will need it in the next step.

7. Create and boot instance in OpenStack 

Now we have everything prepared to create an instance from the Cinder volumes and an optional neutron port.

Note the volume-id of the boot disk.

Now you only need to know the id of the flavor you want to choose. Run nova flavor-list to get the flavor-id of the desired flavor.

Now you can create and boot the new instance:

nova boot <instance_name> --flavor <flavor_id> --boot-volume <boot_volume_id> --nic port-id=<neutron_port_id>

Note the Instance ID. Now, add each other disk of the instance by executing this command (if there are other volumes you want to add):

nova volume-attach <instance_ID> <volume_id>