By Calvin Zito, @HPStorageGuy
Last week I found a white paper talking about VAAI on the P9500 and wrote a blog post based on it. Today, I found a whitepaper titled "Reduce resource and storage bandwidth consumption - simplify and significantly" with a subtitle of "VMware vSphere VAAI for the HP 3PAR Storage performance benefits". It has a great introduction to the VAAI primitives that can help improve performance so I thought I'd reproduce that for you on this blog post.
Introduction to VAAI
The vSphere Storage APIs for Array Integration (VAAI) are one of the storage application programming interface (API) sets in vSphere 5.0. VAAI is an API that storage partners can leverage to enhance performance of virtual machine (VM) management operations by delegating these operations to the storage array. With hardware offload, ESXi hosts may perform certain operations faster and consume less host CPU and memory resources, storage fabric, and network bandwidth. VAAI includes high performance and scalable VM data path primitives. HP introduced VAAI support for the HP 3PAR Storage products starting with the HP 3PAR operating system(formerly InForm OS) v3.1.1 or 2.3.1 firmware release.
Storage hardware primitives for VAAI
Primitives are specific functions used with VAAI that serve as integration points to storage arrays. VMware develops the primitives and array vendors integrate them into their arrays to support them. Primitives in VAAI allow the hypervisor to directly communicate with storage arrays in a way that they both understand. Their purpose is to allow close integration between the hypervisor and storage array as well as to offload storage functionality traditionally handled by the hypervisor. Storage arrays can handle these functions more intelligently and efficiently as they are purpose built to perform storage tasks and can complete the request much faster than the host.
VAAI uses these fundamental primitives which were first introduced in vSphere 4.1:
- Hardware-assisted locking: Also known as Atomic Test and Set (ATS), this primitive provides a method to protect metadata for VMware Virtual Machine File System (VMFS) cluster file systems, reducing Small Computer System Interface (SCSI) reservation contention between vSphere hosts.
- Copy offload: This primitive is used to copy virtual machine disk (VMDK) data. This enables full copies of data to be made within the storage array, reducing data reads/writes required on the part of the vSphere host.
- Block Zeroing: This primitive allows the array to handle the process of zeroing all the disk blocks instead of having the host wait for the operation to complete. The array simply signals that the operation has been completed immediately and handles the process on its own without involving the vSphere host.
Additional primitives were introduced in vSphere 5 that are focused on enhancing thin provisioning. These primitives do not impact performance so they are not covered in this paper.
Hardware-accelerated full copy
Full Copy enables storage arrays to make copies of VM data within the array by using the EXTENDED COPY SCSI command. Using the Full Copy primitive the vSphere host does not have to read VM data from the array and then write the data back to the array which results in significant network overhead.
Figure 1. Diagram representing old method before VAAI on the left and Full Copy offload method with VAAI on the right.
vSphere host involved in data movement when cloning VMs or using Storage vMotion must copy data back and forth between the host and array which results in significant network overhead. Diagram on the right represents Full Copy offload available with VAAI, now all data movement occurs within the array and does not involve the host.
Copy operations that would normally take minutes to complete without VAAI may now be accomplished much faster, due to the reduced traffic between the vSphere host and the storage array. Operations such as creating VM clones, deploying VMs from template, and Storage vMotion can be improved as follows:
- Reduce the time for many vSphere storage-related tasks by 25 percent or more (in some cases up to 10x).
- Reduce the amount of CPU load on the vSphere host and the array by 50 percent or more (in some cases up to 10x).
- Reduce the traffic on the storage network by 99 percent.
Hardware-accelerated Block Zeroing
VMware supports several options of storage space allocation during creation of new virtual machines or virtual disks. Eager zeroed thick (EZT) disks are often used as they provide performance and security benefits over lazy-zeroed thick disks whose blocks are zeroed on-demand the first time a block has data written to it. When utilizing the EZT format during virtual disk creation, all the space is pre-allocated and pre-zeroed which can be both time consuming and resource intensive. The Block Zeroing primitive uses the SCSI WRITE_SAME command, which enables storage arrays to zero out a large number of blocks to speed up the provisioning of VMs. It eliminates redundant and repetitive write commands from the vSphere host by offloading the zeroing task to the array.
Figure 2. Diagram representing old method before VAAI on the left and Block Zeroing offload method available with VAAI on the right.
Block Zeroing provides significant performance improvement during initialization of EZT virtual disks. It also provides better security as previously written disk blocks that contain deleted data that may contain sensitive data are not exposed to new VMs. Using the VAAI features to do hardware-assisted zeroing at the array reduces CPU and I/O storage resource needs from the host and improves performance.
To assure consistency in clustered environments where multiple hosts have access to the same volumes, earlier versions of vSphere relied upon a method of mutual exclusion called SCSI reservation/release. With this model, a host would place a lock on an entire LUN, which prevented other hosts from writing to the locked LUN and VMFS volume until the lock was released. With the SCSI reservation, a host could update metadata on the VMFS volume exclusively for VM operations that require it, without the risk of another host’s interference.
Hardware-assisted locking uses the COMPARE AND WRITE SCSI command, which provides an alternative means to protect the metadata for VMFS cluster file systems and thereby improves the scalability of large vSphere host farms sharing a datastore. The ATS capability allows locking at the block level of a LUN by a vSphere host instead of locking the whole LUN (a SCSI reservation). Only the VMDK files of a VM are locked for use by the host, leaving the remainder of the LUN unlocked for other hosts. This approach resolves contention amongst vSphere hosts accessing the same VFMS volume on the same LUN.
Figure 3. Diagram representing old method before VAAI on the left and hardware-assisted locking method available with VAAI on the right.
The following are examples of operations that require locking metadata:
- Creating a VMFS datastore
- Expanding a VMFS datastore onto additional extents
- Changing the power state of a virtual machine
- Acquiring a lock on a file Creating or deleting a file Creating a template
- Deploying a virtual machine from a template Creating a new virtual machine
- Migrating a virtual machine with vMotion
- Growing a file, such as, a Snapshot file or a thin provisioned virtual disk
- Hardware-assisted locking can both improve performance as hosts no longer have to wait for locks to be released and improve VM density as more VMs can be put on a single LUN without having to worry about excessive locking happening.
Benefits of VAAI
Storage related operations in a VMware environment can be very I/O intensive, which can overburden a host and take away resources from virtual machines. The VAAI helps to create a separation of duty between the hypervisor and its storage devices and allows each to focus on what they do best, which is virtualization-related tasks for the hypervisor and storage-related tasks for the storage arrays. The hypervisor traditionally handles all high-level storage related tasks, for example when you clone a virtual machine the host has to use its own resources to perform the file copy. This means the host would have to access the source file on the storage array to read it and then copy all of the data through the host and back to the array to write it.
VAAI allows certain storage tasks like cloning to be offloaded to the storage array instead which can complete them more efficiently than the host can. So rather than using the host resources to clone a VM, the host can simply pass the task onto the storage array, which will perform the operation while the host monitors the progress of the task. Reducing host resource consumption and overhead associated with common storage-related maintenance tasks is not the only benefit of VAAI; the storage array is purposely built to perform storage tasks and can complete the request much faster than the host could complete it. As a result the benefits are twofold as the task completes quicker while reducing the burden on the host.
Overall performance improvements are not the only benefit provided by the VAAI primitives. VAAI also allows for higher storage scalability as well as better integration between the host and storage array. With the new locking mechanism provided by ATS, the barriers that limited VM density on a LUN are gone, which allows for increased efficiency. VAAI also allows the host and storage array to better communicate with each other, which creates a stronger relationship between them and allows them to work more effectively.
VAAI helps reduce the storage bandwidth consumed by a host and improves overall scalability of the vSphere environment. Storage operations like VM provisioning, Storage vMotion, creation of virtual disks, and so on will consume less resources and network bandwidth when using the HP 3PAR array running firmware which supports VAAI.
Limitations and conditions of VAAI
To move and copy VM data both within and across datastores, the VM kernel uses a software component called the VMFS DataMover. The VMFS DataMover enables the cloning and initialization of VM data to provide high data throughput without moving the data between the kernel and application levels. The VMFS DataMover is designed to work with VAAI and will automatically offload the data movement to the storage hardware. However, in certain cases the VMFS DataMover does not leverage hardware offloads and instead uses software data movement instead. These cases include:
- The source and destination VMFS volumes have different block sizes.
- The source file type is raw device mapping (RDM) and the destination file type is non-RDM (regular file).
- The source VMDK type is eagerzeroedthick and the destination VMDK type is thin.
- The source or destination VMDK is in any sort of sparse or hosted format.
- The source VM has a snapshot.
- The logical address and transfer length in the requested operation are not aligned to the minimum alignment required by the storage device (all datastores that are created with the vSphere Client are aligned automatically).
- The VMFS has multiple LUNs/extents and they are all on different arrays.
- Hardware cloning between different arrays (even if within the same VMFS volume) does not work.
Help me HPStorageGuy - where do I go from here?
From here, the paper goes on to talk about how to enable VAAI with HP 3PAR, the test set-up and lots of comparative results. Read the whitepaper to get HP 3PAR with VMware VAAI configuration and performance details. I highly recommend it!
To date, I have 43 blog posts that I've labeled with 3PAR - some are high level, others include video demos, others are podcasts. If you're new to 3PAR, this is a great way to learn about HP 3PAR.
If you have suggestions about future Around the Storage Block topics, send me an email. Thanks for reading.