Around the Storage Block Blog
Find out about all things data storage from Around the Storage Block at HP Communities.

HP 3PAR - Fit for VMware environments

Andre Carpenter.jpgBy Andre Carpenter, Senior Solution Architect in Australia

(Editor's note: I did a podcast introducing Andre while at VMworld San Francisco)

 

My colleague Calvin Zito wrote a great blog recently talking about why he thought 3PAR is awesome for VMware.  I agree with him and wanted to share a deep dive post on why I agree.

 

The traditional RAID era and how the storage world has changed.

I started my storage career doing design and implementation services in this era; not all RAID groups were created equal, and it seems (at least to me) that more thought and planning had to be put in with traditional RAID because the arrays weren’t as capable and smart as they are today.  There was no concept of automated storage tiering or shared storage pools for example.

 

The spindle count in RAID groups were calculated by storage architects based on host IOP workload requirements.  There was no real concept of throwing all spindles into one big “pool”, and then carving and provisioning storage from that pool.

 

The architecture is similar to this example image depicting the traditional era; each RAID group was more or less dedicated to one particular workload.

 

2.png 

 

Things have changed since then, thus the concept of shared pools or shared storage was born; this was to drive initiatives like cloud computing, deduplication (if your array supported it natively), and storage tiering amongst other things. By having this shared pool of resources, workloads were “spread out” across the storage resources thus generating a bigger pool of grunt to draw from.

 

HP 3PAR does this in the form of wide striping, breaking storage down into “chunklets”.

 

Chunklets

1.pngThe term chunklets may sound like some sort of breakfast cereal, but although not of the food variety the concept definitely still holds some nutritional value for your storage requirements.  Here’s how they work:

 

  • An HP 3PAR array is populated with one or more disk types; these can be either Fibre Channel, SATA, or SSD.  In order to provision storage from these drives to a host, there needs to be a Common Provisioning Group (CPG) created; this serves as a template for creating LUNs.  Typically, the CPG needs to be of the same disk type and the same RAID characteristics.
  • From there, LUNs can be created and provisioned to the host.  When ESXi hosts starts storing virtual machine data - whether its virtual disks data or meta data to the LUN - each physical drive is broken down into 256 MB chunklets that the LUNs can use to store the data.  One point to note is that there is also chunklets for distributed sparing as well.

As an example, for a single 600Gb drive you will have 2400 chunklets at your disposal for virtual machine use (600Gb*1024Mb/256Mb).  When you add more shelves of drives, the picture gets bigger as does the performance.

 

Wide Striping

From physical disks right through to the LUNs that are provisioned to the ESXi host, the result is that the chunklets are created across all of the spindle types in the array as defined in the CPG. This system wide allocation super charges performance for virtual workloads.  

 

3.png

 

Ready for provisioning to the host!  

 

4.png

 

 

The result is an architecture that is optimal and balanced for mixed workloads, including virtual environments with their own mixed workloads within.

 

Multi Raid? Sure!

One hard question as a storage architect to answer is “what type of RAID shall I use for this virtual environment?”. This question is typically answered with the usual “It depends” response.  Different workloads call for different strategies as different RAID types have different RAID penalties\performance considerations.

 

There is a consensus in the industry to consider the following rules of thumbs (these are only rules of thumb and are not best practices in any form):

 

  • RAID 1/0 – Usually higher write intensive random workloads suit this.
  • RAID 5 – Arguably one of the best all-rounders, offering a good balance of performance and redundancy. Modest random workloads are a good fit.
  • RAID 6 – HP 3PAR offers double parity protection in the form of RAID-MP, offering a higher redundancy (double failure) than RAID 5 but at the cost of usable storage and performance because of the added write penalty.

 

It is important that all crucial data is protected; this includes virtual machines that provide various services for your organisation. In HP 3PAR, RAID protection is achieved by striping the data (whether RAID 5, RAID 6 or RAID 1/0) across chunklets that are spread across multiple disks, across disk magazines and across shelves in the array, so that your data can survive a failure of any of these components.  VMware environments rely on a solid robust RAID architecture to provide the redundancy for the virtual machines, features such as VMware HA, VMware Fault Tolerance and even DRS features all benefit from well-architected RAID platform.

 

Regardless of which RAID type is used, making a write I/O takes time. The quicker the write is made, the better the latency and throughput and the less write penalty is observed.

 

Gen4 ASIC to super charge performance and efficiency

HP 3PAR has a secret sauce to handle storing data fast.  Each HP 3PAR node has two Gen4 ASIC with the ability to do thin conversion data on the fly.

 

This architecture steers the workload away from the controller node to avoid driving the CPU usage up trying to convert and write data, allowing the nodes to focus on serving data and delivering high service levels rather than converting.  The VMware VAAI primitive Zero Blocks/Write same can also utilize this resulting in a true thin environment!

 

VMware eager zero thick (EZT) datastores thrive on this technology.  An example is that EZT’s are required for VMware Fault Tolerance; traditionally thick in nature, moving to 3PAR offers a unique way of “thinning” those EZT datastores without the hypervisor even knowing!

 

Dynamic Optimisation (DO) and Adaptive Optimisation (AO) – Where hot is hot and cold is cold.

Here is something really cool: virtual workloads have different performance requirements with some VM’s demand more resources and performance than others – that’s a known fact.  What 3PAR can do with these variety of workloads is offer an automated tiering function which comes in two flavours.

 

Dynamic Optimisation (DO) is the automated tiering of the whole datastore or LUN, so when the virtual machines that the datastore houses become busy or “hot” then the LUN is promoted into a faster tier.  This is similar to VMware Storage DRS but the notable difference here is that the array moves the data not the ESXi host.

 

Adaptive optimization (AO) goes one step further and moves the blocks within the datastore (as opposed to the whole LUN with DO) up and down the tiers.

 

So VMware datastores now become balanced across different tiers, hot blocks exist in faster storage tiers and cold blocks reside on slower disk all with the help of HP System Reporter, which acts as the brains behind the magic.

 

5.png

 

The end result is that your data gets automagically spread across all disks and all disk types in the 3PAR, with hot regions on fast disks and cold data on slow disks. The whole performance capability of the entire array is made available to all of your data automatically; this is how virtual workloads should be stored!

 

In Closing.

Here's the key takeaway to remember for this architecture built for VMware.

 

The main contributor of an array’s performance is determined by how many disks of each disk type is installed in the array, the more drives you have in the CPG then the more throughput and overall IOPS is available to all of your VMFS datastores and subsequently your virtual machine workloads. With HP 3PAR architecture and functions such as ASIC, DO, AO and wide striping the array works out the best way to store your virtual workloads driving up performance of the workloads while keeping your environment thin and efficient.  And that’s why I firmly believe that HP 3PAR architecture is fit for VMware environments.

Labels: 3PAR| VMware
Comments
nate | ‎10-23-2012 07:33 AM

[INCOMING EMERGENCY ACTION MESSAGE]

 

Nice but still forgot some things! hmm where to begin - I don't know how to remove the stupid smiley faces

 

In no particular order -

  • Chunklets provide for parallel RAID rebuilds (means faster recovery from failure, and in general lower latency during rebuilds). This extends the life of RAID 5 by a lot in my opinion. I wrote a big blog post on this topic a couple of years ago. Think of the math probabilities of suffering a double disk failure in something that can recover in a matter of hours vs suffering a triple disk failure in something that takes many days to a week to recover. Add in cage level availability (below)..
  • No dedicated disks for parity
  • No dedicated "hot spare" disks
  • Ability to go above & beyond "reserved" spare capacity in the event of multiple disk failures - the array reserves a set amount of space on each spindle (customizable), if for some reason that is exhausted the array will (assuming capacity is available) dip into the free space on the disks in the event  you don't replace disks fast enough. So you don't have to worry  - in many cases if you suffer multiple disk failures - about running out of "hot spares" (or the 3PAR equivilent of them)
  • 3PAR by default builds for "cage level availability" meaning you can lose an entire shelf - and in RAID 6+2 (assuming you have 8 shelves) you can lose 2 shelves w/o data loss  (though I/O would be painful I bet). On the big systems that is up to 80 drives!
  • 3PAR chunklet RAID allows for very low data:smileytongue:arity ratios - RAID 5 3+1 for example is extremely common on 3PAR - I doubt you'll see it anywhere else. By contrast you won't see RAID 5 12+1 or something crazy like 24+2 on 3PAR. Low data:smileytongue:arity ratios provide for closer to RAID 10 performance without the space overhead. With the chunklet design you don't waste spindles for parity since there are no parity spindles only parity chunklets!
  • On the topic of parity - of course truely distributed parity, as in if you do run RAID 5 or 6 you will have parity chunklets running on every disk in your box.
  • Oh! Don't forget that the new V-series that came out last year uses 1GB chunklets
  • Adaptive optimization operates on regions (I think that's what they are called) - which are 128MB instead of the 256MB (or 1GB depending on platform) chunklet size
  • Good to clarify the Gen4 ASIC is only on the V class - the T and F which are still being sold last I checked - are using Gen3, single ASIC/controller on Gen3.
  • Provisioning from storage pools to volumes occur in 16kB chunks, though physical provisioning to storage pools occurs in much larger increments (as large as 128GB in an 8-node system)
  • Good to point out that you can run multiple RAID levels on the same physical spindles simultaneously
  • You do have control over where your data is placed (inside of the disc track vs outside - outside is a bit faster), I set all my volumes by default to be the inside(NOT the default), except in rare cases where I feel performance of some volume is better suited to the outside.
  • While many do run multiple tiers on their 3PAR - I never have. Everything on each of my arrays has always been a single tier. Stripe it wide, if I want more IOPS I add more disks, when I add more disks I get more space for "free" which I can then use to stuff more cold data onto the system. The limited tiering I have done has been different RAID levels primarily (+ innner vs outer data placement on the platters)
  • 3PAR architecture is active-active, this is really important. I/O can come in on any controller, on any HBA, it doesn't matter. No manual load balancing by putting some volumes on controller A and others on B. When you add more disks you can maintain your existing RAID configuration - you don't have to for example expand your data:smileytongue:arity ratio.
  • 4th generation design (at least). I've been using 3PAR since the 2nd generation, so it is a mature platform, not something that just came out of someone's garage last week.
  • I think 3PAR should get rid of Logical disks in their presentations. I always forget about those things, the user never interacts with them, it's just an extra level of complexity that can confuse people.
  • How could you forget about persistent cache! In any sort of cloud setup, you really need to have persistent cache to eliminate that write-through performance hit when a controller goes down (scheduled or not). How many other architectures do this ? Yeah not many.
  • The vmware software for 3PAR still sucks..hopefully it will get better. I want to see things like auto detection of 3PAR resources, auto checking of MPIO/queue depth/iSCSI/other types of settings to make sure it's properly configured, I've seen some pretty neat stuff from NetApp and I think EMC that does this kind of thing. (3PAR has to take the good with the bad!)

By contrast my first "real" storage system was a 3PAR. Before that I wasn't involved in storage, I did servers, networking, security those types of things. I saw the storage people working on EMC/HDS equipment with visio diagrams and excel spreadsheets and I told myself I was NOT interested in storage. Later came some folks that were so happy about NetApp and their auto expanding thingamajig, easy to use, etc. Though I later learned how defficient their architecture is, no active-active (even in their latest kit), aggregates are still very limiting, RAID 6 forced on everything regardless, could go on and on. What got me on 3PAR gear originally though is NetApp refused to give me a system to evaluate. 3PAR was more kind, and fortunately it worked out for the best over the longer term.

 

One time I ran DO on a busy system 24 hours a day 7 days a week for four months to re-lay out a ton of data. Of course nobody notice it was running. It was quite a optimization process.

 

I'm sure I've forgotten some things, I've written so much about 3PAR over the years sometimes I lose track.

 

Anyways all 3PAR needs now is tight integration with flash, sub LUN auto tiering (AO) is alright, but insufficient in my mind. I love 3PAR but that is one area that the archiecture/hardware really needs work. DON'T LET ME DOWN.

 

Repeat after me - AO IS NOT ENOUGH.

 

We now take you back to your regularly scheduled programming.

 

 

nate | ‎10-23-2012 07:39 AM

One more note - the 3PAR F400 is the most efficient system out there, at least for SPC-1. It even blows the doors off of the V800.

 

By my calculations the F400 had an unused storage ratio of 0.133% vs the V800 which came in at 14.53% (still not bad, but it's no F400!)

 

Can you build a storage system and run SPC-1 on it and get good results with greater than 99.86% storage utilization ? I think you'd be hard pressed!

 

 

 

 

| ‎10-23-2012 03:07 PM

Nate - wow, I need to get you to write a guest blog post or do a podcast with you (but I knew that already) - I think I need to turn your comment into a guest blog post!  I also love your :smileytongue: faces.  I'm sure you were trying to say "data-to-parity ratios but our blog platform interprets a : and p squished together as a sticking your tongue out smiley.  Thanks for taking your real-world-multi-year experience with 3PAR into more points about the special sauce in 3PAR. 

 

I think you know that several of the guys that were at 3PAR were in HP Storage in the early 2000's.  They had called me and asked me to come to 3PAR but I put my love of Boise (and inability to afford housing in the Bay area) above going to a start-up.  I still don't regret that decision but was always secretly rooting that they'd do well.  I was at VMworld 2010 when HP won the bidding war and was doing my happy dance.  And funniest thing was that Craig Nunes was my boss when he left HP and became my boss when he came back.  The guy just can't get away from me!

 

Thanks again for your thoughts - good stuff!

John_H(anon) | ‎10-23-2012 09:40 PM

For Nate, some additional HP 3PAR & VMware integration points using HP Insight Control Storage module, gave it a test drive today and it's particularly useful if you're running different flavours of HP storage.

 

http://h20195.www2.hp.com/V2/GetPDF.aspx/4AA3-9970ENW.pdf

http://h20195.www2.hp.com/V2/GetPDF.aspx/4AA1-0290ENW.pdf

andre | ‎10-30-2012 02:59 AM

[INCOMING EMERGENCY RESPONSE MESSAGE]

 

Firstly, thank you for reading my blog post!  You have highlighted some very good points and elaborated further on why 3PAR is great in general not just for VMware environments!

 

My post was written with VMware environments in mind, but that said the offer of multi-raid,full mesh capabilities that any host can leverage is market leading in my opinion, there are other storage vendors that lock you in to specific RAID-types as you say.  

 

All RAID types are not created equal, so the method we use to carve up storage into chunklets allow better flexibility around placing data, and other elements such as rebuilding from failed drives. 

 

You made a great point on drive rebuilds, drive reconstructing has always been something that I believe plagued other storage vendors in the market, rebuild times were slow and/or generally affected performance when there was drive levelling to be done.. 

 

r.e AO is not enough - Perhaps not, but it certainly does take a lot of the thinking process out of the architect's hands and offers a more granular approach to VMware's storage DRS which is VM tiering and host based rather than block. (I don't think these two should be run in conjunction with each other just for the record)

 

I cannot comment on whether or not the VMware integration software we offer will get "better" as I have not seen the road map but I would hope that any vendor would try and make their products better than the last release.  

 

VAAI integration is personally something that I am looking forward to watching the developments - generating synergy between a server's compute capability and HP 3PAR functionality will be amazing. Watch this space!

 

Going back to my closing point in my post - performance for the most part comes back to the spindles. HP 3PAR just gives you more control, more smarts around what you can do with those spindles.

 

Thanks for your comment and excellent points, and look out for my next post!

 

[Returning you back to your regular program]

nate | ‎10-30-2012 04:17 AM

Hey John!

 

Where do I get that insight software stuff at? Certainly worth checking out.. I do have another HP P2000 for which I have VAAI integration enabled, but the documentation/best practices make no mention of any other integration points like insight control.  All of my servers have insight control licenses as well - and I've installed the HP management agents(ESX 4.1U3 - not ESXi), though I don't have any(that I know of) specific insight control integration software into vCenter itself right now.

 

One thing that was a nice update for the 3PAR vcenter stuff was they allow read only accounts now. I think in earlier versions they did not. Of course you can't provision storage with a read only account - but for just viewing stats, I saved read-only account credentials to my vCenter plugin thing. I've clicked on the 3PAR link one or two times since then, but still not all that useful yet.........

 

Client side software has never been 3PAR's strong point, fortunately the server side(array-side) more than makes up for it. For vmware - I would not want to use anything else - I suppose at the end of the day that's the main point! :smileyhappy:

 

Calvin - guest blog post - I don't need any more people trying to claim I am in some way compensated by 3PAR for the stuff I've written about them over the years :smileyhappy: [41 posts covering 3PAR over the past  3 years - though about a dozen of them was the back/forth between Dell/HP on the acqusition].

 

I'm hoping to get some new things to write about here at some point, 3PAR innovation has been well shall we say slow since InformOS 2.3.1 - other than Peer motion. V-class is great of course though at this point it's just a bigger & badder T-class with end to end T10 support. I can't keep writing about the same thing over and over !

 

Oh - I wrote a basic little script a few years ago to calculate the number of RAID arrays on a 3PAR box - this really drives the distributed point home I think. I think it'd be really cool if they included this metric somewhere in the management stuff. On that array I had at the time there was 81,000 RAID arrays.

 

My current 3PAR array is much smaller (different company), so only 17,000 RAID arrays here.

 

 

 

| ‎10-30-2012 05:33 AM

Hey Nate - writing a blog post about the new Insight Control for Storage Module now.  The HP Insight Control Storage Module for vCenter v.7.1 is available as a free download from the HP Software Depot.

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the community guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
This profile is for team blog articles posted. See the Byline of the article to see who specifically wrote the article.


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation