Around the Storage Block Blog

[#1]

My wife and I are home-schooling my 8th grade daughter.  As part of her science class this week, she had to do an experiment to see if objects fall at the same rate when dropped from the same height.  The experiment started with dropping a piece of paper, cardstock, and a book (all of the same size), determining which hit the floor first.  With the book landing on the floor first, my daughter concluded that things fall at different rates based on their weight.  But then she crumpled up the piece of paper and dropped it. She was surprised that the book and crumpled paper landed at the same time. If forced her to challenge her initial conclusion that objects fall at different speeds based on their weight.  She realized that objects fall at the same rate if you remove the affects of air resistance.

So what does this little science experiment have to do with storage?  Over the last several weeks there has been a couple of discussions, one on capacity efficiency and the other on performance.

The capacity efficiency discussion was initiated by EMC. They calculated theoretical capacity efficiency of several disk arrays, including the StorageWorks EVA, which initially resulted in very favorable results for the midrange EMC arrays.  It was far from science and I'd say heavy on artful marketing.  I won't try to catch you up on all of those discussions but you could read a couple of posts on this site to get "caught up".  Those posts are:

1. EMC Distortion about capacity efficiency
   
2. Storage Capacity Efficiency - Not Quite So Final

HP has suggested a couple of times that we'd be more than happy to participate in a third party evaluation of capacity efficiency - a real storage capacity science experiment.  It seems to me what customers need is real data and not one vendors' biased view that does nothing but add to our customers' confusion.  To date, EMC has not responded to our challenge - they didn't even acknowledge our offer.  I do appreciate that Chuck Hollis did admit on his September 12 blog entry that his analysis was wrong.  Here's what he said:

I think we got things wrong on the HP EVA.  We configured 7 disk groups, HP recommended 2.  The rules of the game were clear: go with what the vendor recommended.  So we were wrong in that regard.  Even with the changes, I think EMC still has a usable capacity advantage, but it's not as pronounced.

I appreciate Chuck's honesty here but as of now, there's nothing conclusive that either array has any advantage.  I don't appreciate that he still claims that EMC has an advantage - he has no proof and no substantiation.  I believe the best path forward would be for EMC to agree to a third party assessment - a real storage science project - instead of claiming some victory that doesn't exist.

Another storage-focused science experiment was discussed last week too.  In this case, it was around storage performance and it did have the makings of a real science experiment.  HP announced the results of our SPC-2 benchmark.  This testing is defined and standardized by the Storage Performance Council (SPC).  The SPC is a vendor-neutral standards body focused on the storage industry.  You can read more about them on the SPC website.  Every major storage vendor and even many of the smaller ones participate in the SPC.  Guess who refuses to?  Yup, EMC.  Here's what Chuck wrote in his September 12th blog:

Not a month or two seems to go by without some vendor running an SPC test, bragging about the results, and challenging EMC to a showdown.  And, just as predictably, we go look at what they've done, shake our heads, and further resolve never to get involved in any of this nonsense.

This all has me shaking my head.  EMC refuses to participate in meaningful, third party validated testing around capacity efficiency and performance but is more than happy to artfully market their "leadership" in these areas based on who knows what.  Am I biased - absolutely I am.  But I also totally get the fact that marketing fluff and spin doesn't benefit our customers when trying to make decisions.  I hope in the future these topics are covered by EMC with more of a scientific rigor instead of marketing talking points and spin.

As customers, you have to challenge what vendors are saying - make sure that their claims are backed by substantiation.  Be careful when you hear about experiments (remember my daughter's science experiment) that ignore facts and give wrong conclusions. HP's "Standards of Business Conduct" require that what we say can be substantiated.  Does that mean that we are always perfect - no, of course we aren't.  But I think it means that customers can trust what HP says... so challenge us. 

[#2]

Not on topic, but you should rig an experiment to get her to discover that it is the density of an object that determines its acceleration due to gravity. This is easiest to show in water. Or just watch disc 3 of the Young Indiana Jones chronicles, as my (8 year old) son just did, where little Indy tries the experiment at Pisa!

Yay, Geek Dads!

Oh yeah, and I'd love it if EMC would submit their arrays for performance testing. I suspect they'd not do all that bad, so why not just let the test proceed instead of relying on spurious tests by other manufacturers?

Stephen

[#3]

Hi Stephen - now you got me curoius on the density issue.  I thought acceleration due to gravity was a constant in the absence of air resistance.  Are you saying that density accounts for the real world air resistance?

As to the topic of EMC performance, I can only guess that EMC won't participate in SPC benchmarking because they have something to hide.  I don't know what that is but I'd bet a lot of money that if they thought they could win, they'd be running the test; I suppose there could be other reasons but I don't know.

Thanks for stopping by!

[#4]

Hi Calvin

I think you misread my statement.  

I said "I think that EMC CX still has a capacity efficiency advantage".  

If you've ever sat through a logic or philosophy class, you'll realize that the standard of proof are decidedly different for "proving that I believe it" and "proving that it is factually true in all circumstances".

I think you confused the two.

I wished you had copied over the core of my criticism here, namely, HP tested a benchmark special that no rational customer would ever purchase for the workload tested.

If you'd like to argue persuasively that there is a significant market for enormous $1.6m enterprise arrays lightly populated with a small number of mirrored drives to support sequential workloads, I'm all ears ...

So, how do you expect EMC -- as the market leader -- to participate in these benchmarketing shenanigans, and keep a straight face?

I mean, seriously, if I put something like this in front of my customers, I'd just bust out laughing ... :-)

-- Chuck

[#5]

OK, I'm bored, so I decided to poke around the SPC-2 site and figure out what might be the best price performance capacity deal.

Behind door number 1 we have the XP24000 or USP-V, serving up 18TB of usable capacity at 8,724 SPC-2 MBPS (a unit of performance goodness, I guess) at a breathtaking $1.6m.

In terms of usable capacity, I figure $89 per GB.  And in terms of price per "SPC-2 MBPS", we've got approximately $183 per unit of performance goodness.

Behind door number 2 we have 20 of Sun's new J4200 arrays, using 1TB drives.  I get 240TB usable for a princely sum of $252,000, giving me a price per usable GB of just north of $1 per GB, a bit better than the $89 offered by HP and HDS.

Here's where it gets great: I get an aggregate 10,976 "SPC-2 MBPS" units of arbitrary performance goodness, beating the XPS24000 and the UPS-V handily.

And the great part of this comparison is that I have about $1.35 million dollars still left in my storage budget for, say, an offwite with my IT team in the Bahamas, charter a private jet, and spouses can come along!

Now, before you cry foul and say 'how can you compare a large-scale XPS24000 with a bunch of low-end JS4200 arrays?", well -- you've made my point.

And that's exactly the kind of useless comparisons that the SPC encourages.

Have fun with this, ok?

-- Chuck

[#6]

Hey Chuck -

I understand the difference between a belief and proving something.  I'd say the way the conversation went on your blog was lots of "analytical data" (complete with pie charts even!) that looked like a science experiment - backing up your belief.  Later you stated that your data was wrong and the initial conclusions you had were invalid but you stated your belief that the CX had the advantage anyway.  So you started with science and switched to what you believed with nothing to back it up.  Do they teach bait and switch in those philosophy classes you took?

As to your question about "benchmarketing" shenanigans - I'd suggest that it was you that started the shenanigans.  Come clean Chuck and let's get to the real data.  What I'd propose is that we go out to our installed base of customers, via a third party, and do a REAL analysis of useable capacity.  There would obviously need to be parameters that this is evaluated against to normalize the data (RAID level, number of drives in the array, etc) but I'm confident that we could turn your benchmarketing shenanigans into a real third-party driven study.  

With this data, we could show customers your initial capacity efficiency benchmarketing; I agree -  they would indeed be laughing.  So are you game to do a capacity efficiency study of our installed bases or do you want to keep playing the benchmarketing game?

[#7]

Your first point -- well, that's just mudslinging, Calvin, I'd hoped for better from you.

As far as your second point, sure, what exactly do you propose?  Who would pay for it?  What would be looking for?  

And, ultimately, how would it benefit customers, rather than turn into a marketing contest?

I mean, the idea sounds good (e.g. "let's solve world hunger, shall we?"), but I think you might want to think through a few more details, otherwise it comes across as a one schoolkid calling out another one.

Just a surface analysis raises some fundamental issues, e.g. EMC is in many markets (and use cases) that HP is not.  Similarly, portions of the HP line might play in places (and use cases) where EMC is not as strong.  

Those differences matter -- how a financial trading firm views their online apps is very different than how an SMB views their accounting system.

Capacity and availability are related (RAID 1, RAID 5, remote replication, etc.) -- what saddle point do you recommend?

Similarly, usable capacity and performance are also related (reference recent HDS - er HP -- SPC-2 submission).  Got a saddle point in mind for that one?

And there's more, but I digress ...

Before anyone puts any serious thought into this, we'll need the answer to Question Number One -- how does this benefit customers and the industry?

And, frankly, I don't see a clear answer to this one ...

That being said, any and all serious proposals will be seriously considered.

Finally, I think you're smart to be backing away from the SPC nonsense in your last comment.

It's all an enormous waste of time, IMHO, promoted by those who have nothing better to do, or nothing to lose.

Maybe server vendors ARE starting to figure out storage a bit :-)

Cheers!

[#8]

Hi Chuck,

My comment was not intended to be mudslinging - I think I accurately pointed out that you started the capacity analysis with your data, you realized you had made a mistake (and admitted the CX wasn't 2X more efficient as you originally claimed) but then concluded the CX wins without anything to substantiate that claim.  That's the way I saw the discussion - I'm sorry you took it as mudslinging.

As to your comment about backing away from the SPC "nonsense", HP isn't backing away.  I'm sure you've recognized that this is a group blog.  I'm not an SPC expert and have asked Craig Simpson who wrote the original SPC entry on our blog to respond to your coments.  He'll do that as soon as he can but has a "day time job" and hasn't had time to do that yet.  So stay tuned.

As to the third party review of our installed base of EVA and CX arrays to determin capacity efficiency - I have several things here:

1. You asked why would such a test would benefit customers or the industry?  I guess maybe you should have had an answer to that before you posted your blog about capacity efficiency.  Was it important a few weeks ago but now it isn't?


2. In terms of who would pay, we had suggested when we originally offered this test that the "loser" pays.  If that doesn't work for you, we can both pay half the cost to the third party that we mutually agree on.


3. How do we normalize the data (your example of having our products in different markets)?  I'm sure there's probably a number of applications that are common to our installed base - things like Exchange, SQL, Oracle, etc..  That's a non-issue.


4. You also brought up things like availability and capacity.  I'd add performance and management to that list.  My question back to you is did you think of those issues before you did your orginal blog post? 

Seems to me you put out your original analysis without considering all these other vectors but now that there is not a clear CX victory, you don't think answering the question you raised is worth it.  

If you're interested in pursuing a joint study, let me know.  I'd be happy to talk offline about it with you.

[#9]

Hi Calvin

I think you're confusing a few things.

I personally posted an entry that had nothing to do with "research" per se: we simply did a paper experiment to show the differences in usable capacity in an Exchange environment based on (what we thought at the time) was each vendor's published best practices.

The purpose was to highlight a single issue -- differences in usable capacity -- among many.  

The purpose of the blog post was to start a discussion on the topic -- which we succeeded in doing.  

I think that customers who followed the online debate benefitted from the broader discussion.  Many, many interesting perspectives were shared, including yours.

It was not a formal study.  It was not a customer campaign.  It was not a white paper.  It was not independent research. It was not a professional service.

It was a blog post, folks, based on a paper analysis of published documents -- limitations and all.

One of the 400 or so blog posts that I've written over the past two years on all manner of topics.  

Nothing more, nothing less.

Now, on to your additional points.  

"Loser pays" sounds like a bar room bet to me.  If you believe this is an important topic to the industry, we should think in terms of a construct that achieves the goal.

Ummm -- as far as your proposals for normalization, I'd offer that you might be oversimplifying a tad ...

Let's take something as simple as Exchange.  

There are huge differences betwen an Exchange implementation for an SMB and one for Citicorp.  Differences in regulated vs. unregulated industries.  Centralized vs. distributed architectures.  DR or not?  Archived or not?  Exchange 2003, 2007 or 2009?  Cached clients or not?

Now, let's play that same game with Oracle: OLTP or DW?  Business apps or test-and-dev?   10g RAC or a more traditional environment?  Service provider or data center?

My friend, you're oversimplifying to the extent that any formalized approach would be more about bragging rights rather than creating customer value.

So, feel free for criticizing my blog post.  Everyone is entitled to their opinion, including you and me.

Looking forward to more thoughtful posts from the HP Storage Team!

-- Chuck

[#10]

Hey Chuck,

I can't begin to claim I know what you were thinking when you originally posted your paper research, but you summarized it by claiming victory not a marginally better but that EMC was 2X better.  I'm sure you're aware that several storage reporters picked up your original post and they called it a fight.  Maybe that wasn't your intent but I can only judge your intentions by what you wrote.  The discussion certainly has been enlighting and customers have learned what from you?   Well, I learned that when you make claims that an EMC product has some great advantage over other products, check the facts.

I'm sure with over 50,000 EVAs shipped, we could find a core of EVA and CX systems that are used in very similiar environments.  I'm well aware that there are lots of factors that we'd have to address (not oversimplifying my friend)  but you've made it clear its not a study you're interested in pursuing so let's drop it. Maybe it's something HP will do on our own with a third party.

Have a good weekend,

Calvin

[#11]

Calvin --

One of the more interesting things we brought to light during all of this was the strange nature of disk groups in EVA.

As you remember, we probably overconfigured in our blog post, something we now know.  But the explanation wasn't entirely satisfying to me, nor was it to several readers.

HP says "fewer disk groups are better", yet never, ever seems to recommend just 1.  

2 seems to be a minimum, 3 or 4 or more deemed excessive or wasteful.  So, what could it be about disk groups that point to most always wanting 2 of a thing, but almost never 1 or 3 or 4?

So, I think I've figured out what's going on here.  

One view of the EVA disk group is that's its' a relatively isolated construct in terms of spindle performance isolation and protection against multiple disks failing.

Take a bunch of spindles, and make it a "virtual array": stripe'em all, dynamically rebalance them all, autoraid them, spares, etc. -- if I remember back to the basic idea.  

Sure, customers can do this multiple times within an array, but at the expense of progressively poorer and poorer disk capacity utilization.  

And let's not forget that there's a shared storage controller that sets a hard limit on how many disks, how much cache, how fast in aggregate, and so on.  The virtual entities live in a shared physical box which has certain boundaries and limitations.

And then it struck me.  

If you told people to only use 1 disk group, the "multiple virtual array" feature of the product would not come in to play, and then EVA would be just another me-too spindle-randomizing dual-controller array like so many others out there.

No, I think the HP storage and sales marketing folks want a Very Unique Feature to talk about and help steer the customer discussion, and being able to have multiple "virtual arrays" within a single one is Just Too Cool to pass up.  

Especially using the V word and all.

So, HP works very hard to make sure that the feature is exposed -- but not too much!  The best number is always 2!

Every vendor makes design choices and tradeoffs with the products they build (that's one of the reasons EMC builds so many dfferent kinds of storage platforms), and I think I just began to understand HP EVA's tradeoffs a bit better.

Now, as we look to the transition towards enterprise flash drives, you've got a problem coming up.  

Since the USP-V aka 24000 uses a more traditional high-end architecture, it won't be too hard to support these drives, and have customers put them to work right away, as we're seeing on the current DMX and CX.  

As an example, customers wanting to take Oracle performance to the next level seem to be eager adopters, at least at this point in the market.

I'm guessing we'll see them from HDS and HP sometime towards the end of 2009, or perhaps early 2010.

But we won't see them anytime soon in the EVA, will we?  Because there's no way to isolate their characteristics, other than to put them in their own disk group.  

Ouch, that's gonna hurt.  

Four Oracle applications on an EVA, each wanting acceleration, that could mean a *very* large number of disk groups.

An, as we keep going down the HP product line, the MSA looks like it *could* support something like an EFD with a moderate amount of work, but I don't think anyone will want it in that sort of form factor.

No wonder HP has to FUD the whole flash issue!  Now it all makes sense!

And you have a good weekend as well ..

[#12]

Hey Chuck,

We really need to enroll you in one of our fundamental EVA education classes.  There you too could learn the innovative design capabilities of the HP StorageWorks EVA.  Unfortunately, without this understanding, you continue to get the facts confused.  Almost every point you raised in you comment (from Saturday) was wrong. Here is a quick course to correct those errors.

• 
  
  Many EVA customers are using just one disk group. We in fact do recommend one disk group unless customers have both FATA and FC drives, in which case the recommendation is two disk groups. There certainly are cases where customers may want more than that but that would be based on specifics of their environment. I've talked to many of our storage architects who are configuring EVA's and one or two disk groups is very, very common. One in particular told me this: "I set up EVAs every week and only have more than one disk group if FATA drives are involved. Many of my clients have huge SQL or Oracle databases and most have multiples of them."


• You raised something about the "multiple virtual array feature". There's no such thing. The beauty of virtualized storage with the EVA is that there are fewer things to manage. I have no idea where you got the idea of "multiple virtual array feature" but it's not from the EVA. The only thing that comes close to this concept is the fact that within a disk group we can have multiple levels of Vraid protection. The construct of a disk group and the Vraid protection are different and I think you are may be confusing the two.


• For you and our customers reading through this discussion, I'd recommend two papers to help you better understand the value of virtualized storage with the EVA:




• First, is a white paper that describes EVA virtualization - what it is, what the benefits of it are, and how it works. EVA virtualization makes it significantly different that a traditional disk array like the EMC CX.


• The other is a study that was done to compare EVA and CX (as well as NetApp) common management tasks. For customers dealing with complexity, this third party study conclusively demonstrates the ease of use benefits that the EVA has compared to traditional disk arrays like the CX.


• I'm not sure what enterprise flash FUD you are referring to. HP's position has been to move from the hype EMC started and bring reality back into the discussion. And while I won't share any roadmap details with you, you are way off in your dates as to when HP StorageWorks will have solid state technology available in our arrays.


• You also don't seem to understand the tiers available with an EVA today or haven't paid attention to history. As I recall, HP was the first major storage vendor to offer ATA class drives in a disk array. This was in mid-2004 with the launch of the 250GB FATA drive. Again, I won't share any roadmap details with you but one could assume since we offered FATA as a lower cost tier that we can offer solid state technology as another tier option for the EVA.

I hope you had a good weekend - mine was great and I especially enjoyed Boise State beating Oregon and moving into the Top 20. 

If you're interested in an HP Education class on the EVA, let me know and I'll find out when the next one is.

Calvin

[#13]

Hmmm -- I guess Craig Simpson is too busy to help explain what HP could possibly have in mind regarding their latest SPC-2 submission.

He left a comment on my blog stating essentially "well, you know, arrays occasionally have sequential workloads".  I would think so.  Captain Obvious strikes again. But that's not what was tested, was it?  

Look, I think HP is a pretty straight-shooting company as such things go.  Why continue to erode your credibility with this kind of marketing nonsense?  Find anyone who's serious about storage, and ask them their opinion of the SPC test, and you'll usually get a blast of disparaging comments.

Here's to hoping y'all move on ...

-- Chuck

[#14]

Read through your last comment, and would like to refine the discussion a bit.

1 -- Regarding your statements that multiple disk groups are only used when mixing drive types (e.g. FC and ATA), go re-read your own documentation that speaks to the desirability of separating workload types (e.g. random and sequential) using disk groups.

2 -- I'm sure you can configure things very often with one disk group.  Since size and scale matter, words like "big" or "often" don't have much meaning unless we talk specifics.  I am under the impression that EMC probably gets involved in more large application environments than most storage vendors, hence our natural bias towards more challenging performance requirements.

3 -- Regarding "flash FUD" -- what else would you call that video link you're promoting?  The basic message to customers is "wait, we're figuring this out, we'll be back to you soon".

4 -- As I remember things, HP went for a "special build" for an FC-based ATA drive (hence FATA) to retain compatibility with their existing FC arrays.  We thought that was goofy, and went for a FC-to-PATA bridge so we could use real ATA drives rather than some "special build".  I think we were in the marketplace at roughly the same time.  Does Seagate still make FATA drives for HP?  Anyone else in the industry using them?

5 -- If you go re-read my posts, I didn't say that you can't support EFDs.  What I did say is that you'd have a really hard time supporting them cost-effectively.  

BTW, you really don't want me taking one of those EVA classes.  Just think how dangerous I'd be afterwards ....

[#15]

Hey Chuck -

1.  I didn't say "only use multiple disk groups" when mixing FC and ATA (by the way, we use FATA, not ATA) - I said there certainly are cases where customers may want more than two disk groups but that would be based on specifics of their environment.

2.  I'm sure none of the 50,000 + EVA's or the thousands of XP's that HP has shipped are in complex, challenging storage environments.  I conceed that only EMC has done this to date ---NOT.  Get real!

3.  Are you shipping more flash than FC or SATA drives yet?  The day that HP announces that we are shipping Solid State Technology, it will probably take us a few months to out ship your quantities.  We will have them in our servers and storage.  The FUD is you trying to claim that HP won't have Flash technology until late 2009 or 2010.  

4.  Yes, we do use FATA drives and our customers didn't need any additional hardware to support it.  FATA drives work in the same shelf as the FC drives.  I don't know who is making the drives and don't know if anyone else is using them.  Given that HP ships over 45% of all disk drives on the planet, we have a bit of influence with our suppliers.

5.  Ok, you didn't say we can't support them.  We cost-effectively support FATA and FC today.  No reason we won't do the same with Solid State Technology.  

If you took the EVA class, you'd at least understand the basics of EVA virtualization and the benefits customers get from it.  Then again, maybe you would be dangerous afterwards - you'd probably try to figure out how EMC could offer the same ease of use that our EVA customers enjoy today.

Time to move on to a new topic... thanks for helping clarify things.  BTW, I'm sure you'll also hear more from Craig ... yes, he has been busy I'm sure.  We all have day time jobs and none of us are Chief Blogging Officers.

[#16]

Chuck - Time to talk SPC-2 again.  You seem to be hung up on the idea that running sequential workloads or having 260 disks in an XP is somehow unrepresentative.  It’s as if you believe there’s some cookie cutter workload and config that defines how everybody uses these arrays.  Well I can tell you for sure that isn’t true of HP products, which is why we provide choices of products and information.

On XP’s (I can’t speak for EMC products) you’ll find workloads that include transaction processing (SPC-1), backups, data mining, and large file number crunching (SPC-2).  By providing both sets of performance numbers we giving our customers a pretty complete set of standard, audited data with which to judge how XP will perform on the different aspects of their workload.  Everybody will have a different mix that they can evaluate with our data.  We just want to make sure they’ve got data to judge with that’s based on something more objective than murky proprietary testing.  So that’s what we’ve done.  And you’ve not.

About comparing XP to the price/performance of low end arrays, there are a bunch of use cases out there where very competent people have concluded that the extra cost of an XP is well worth it.  Bullet proof availability, single array performance, and the ability to consolidate many diverse workloads in one place are sometimes mandatory.  Otherwise there’d be no market for high end arrays, ours or yours.  Where those needs are paramount your comparison to low end arrays is meaningless.

As for video streaming, it isn’t the primary point of XP’s SPC-2 results.  I expect we’ll meet the majority of our customers’ video storage demands with our ExDS product line in the future.  Most such workloads need that kind of price/performance point.  However, there are some very successful businesses in this world that need bullet proof, blazing fast access to some of their video content.  The ones we know of use XP’s.  And NOBODY in HP is laughing at them!