Around the Storage Block Blog
Find out about all things data storage from Around the Storage Block at HP Communities.

More on SPC-2

Last month we announced a world record SPC-2 number by the XP24000.  At the same time we extended yet another challenge to EMC to join the rest of the world in publishing benchmarks.  They continue to decline the offer, arguing “representativeness”.  I thought I’d clear up the “representativeness” question.

 

EMC’s argument that this XP is too costly starts from the assumption that SPC-2 only represents a video streaming workload.  To quote, “128 146GB drive pairs in your 1152 drive box? A pure video streaming workload?”  We actually see a widely diverse set of workloads used in the XP.  The power of having both SPC-1 and SPC-2 benchmark results is that they provide audited data that applies to almost any workload mix a customer might have.  But if one had to pick a most common workload it would probably be database transaction processing by day, then back up and data mining workloads joining the transaction processing by night.  SPC-2 models the back up and data mining aspects, with SPC-1 representing the transaction processing.  SPC-2 is about a lot more than video streaming.

 

When people need bullet proof availability and high performance for transaction processing they turn to high end arrays like the XP24000.  It’s probably the most common use for a high end array.  Our data indicates that on average the number of disks in an initial XP purchase is right around the 265 in our SPC-2 configuration.  Some of those won’t have the levels of controllers in the SPC-2 configuration.  But an increasing number use thin provisioning.  In those cases they will often get all the controllers they’ll need up front, delaying the disk purchases as you’d expect with thin provisioning.  So the configuration and workloads look pretty representative.

 

Then consider a real use of the benchmark.  A maximum number is key in assessing an array’s performance.  Below that you can adjust disks, controllers, and cache to get fairly linear performance changes.  But when you reach an array’s limit, all you can do is add another array.  So once you know an array’s maximum number you know its whole range of performance.  By maxing controllers we provide that top end number, giving the most useful result.  For sequential workloads like back-up and data mining maxing disk count isn’t necessary, whereas it generally is for random workloads like transaction processing.

 

Now let’s discuss how one might use XP’s SPC-2 results.  Let’s say you need a high end array for transaction processing.  The most common case we see requires backup and data mining operations at night in a limited time window.  Since the XP’s SPC-2 result is twice that of the next closest high end array, you can expect it to get the backup and data mining done with half the resources of the next fastest array.  But with SPC-2 you can go further.  You can look up the specific results for backup and data mining workloads which are around 10GB/s for the XP24000.  Knowing how much data you need to backup and mine you can estimate how much of the system’s resources you’ll need to get those things done in your time window and therefore what’s still left for transaction processing during that window.  You can scale that for the size array you need for transaction processing.  And you can compare to other arrays that have posted results.  All using audited data before you get sales reps involved.

 

SPC benchmarks are all about empowering the storage customer.  XP24000’s SPC-2 result is important to the most common uses for high end arrays, as well as for less common uses like video editing.  The configuration we used looks pretty typical, with choices made to make the result most useful to customers.  The cost is pretty typical for this kind of need.  At HP we expect to continue providing this kind of useful data for customers.  And our challenge to EMC to publish a benchmark result still stands, though they’ll probably continue inventing reasons not to.

Comments
Anonymous(anon) | ‎10-03-2008 05:31 AM

Hi Craig


We must be looking at different descriptions of the SPC-2.


Take a look a this doc here:


www.storageperformance.org/.../spc2_v1.2.1.pdf


It's pretty clear to me this is all about sustained sequential transfers -- period.  That's file serving, video streaming, certain classes of data mining, etc.  Did I miss something?


Part of this document states that submissions have to correspond to intended use.  I don't really think you're selling XP24000s as file servers and video streamers, nor data warehouse support (didn't you guys just announce something different for that?), so -- technically -- you're not following SPC's rules, are you?  Minor point, though ...


BTW, congrats on the LeftHand acquisition.  A bit pricey, but they seem to be a great little company.


Finally, we publish literally hundreds of performance characterizations of our products on real world apps like Exchange, Oracle, SQLserver, SAP, DW, DB2... etc.  


I can send you a few dozen, if you'd like to take a look.


They're not benchmarketing exercises, though.  They're real world examples of expected performance.  


I'd suggest the same for HP, or any other vendor who's trying to create customer value.


-- Chuck

Anonymous(anon) | ‎10-04-2008 05:13 AM

Chuck,

You missed backups which are well represented by SPC-2 whether or not they’re spelled out in that document.

People run plenty of those workloads on XP24000’s.  Otherwise SPC and their industry wide reviewers would have stopped us at the door.   Accusing HP of not following rules when you clearly don’t know what people run on XP’s is frankly out of line.

Yes we’ve got tons of those other performance tests too, none of which are audited and reviewed the way SPC is.  HP and all the other major vendors provide this extra level of independently audited data.  EMC does not.  We’re not the first to point this out.   Either join the rest of the industry or let’s stop beating the dead horse.

Anonymous(anon) | ‎10-04-2008 10:56 AM

Craig -- I'm an industry-wide reviewer, and I'm calling BS on this last submission. But, because I haven't paid my mandatory entrance fee to the SPC, I can't officially challenge it.


As far as EMC "participating", I hate to disappoint you, but I'm seriously doubtful that you're going to see EMC stoop to this sort of benchmarketing shenanigans anytime soon.


Independently audited and reviewed data?  C'mon!  Who do you think you're fooling?


It's just too easy -- put together a benchmark special, run a few quick tests, and email it in.


All you have to do is write a "I've been a good doobee" letter, send along your payment, and every vendor's best friend Walter puts it up on his personal site, no questions asked.  


It's got to be one of the cheapest marketing gimmicks in the industry, which is why all you folks take your turn at paying for bragging rights.


Who's next?  Another IBM claim in a category of one?  Or perhaps NetApp running a test on 5% of usable capacity?  Another test with 36GB drives that aren't made anymore?  


How about HP benchmarketing a configuration that'll never get sold, running a workload that'll never get run?


It's almost as bad as a vendor turning off write cache protection.  


No, wait, that's already been done ...


I can't wait for the next submission and press release combination -- they're all like shooting fish in a barrel.  


I'm continually amazed how the weaker storage vendors try to hide behind the supposed pomp and circumstance of the SPC.  Guess what?  Knowledgeable customers aren't fooled.  


Maybe if these vendors gain a bit of confidence in their products, they'll move on to more productive activities


My advice?  Keep the focus on things that actually help customers, not mislead them!

Anonymous(anon) | ‎10-07-2008 08:40 PM

Chuck Hollis,


Glad to know that you are an "industry wide reviewer". How does one get such an appointment? I believe it is self-appointed, right? Well, I am a biased member of HP's marketing organization. I think that appellation is a little more direct and not misleading at all. I do not pretend to be anything more.


Your post is transparently self-serving, even though you try your rhetorical best to make it otherwise. Oh but wait, are we supposed to think you are an unbiased "industry reviewer", inspite of who you work for? Do you really think people are that naive?


Thanks for championing the cause of the gullible user. As you state, knowledgeable customers aren't fooled. I guess that means you are concerned that HP and the other vendors' support of SPC is an attempt to deceive the less than knowledgeable user. I would suggest HP's participation in the SPC 1 and 2, with its public review process (not just an audit) is exactly to the contrary. The SPC is an open reviewable benchmark. The methods used and the attempts to speed up the results are clearly published. The SPC, is quite unlike some benchmarks published by individual vendors, where the benchmarking methods are not clearly stated and the results are subject to many kinds of artificial enhancements. You noted some of the techniques for benchmark enhancement. With SPC, the publik knows when those tricks are used. With a vendor benchmark where only the Powerpoint results are published, there is not nearly so much possibility for review.


I suppose you can "call BS" all you want. As a master of the blogging process and a self-described "industry reviewer", you seem to be an expert in the topic. As you well noted, knowledgeable customers aren't fooled.


HP actually has a lot of confidence in our XP products, as evidenced by our publishing these publicly reviewable and audited benchmarks. EMC apparently does not have that confidence. The fact remains that EMC has not published. Instead, you as an undeclared representative of EMC rhetorically attack the benchmark and the companies who submit them. Sounds like so much whining to me, the biased HP marketing guy.


Our challenge to EMC remains on the table. Put out a pair of SPC 1 and 2 results for DMX. Show why our benchmark results mislead the customers. If yours are faster, then we are wrong. Try to prove that XP24000 is not the fastest disk array on the planet. If not, its not HP who is misleading customers.

Anonymous(anon) | ‎10-08-2008 04:58 AM

Chuck, your comment about “a configuration that'll never get sold, running a workload that'll never get run” is wrong, but we’ve been over that in this post and its predecessors.


All SPC benchmarks and results are reviewed by our storage industry peers (except for EMC who hasn’t joined), audited on site by an independent storage professional, and disclosed in great detail so there’s no place to hide.  It’s a lot more than “emailing it in.”  And a whole lot more open than what EMC provides.


No, I don’t expect EMC will join the SPC any time soon.  Therefore HP will continue to provide standard, independently audited performance results and EMC won’t.

Leave a Comment

We encourage you to share your comments on this post. Comments are moderated and will be reviewed
and posted as promptly as possible during regular business hours

To ensure your comment is published, be sure to follow the community guidelines.

Be sure to enter a unique name. You can't reuse a name that's already in use.
Be sure to enter a unique email address. You can't reuse an email address that's already in use.
Type the characters you see in the picture above.Type the words you hear.
Search
Showing results for 
Search instead for 
Do you mean 
About the Author
Featured


Follow Us
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation.