Around the Storage Block Blog

Are NetApp performance claims logical?

by on

By Jim Haberkorn

We at HP like to test competitor arrays and see if we can match their published results. In our internal testing, only the NetApp results are so out of whack that we are left scratching our heads over what the heck they did to achieve them. So for NetApp, we have published our results and challenged them on it. And in fact, the NetApp folks at first put up a vigorous defense which included a fair number of attacks on the integrity of our engineer who did the testing, but then as the discussion progressed, the NetApp enthusiasts went quiet. See the blog post titled Making sense of WAFL.

So, we've argued against NetApp performance with numbers and the argument got very technical, and we won. But recently, NetApp, on one of its blogs, waited till the discussion had died down and then, to our surprise, acted as if they had won the discussion and that the issue now was totally resolved and that they had once again defeated the forces of evil. So, now NetApp has left us no choice but to try a new tactic: Logic.

The NetApp counter-arguments on the subject of their performance are becoming more and more like the man who admits it is raining, admits he is standing outside, admits he doesn't have an umbrella, but then denies he is getting wet. If you ask a NetApp engineer these performance related questions you will get the answer 'yes' to all of them.

Do you build your LUNs on top of a file system?

Does your file system write only to free block space rather than updating the old blocks?

Do you ship a de-fragmentation tool on every NetApp filer?

Do you use software RAID?

Does your file system spread the metadata over the entire disk system?

Does WAFL require you to build secondary inode trees if the file is bigger than 64KB?

Can IOs to a particular file be serviced by only a single controller in a NetApp cluster?

Do NetApp snaps reside in the same disk group as the primary volume?

Is only a quarter of your NVRAM usable?

Is the processing power of your largest filer, rated by NetApp to handle 1176 disks, based on a maximum 8 x 2.6 GHz AMD dual core Opteron processors - about the same processing power as a Proliant DL785?

Is your file system optimized for writes?

But, if you then ask a NetApp blogger, if taken together, do these features negatively impact NetApp performance, you'll get an answer to the effect, "What are you crazy? How did you ever come up with that conclusion?"

Now, I am sure that every one of those eleven engineering decisions was made by NetApp for a logical reason from their perspective. But I am also sure that every one of them has a negative impact on NetApp performance, and the cumulative effect is pretty significant in most real-world environments. And those features put NetApp at a huge performance disadvantage in a storage world dominated by vendors who don't require the extra overhead of having to build LUNs on top of file systems, or don't need to ship a de-fragmentation tool with their array, or don't use software RAID.

Conclusion: Is every NetApp NAS customer unhappy with NetApp performance? Answer: No, of course not. But, of the storage arrays we've tested, is NetApp block performance the worst by a pretty wide margin? Yeah, looks that way to us.

Best regards,

Jim

Tweet this!

Labels:

Comments

by on

Hi Jim,

Nice post. By the way, on that same line of thinking. I recently did some testing on my iPod and discovered that my MP3s are all actually compressed, and get this -- on purpose! Can you imagine that insanity?

I'm happy to report I've tossed out my iPod, and replaced it with a good old-fashioned 8-track player. Furthermore I've writen an angry letter to Steve Jobs and the Motion Picture Experts Group - and I am proud to report I will be starting my own blog on the subject.

Permalink

by on

Jim

Go past your benchmarking folks and and encourage them to submit an SPC benchmark for an EVA or LeftHand SAN. Get them to explain the difference to you between specSFS and SPC while you're there, as your previous comment about specSFS here ( www.communities.hp.com/.../box-score-for-netapp-capacity-calculator.aspx ) was so off the wall I thought you were joking. Talk to a few NetApp customers and get a feeling for what they use their systems for. Try and understand where and how you fail to get NetApp technology, and how you end up drawing the strangest conclusions. Have a chat with John Spiers of HP LeftHand and ask him the same questions about how a LeftHand SAN works, and try not to get too despondent at the number of "yes" answers. Don't think in terms of umbrellas, rain, and the forces of evil while speaking with him.

Then get back to us here. Take your 11 points and do something solid and substantial with them other than wild specualtion. Use another tactic that works well for us here at NetApp; facts. Give us facts.

Permalink

by on

Hi Lee,

Thanks for your comments. I've got some old Bee Gees on 8-track if you're interested.

Best regards,

Jim

Permalink

by on

Hi Alex,

I notice you didn’t refute any of my eleven points – I’ll take that as a confirmation of their accuracy. In any case, thanks for the classic blog response – I must admit, Alex, you do ‘feigned outrage’ better than anyone else I blog with.

I was having a discussion with an ex-NetApp reseller the other day and he explained to me why his company dropped NetApp as a partner. While there were several reasons, one of his complaints was that he was getting really tired of the ‘our product is perfect and has no weaknesses’ cool-aid.

Now, in these blogs, I don’t expect any competitor to fall on their sword and to humbly admit they have weaknesses. However, what I’m used to seeing from competitors when weaknesses are pointed out are explanations not denials. As I said when I first kicked off this blog, there are hundreds of decisions that go into designing an array. And I would say that almost all those decisions involve tradeoffs of one kind or another. For example, if you start out designing an OS and storage system to be 100% optimized for files and NAS and then decide to add block functionality ten years later, there are going to be tradeoffs.

In my experience, when a competitor points out another competitor’s weakness, the response typically takes the form of either one of two arguments: The competitor will either show how the weakness can be mitigated, or else say that the weakness was a necessary trade-off to enable another more valuable feature. For example, if a vendor uses software RAID in their storage system then the natural counter argument is that, while it may be slower than hardware RAID, it saves the customer cost, and the added latency can be partially offset by having more powerful processors. There – pretty simple to explain. The positive tradeoff is lower cost and the mitigating factor is the powerful processors.

But what if your position is that your product has no weaknesses? Then you are stuck in a bind. You can’t use any of the two normal and reasonable arguments above because to do so is to admit that you had a weakness in the first place. So what’s left to argue with: I suppose counter-attacks, feigned outrage, and sarcasm.

Alex, you are not looking for facts -- I gave you eleven --you are looking for facts you can refute.

Best regards,

Jim

Permalink

by on

Hi Lee,

Loved your analogy and think we should play it out. iTunes is not a great interface nor user experience – it’s complicated and there are better applications for managing your MP3 collection. And in fact, iTunes isn’t optimized for MP3’s but their own proprietary format. So I’m sure there are people that love their iPOD and use iTunes but for those of us that saw early on that MP3 was the format that was best for our music needs, we avoid iTunes like the plague.

Kind of sounds a lot like NetApp to me – design a file serving architecture, re-jigger it to work with block data, then claim perfection has been achieved on all fronts.

But that little analogy was really just a diversion tactic, right? Jim and I are both anxious to know which of his 11 points NetApp can refute.

Permalink

by on

I dunno Calvin, on the other hand I think there are a good number of people who specifically use iTunes to manage all their music and videos, even stuff that's not MP3 or Apple format. I guess you might say they use it as a sort of virtual gateway to their music collection.

But hey, let's be honest, there is still a healthy market for all kinds of approaches to solving the problem:

www.gadgetell.com/.../cheap-trick-goes-retro-brings-back-the-8-track

Permalink

by on

Jim, Calvin:

You've discovered our dirty little secret. However our sales pitch over the past 17 years has been pretty honest. Usually with a dumb look on our face we say:

"Mr. Customer, You should know that there are exactly 11 horrible problems with our storage arrays, yet somehow it still manages to outperform HP's solution... Yeah I don't get it either!"

;-)

Permalink

by on

Jim,

I'm not sure but I have a feeling Lee didn't refute the 11-points, but your entire logic.....

Permalink

by on

@Jim

When you first started blogging your "thinking" on this subject out loud, Pat Cimprich of Avanade replied (in some detail) here; blog.avanadeadvisor.com/.../12107.aspx.

But 8 months later, you're back. This time it's "11 facts". And you're blogging on what *you think* these facts mean.

As in this; "But I am also sure that every one of them has a negative impact on NetApp performance". And this; "is NetApp block performance the worst by a pretty wide margin? Yeah, looks that way to us."

That's just two of the many steaming piles of "what Jim thinks" that make up your conspiracy theory.

Jim, *no-one cares what you think.*

At this point, I'm probably going to be accused of being rude, arrogant and offensive. So be it. But missing the point I'm not.

Please show how "having a file system optimized for writes" or any of the other 10 "facts" relates to NetApp storage systems that provides our customers with flexibility, reliability, efficiency and data protection. With SPC and specSFS performance benchmarks, storage efficiency guarantees, and demonstrable and audited 5 9s availability.

Facts. Not your thinking on the subject.