NetApp Apparently Still Lags in Cost of Ownership

[#1]

By Jim Haberkorn

Never let it be said that NetApp doesn't learn from its competitors.  Back in 2006 NetApp published a white paper titled, "Total Cost Comparison: IT Decision-Maker Perspectives on EMC®, HP®, and NetApp® Storage Solutions in Enterprise Database Environments" whose purpose was to analyze the cost of ownership (COO) between competitor arrays.  The 22 page paper had at least five major ‘fairness' flaws, the most glaring of which was not clearly publishing the full configurations - which was pretty significant because if you read the full paper and actually worked out the math, it was clear that at the very least, NetApp was comparing its parity RAID solution with snapshots vs. its competitors' solutions with full-copy clones.  And there wasn't even a mention as to what RAID levels were being compared. 

I've always taken that white paper as the clearest proof NetApp could provide that its cost of ownership lagged seriously behind its competitors.  When you think about it, it is pretty funny: while NetApp was taking that paper around to its customers and using it to ‘prove' that it had the best cost of ownership, at least one of its competitors was taking the same white paper around and using it to prove just the opposite.  I mean - it was actually a quite compelling argument to use against NetApp - to ask customers why, if NetApp's COO was even close to that of their competitors, would NetApp even think to use these tactics.  

But now NetApp has a revised version of that paper out on their website with the same title as before but with a more recent date in which they do fix some of their previous problems, but not all.  In their previous paper NetApp broke several of my personal fairness rules including:

• Fairness rule #1 - you have to make it clear what is being compared. No leaving out RAID levels.


• Fairness rule #2 - no making up your own confusing names as a substitute for common industry terms - you can't refer to your competitors' full copy clones in the configuration table as ‘snapshot equivalent functionality'.


• Fairness rule #3 - no fair comparing apples to elephants - you can't, I mean in a COO study you JUST CAN'T compare solutions where one uses snapshots and the other uses full-copy clones.

But the new NetApp paper now clearly spells out in the configuration table that they are comparing their parity RAID with snapshots against their competitors' parity RAID with full-copy clones.    Which I think is NetApp's way of saying that an unfair comparison is not unfair as long as you admit it upfront.  Now, as to the research, the study is based on interviews with "more than 25 customers"  (this is marketing code for 26 customers) which means their sample size averaged out to 6 or 7 customers per product (the study compared four products - FAS3070c, EMC CLARiiON CX3-80, EMC DMX-3 950, and an EVA8100).       

The NetApp fairness logic is that they just happened to pick these ~20 HP and EMC customers to interview and, well, what could they do, all these customers just happened to be using only full-copy clones and not snapshots.  And it's not NetApp's fault.  They're just calling it like they see it in an unbiased and neutral way - just like any other competitor would do.  Still, whatever they paid to have the study done, they could have saved themselves the money and just asked me.  I would have readily conceded the point:  snapshots take up less space than full copy clones. 

But NetApp wasn't through yet. Their final conclusion went on to claim not that they were 10% better than their competitors, or even 20% - but that they were over 100% more efficient in acquired capacity than HP and EMC, the two biggest, most successful SAN vendors in the world.  In the paper NetApp stated that on average HP EVA 8100 customers bought 30.7TB for a 4TB database while NetApp customers only had to purchase 15TB.  When I read that I asked myself ‘if this were true what would I expect to see in the IDC numbers for NetApp's GBs per SAN unit'.  And the answer I decided was that if the NetApp conclusion was accurate I would expect to see NetApp with a much smaller number than HP.  In fact, since HP has a much bigger SAN business than NetApp and has been in the SAN business much longer, I would expect NetApp to generally be operating in smaller SANs on average than HP.  In other words, if the NetApp capacity advantage were real I would expect them to have a MUCH smaller SAN GB/unit number than the EVA. 

Instead, I find the opposite.  According to the latest IDC report the average EVA 8000 SAN TB per unit is 20.7TB while for the NetApp 3070 it is 38.6TB - which makes a lot of sense to me based on what I know about how EVA and NetApp filers are typically configured in SAN/block environments.   

So to conclude:  If NetApp and HP sold into equal-sized SAN environments (which premise I personally have serious doubts about but would be willing to accept for the purposes of keeping the argument simple) and if the NetApp report were accurate then you would expect HP's IDC SAN GB/unit number to be double that of NetApp.  But it's not: in fact it's about half the size (53%). 

So if you still believe the NetApp report is accurate then you have to also believe that NetApp sells into SANs that are approximately 4x the size of HP's.  Personally, I find that about as likely as an elephant stampede in Zurich, but in the end it's up to each reader to decide for themselves.  But now that I think about it, there is one more conclusion possible:  that despite what the white paper says, it is NetApp and not the EVA that has the issues with usable capacity.      

Jim Haberkorn, HP StorageWorks EMEA (Europe)

[#2]

"Fairness rule #1 - you have to make it clear what is being compared. No leaving out RAID levels."

OK, what RAID levels would you like to compare RAID-DP with? RAID-10? Why is it important? We've only got the one RAID level.

"Fairness rule #2 - no making up your own confusing names as a substitute for common industry terms - you can't refer to your competitors' full copy clones in the configuration table as ‘snapshot equivalent functionality'. "

Fair point. See below.

"Fairness rule #3 - no fair comparing apples to elephants - you can't, I mean in a COO study you JUST CAN'T compare solutions where one uses snapshots and the other uses full-copy clones."

Another fair point. Do you advise the use of snapshots on EVA instead?

Before casting stones, may I also remind you of the HP commisioned Edison report "TCO White Paper; EMC, NetApp, and HP Midrange Storage Arrays". That's where the HP beats NetApp and EMC because it requires fewer mouse clicks per second, apparently.

I'd ask yourself the question you pose at the top of your piece.

Why, if HPs TCO was even close to that of their competitors, would HP even think to use these tactics?

[#3]

Hi Alex,

 Thanks for your comment.  I had already made up my mind that whoever was the first person to respond to my first blog that I would offer them a dinner with me in Zurich.  Now, you're going to have to get yourself to Zurich, but if you ever do, you've got yourself a free dinner.  The offer is sincere.  

Now, in regards to your first question 'why is it important what RAID level?'  Answer: NetApp may have only one RAID level but the arrays the paper was comparing to had multiple ones.  In a study that was measuring COO based on number of disk drives it was crucial to know if NetApp was comparing against their competitors' RAID-1 or RAID-5. And if it was RAID-5 to know the size of the parity stripe.  

Second question:  Do I advise the use of snapshots on the EVA? Answer: it depends on what the customer is trying to achieve.

To your third point about disagreeing with the HP Edison paper using number of clicks as a way of measuring ease-of-use.   Answer:  when we wrote the paper, we really struggled to come up with a way of measuring something so subjective as ease-of-use in a storage system.  If you can think of a better way of measuring ease-of-use, I'd appreciate hearing your opinion.  If it sounds fair and logical and will result in helping customers to better judge ease-of-use in storage systems, then I will take your idea and propose it to the appropriate HP marketing team.

[#4]

If I was comparing raid levels, I would compare RAID-DP with vRaid1based on this quote from a NetApp whitepaper:

RAID-DP approaches RAID 1 mirroring for fault tolerance, but at RAID 4 pricing.

This statement is a bit specious on the pricing side since it takes an additional parity disk over Raid-4 - or a they giving the second parity disk away for free? I have seen suggestions that 8 data + 2 raid is typical for Raid-dp. Couple that with a recommended 65% utilization and you get a maximum usable capacity multiplier of .8*.65 = .52 for RAID_DP which is negligibly different from vRAID1 and you have *almost* the same reliability and all the complexity.

We haven't even touched on the storage overhead for features like dedupe and snapshots and etc. but if you are looking at basic SAN storage requirements, I have found the NetApp numbers all wet from the start.

[#5]

@George;

RAID-DP we recommend at 14+2, which would take 28 disks in RAID-1 for the same usable. The rest is very funny, especially space *overheads* for dedupe?!?!? It saves space; we have customers showing over 90% space savings in VMware environments.

@Jim

Those kind of RAID decisions are just one of the additional headaches and costs the EVA admin has to account for. It doesn't matter what RAID level you compare RAID-DP with; it wins on space and reliability every time.

Same with snapshots. NetApp customers just take them, no "depends" required. There's almost zero impact on performance.

As to the Edison TCO report, my main objections were (1) HP used the wrong tools and (2) the T in TCO should mean Total, not Partial.

As for dinner, yes please! Not candlelit though; folk will talk.

[#6]

Hi George,

Thanks for your comment.  Here in EMEA the most common NetApp RAID-DP stripe we run into is 14+2. At some point I think the first blog I wrote in regards NetApp will evolve into a usable capacity discussion.  I'm anxious to get into that as it has been taken up by other blogs recently and seems to have generated a lot of passion without reaching a discernable conclusion.  I think there is a way of nailing it down but we will need some help from our collegues at NetApp for that.  When that discussion is joined, I would appreciate hearing back from you again.  

[#7]

Hi Alex, thanks for your concise comments.  Before I comment on your most recent points, I’d like to go back for a minute to the NetApp report that started this blog:  my criticism of the NetApp report is not that it was incomplete but that it was an error in judgment.  We’re all professionals here and none of us begrudges the other side a little creative spin now and again, but for NetApp to compare the space required for its snapshots against the space required for its competitors’ full copy clones in a COO study was marketing at the spin velocity of a black hole (note: last I read, black holes are theorized to spin at 16% the speed of light).   But the key question in my mind is why did NetApp choose this faulty comparison in the first place.  Without even breaking a sweat I can think of a half dozen reasonable ways of setting up a COO study, as I’m sure could my colleagues at NetApp.  So why choose a bogus comparison that was guaranteed to have people screaming ‘foul’ unless that was the only way you could make your point.  I’m in India right now visiting NetApp customers and if NetApp cost of ownership comes up you can be sure that I will be bringing up the NetApp white paper before the customer does.   I keep a copy of it on my laptop just for that reason, and it does a far more eloquent job of refuting NetApp’s COO claims than anything I could say.  Which brings me to your point about NetApp snap performance.  NetApp wrote a white paper on that, too, and one of these days I’m sure I’ll get around to covering that paper in my blog as well.  

Now on to your other comments:  there is no doubt that there is a place for RAID-DP in the world.  But that doesn't mean that RAID-DP is always the right choice for every customer.  Your comment on RAID-DP's strong points didn't list performance – for obvious reasons.  But HP tends to look at these issues holistically and doesn't generally like to talk about individual features in isolation.  We prefer to talk about the reliability, performance, and required capacity of a system as a whole.  For example, for a NetApp sales rep to talk about space savings in the RAID choice without talking about NetApp aggregate, volume, and LUN space reservation policies is to leave a good part of the argument unspoken.  And to talk about dedupe without referring to the significant performance hit incurred is also leaving out a critical factor.   Now, as to the EVA:  choosing a RAID level is a single mouse click – not quite a headache unless you already had one from trying to set up a LUN on a NetApp filer.  And many customers count the cost of RAID-1 and the extra performance over RAID-DP as a perfectly acceptable trade-off.  Some of our customers would even say that when it comes to protecting their data they’d rather not entertain a mathematical discussion about the safety of various RAID levels, but would prefer to simply have a complete copy of the data on entirely separate disks – for performance and reliability reasons.  If you think about it, two copies of the data on totally separate disks  protects data in a different way than a single copy of the data protected by a statistical calculation based on disk MTBF numbers only.

I appreciate your comment about the Edison report not covering the ‘total’ in TCO.  While your point is valid, in defense of the report, I would say that by your definition I have never seen a Total Cost of Ownership paper by any vendor.  And if there ever was one, no one would read it because it would be as thick as ‘War and Peace’.  The Edison report chose to tackle a very difficult and subjective subject (cost of storage management) by developing a set of objective measures around not just mouse clicks, but time, and number of steps as well.  Perfect?  No.  A valid effort on a difficult subject?  Yes.  

Final note:  I suspect even the McDonalds in Zurich serves by candlelight.  So when you come, be sure to bring your wife along, she’s invited too.

Best regards,

Jim Haberkorn