By Jim Haberkorn
Never let it be said that NetApp doesn't learn from its competitors. Back in 2006 NetApp published a white paper titled, "Total Cost Comparison: IT Decision-Maker Perspectives on EMC®, HP®, and NetApp® Storage Solutions in Enterprise Database Environments" whose purpose was to analyze the cost of ownership (COO) between competitor arrays. The 22 page paper had at least five major ‘fairness' flaws, the most glaring of which was not clearly publishing the full configurations - which was pretty significant because if you read the full paper and actually worked out the math, it was clear that at the very least, NetApp was comparing its parity RAID solution with snapshots vs. its competitors' solutions with full-copy clones. And there wasn't even a mention as to what RAID levels were being compared.
I've always taken that white paper as the clearest proof NetApp could provide that its cost of ownership lagged seriously behind its competitors. When you think about it, it is pretty funny: while NetApp was taking that paper around to its customers and using it to ‘prove' that it had the best cost of ownership, at least one of its competitors was taking the same white paper around and using it to prove just the opposite. I mean - it was actually a quite compelling argument to use against NetApp - to ask customers why, if NetApp's COO was even close to that of their competitors, would NetApp even think to use these tactics.
But now NetApp has a revised version of that paper out on their website with the same title as before but with a more recent date in which they do fix some of their previous problems, but not all. In their previous paper NetApp broke several of my personal fairness rules including:
- Fairness rule #1 - you have to make it clear what is being compared. No leaving out RAID levels.
- Fairness rule #2 - no making up your own confusing names as a substitute for common industry terms - you can't refer to your competitors' full copy clones in the configuration table as ‘snapshot equivalent functionality'.
- Fairness rule #3 - no fair comparing apples to elephants - you can't, I mean in a COO study you JUST CAN'T compare solutions where one uses snapshots and the other uses full-copy clones.
But the new NetApp paper now clearly spells out in the configuration table that they are comparing their parity RAID with snapshots against their competitors' parity RAID with full-copy clones. Which I think is NetApp's way of saying that an unfair comparison is not unfair as long as you admit it upfront. Now, as to the research, the study is based on interviews with "more than 25 customers" (this is marketing code for 26 customers) which means their sample size averaged out to 6 or 7 customers per product (the study compared four products - FAS3070c, EMC CLARiiON CX3-80, EMC DMX-3 950, and an EVA8100).
The NetApp fairness logic is that they just happened to pick these ~20 HP and EMC customers to interview and, well, what could they do, all these customers just happened to be using only full-copy clones and not snapshots. And it's not NetApp's fault. They're just calling it like they see it in an unbiased and neutral way - just like any other competitor would do. Still, whatever they paid to have the study done, they could have saved themselves the money and just asked me. I would have readily conceded the point: snapshots take up less space than full copy clones.
But NetApp wasn't through yet. Their final conclusion went on to claim not that they were 10% better than their competitors, or even 20% - but that they were over 100% more efficient in acquired capacity than HP and EMC, the two biggest, most successful SAN vendors in the world. In the paper NetApp stated that on average HP EVA 8100 customers bought 30.7TB for a 4TB database while NetApp customers only had to purchase 15TB. When I read that I asked myself ‘if this were true what would I expect to see in the IDC numbers for NetApp's GBs per SAN unit'. And the answer I decided was that if the NetApp conclusion was accurate I would expect to see NetApp with a much smaller number than HP. In fact, since HP has a much bigger SAN business than NetApp and has been in the SAN business much longer, I would expect NetApp to generally be operating in smaller SANs on average than HP. In other words, if the NetApp capacity advantage were real I would expect them to have a MUCH smaller SAN GB/unit number than the EVA.
Instead, I find the opposite. According to the latest IDC report the average EVA 8000 SAN TB per unit is 20.7TB while for the NetApp 3070 it is 38.6TB - which makes a lot of sense to me based on what I know about how EVA and NetApp filers are typically configured in SAN/block environments.
So to conclude: If NetApp and HP sold into equal-sized SAN environments (which premise I personally have serious doubts about but would be willing to accept for the purposes of keeping the argument simple) and if the NetApp report were accurate then you would expect HP's IDC SAN GB/unit number to be double that of NetApp. But it's not: in fact it's about half the size (53%).
So if you still believe the NetApp report is accurate then you have to also believe that NetApp sells into SANs that are approximately 4x the size of HP's. Personally, I find that about as likely as an elephant stampede in Zurich, but in the end it's up to each reader to decide for themselves. But now that I think about it, there is one more conclusion possible: that despite what the white paper says, it is NetApp and not the EVA that has the issues with usable capacity.
Jim Haberkorn, HP StorageWorks EMEA (Europe)