Around the Storage Block Blog

[#1]

By Jim Haberkorn

Question of the day: What would happen if Star Fleet used NetApp MetroCluster instead of the Kobayashi Maru as their no-win-scenario war game software?   

Scottie:  “Captain, I’ve just discovered that our primary site has six single-points-of-failure. What do we do?”

Captain:  “Failover to the secondary site immediately, you fool.”

Scottie:  “But, Captain, the secondary site also has 6 SPOF.”

Captain:  “Ha Ha, Scottie, I love a man who can joke in the face of disaster.”

Scottie:  “Captain, I’m not joking. When I bought the product I asked every question in the galaxy, but I never thought to ask about single-points-of-failure. Captain, it’s the year 2010. Who in their right mind sells HA products with single-points-of-failure?”

While you are pondering the IT implications of this scene, here are a few things to consider:

1. NetApp markets its MetroCluster solution as able to “Achieve continuous data availability for mission-critical applications at half the cost and complexity.”
2. What MetroCluster does is break up the no-single-point-of-failure NetApp system on your primary site and turn it into 2 separate systems, each with multiple single-points-of-failure, and place each system at a different location up to 100km apart. This is what passes for high-end, mission-critical HA with NetApp.
3. Your primary site now has a single NetApp filer that is orders of magnitude more likely to fail than the original system. And now, when it does fail, it will failover to a single NetApp filer which is also orders of magnitude more likely to fail. I am not making this up.
4. In other words, the NetApp idea of HA is to make your primary site more risky and then have you failover to a system that you wouldn’t trust your mother’s cookie recipes on let alone your company’s mission-critical data.

Okay, NetApp enthusiasts, your turn.  

[#2]

"What MetroCluster does is break up the no-single-point-of-failure NetApp system on your primary site and turn it into 2 separate systems, each with multiple single-points-of-failure, and place each system at a different location up to 100km apart. This is what passes for high-end, mission-critical HA with NetApp. "

Nice FUD but no cigar.

What MetroCluster does is integrate  a system over two separate geographic locations into a logical site with active-active failover, and remove all those single site failure events such as power, cooling form affecting the system.

I come from a place where we play Rugby - and could laugh at the big strong Football boys with their crash helmets and padding - but they are superb athletes simply playing a different game with different rules. In fact I enjoy watching both on their merits.

Why not tell us how you provide a two site scenario???

[#3]

So basically the author is saying that a storage system that provides similar functionality to a Microsoft Stretched Cluster is not suited for  Mission Critical apps and HA...

I say the post lacks credibility

[#4]

Hi Anonymous,

Thank you for your comments. Since I blog rather infrequently, I’ll make the same offer to you that I make to everyone who first responds to one of my posts – if you are ever in Zurich, contact me and I’ll take you out to dinner.

Now as to your comments: With all due respect I disagree with you on several points:

1. My comments were not FUD. I can back up everything I said. And I notice you didn’t contradict me, as much as you suggested that I didn’t understand that NetApp MetroCluster is ‘a different kind of game’.
2. I believe you picked the wrong sports analogy.
3. You misunderstood my argument: I wasn’t attacking the MetroCluster product. I was attacking the way NetApp positions it.

As to point one above: NetApp MetroCluster is a SPOF failing over to a SPOF. I am convinced that most NetApp customers don’t know that. For that reason, I think of this blog more as a public service than as a competitive attack. I have talked to NetApp customers who are convinced that a NetApp MetroCluster solution is just as fault tolerant as an EVA failing over to another EVA. They had no idea that a single NetApp filer had multiple single points of failure. They never saw that fact in any NetApp literature, were never told that by any NetApp representative, and never thought to ask that question during the sales cycle because it seemed like an idiotic question even to ask.

Point two: When it comes to mission critical data, NetApp may think they are playing rugby but every customer I’ve talked to thinks he is skydiving. What NetApp does is throw the customer out of the plane with two chutes – each with six holes in it. What customers want with their mission-critical data is a fully functioning main chute and a fully functioning backup chute. That’s what they get with HP and most other storage companies selling into mission critical enviroments. .

Point three: You cannot attack a product that is properly positioned. NetApp started out in 1992 as a single filer, with multiple SPOFs, targeted to satisfy departmental requirements. Later on they added clustering. What they’ve done with MC is reverted back to that departmental system and have it failover to another departmental-caliber system. The positioning should be: “Add HA features to your departmental solution.” That would be fair and accurate and un-attackable. Instead NetApp has taken a Yugo and marketed it as a BMW.

Best regards,

Jim  

[#5]

Hmm, so let me get this straight.  If I want ...

  - heavy professional services involvement (vs. little or no PS w/ NetApp),

  - configuration changes manually duplicated in the secondary array (instead of automatic w/ NetApp),

  - host-based failover disrupting my applications,

  - manual failovers with lots of scripting,

  - and no performance improvements for mirrored data

then I should go with HP storage, eh?  Thanks, but no thanks!  My business runs on NetApp MetroCluster!

[#6]

Hi Nick,

Thanks for your comment. With all due respect, your point would have been valid if it were still 1995 when Microsoft first introduced clustering. But Microsoft has made many improvements since then,  including greatly increasing the number of nodes in their stretch cluster. So let me respond to your point by asking two questions:

1. How many nodes can be in a Microsoft Stretch Cluster?
2. How many nodes in a NetApp MetroCluster?

Check this out if you’re not sure:  http://technet.microsoft.com/en-us/library/bb124721(EXCHG.80).aspx

But you force me to make another point. While Microsoft has improved their clustering in many ways since 1995, NetApp, which introduced its clustering about the same time, hasn’t. NetApp’s main clustering software is still 2-nodes. Also, allow me a small digression: During the past 15 years, Microsoft clustering and HA has improved not only through Microsoft’s efforts, but, because it is industry standard, has benefited from the efforts of many ISVs as well – something that NetApp isn’t able to take advantage of because of its proprietary OS. 

Best regards,

Jim

[#7]

Hi John,

Thanks for your comments. Since you are a NetApp customer, I have a question for you: Before you bought MetroCluster, did you know that your primary site had multiple SPOFs and that you were failing over to another system with the same multiple SPOFs. If so, and you were comfortable running your mission-critical data on that,  can you point me to the NetApp documentation that revealed that fact to you?

Also,  since you don't like complexity, let me ask: Have you ever performed a MetroCluster failback? And did you manage to do it without corrupting data? Here is the link to the appropriate NetApp MetroCluster white paper. It was just updated earlier this year. http://media.netapp.com/documents/tr-3548.pdf   

You might consider reading all of section 5 starting on page 35. Pay special attention to the section labeled "Split Brain Scenario" and then the following full page that gives all the manual steps. Question: Did you know about the failback issues when you bought MetroCluster?

Finally, thanks again for your comments. It's not every day we have a customer participate in these blogs. By the way, if you were referring to the EVA in your comments you might consider contacting an HP sales rep and double-checking some of your facts.

Best regards,

Jim 

[#8]

I would like to thank, again, all those who have commented on the blog so far. Calvin Zito tells me that the blog has caused quite a stir on Twitter. I would like to invite all those on twitter who are interested in playing with live ammo, to come on the blog and run their arguments by me. I personally believe that debate is healthy as long as it doesn’t descend into personal attacks.

In my opinion, pointing out a competitor’s legitimate weakness is not vendor bashing if done properly. Customers can’t be expected to inform the public of a vendor’s weakness, and oftentimes analysts don’t own the equipment themselves, so they’re usually not in a position to do it either. So, that leaves the competitors to more or less police each other.  And if it’s done in a blog, and is done a little humorously and with a little panache, all the better. That’s my philosophy.

Best regards,

Jim

[#9]

@Nick, the issue with Metro cluster vs MS stretched clustering is you can only ever have a max of 2 nodes with the Netapp config, one node per site. A single node per site equals no high availability on either site. Whenever I've been involved in stretch cluster configurations we'll go for a high availability config on at least the primary site. When it's an active / active config, then you want a minimum of 2 nodes per site to provide local H/A, last thing you typically want to do is to failover unnecessarily, especially over distance. Failback is generally the biggest challenge and you'll want to avoid having to do that in most situations.

@John, As far as I'm aware MetroCluster has always and still does require manual failover (takeover) iand failback (giveback). So it only handles very specific failure types in a transparent way. http://communities.netapp.com/message/29840 

http://communities.vmware.com/thread/237786

 >heavy professional services involvement (vs. little or no PS w/ NetApp)

Metro cluster isn't customer installable, even the professionals struggle with it, I've seen them.

 > configuration changes manually duplicated in the secondary array (instead of automatic w/ NetApp),

Not so both EVA CA and P4000 duplicate changes to the remote site array.

 > host-based failover disrupting my applications,

Not sure what to make of this one, but as above, Metro cluster is not an automated failover solution, it requires manual intervention or bespoke scripting. Whilst waiting for said intervention your applications are offline. See CLX and P4000, also SVSP below.

  > manual failovers with lots of scripting,

CLX with EVA , P4000 Network Raid or SVSP, all support high availability per site and automated site failover without the need for scripting. CLX supports both storage and clustered application failover, P4000 and SVSP provide transparent failover. http://goo.gl/0F88, http://goo.gl/pK2k, http://goo.gl/j2aT

 > and no performance improvements for mirrored data

Again bit of a vague point, but regardless, all of the above HP solutions provide as a minimum dual active / active controllers per site. I'd suggest local site load balancing and locality of access as well as local high availability are all arguably more important than anyone else's Raid (insert you preference) vs RAID-DP debate.

Lets' get real If you do this over distance by stretching the cluster between sites,  what you save in filer head costs you'll probably need to shell out in back end fibre switching, multiple dark fibre links and the professional services to implement.

[#10]

@John_H AFAIK, stretched sync domain of SVSP is like the NetApp Metro Cluster  - no HA on a single site, no automated transparent failover in case of site failure (need to issue single command for failover). But HP neither hiding this nor calling this "Metro Cluster".

[#11]

Hi John_H,

Great points. And thanks in particular for answering the HP product questions. I didn't want to get side-tracked off of MetroCluster, but it was totally appropriate for you to weigh in.

Best regards,

Jim

[#12]

When people talk about Single Points of Failure, they're often referring to hardware components (PSUs, Fibre Channel / Ethernet ports etc).

What about the controller operating system / microcode?

If you have one kernel which running across multiple controllers, isn't that a single point of failure?

I've seen plenty of software / microcode bugs take down an array -  but only very rarely have I seen a hardware issue take down an array (and in that case, it was actually because someone tripped the circuit breakers for the whole datacenter, so it was more a human error than anything else.)

A lot of people trash talk about netapp controllers not being "true active / active", because they don't share disks etc. But I've always liked the way netapp runs seperate kernels on each controller in a cluster - no SPOF!

[#13]

To continue on with MetroCluster:  The title of my post was "NetApp MetroCluster - Be sure to ask if the car has wheels."  I chose that title carefully.

When buying a car, most of us would never think to ask if the car had wheels. You assume it has wheels. The same thing goes with a solution marketed as offering "continuous data-availability for your mission critical data". You assume it has no SPOFs. And NetApp marketing is very content for customers to make that erroneous assumption. 

Here is what I mean: 

NetApp has a 55-page white paper on MetroCluster titled: “Best Practices for MetroCluster Design and Implementation.”  It captures every imaginable fact you would ever think to know or ask about MetroCluster – except one: That the two systems are each the proud owner of multiple single-points-of failure. But maybe NetApp doesn’t think having SPOFs at one site are important. No. In fact, NetApp thinks just the opposite. Fabric MetroCluster requires four switches – two at each site – for redundancy the white paper says (page 12).

Now, why is switch redundancy at each site so important but filer redundancy not? The answer is that filer redundancy is even more important. If you had one switch at each site and that switch went down, no user would lose access to data and no failover would be necessary.  But if you lose one of the MC filers, users do lose access to data and you do have to failover. So, again I ask, why insist on site switch redundancy but not site filer redundancy in MetroCluster?

The answer is that NetApp can solve the switch redundancy issue, but because it is stuck with 2-node clustering, it can't solve the filer redundancy issue - so it keeps quiet about it in its literature. And that is the point of my post. NetApp clearly recognizes the problem, loudly solves it where it can, but then keeps remarkably silent where it can't.  And I would suggest that because MetroCluster is marketed as suitable for Mission-Critical data, that that is a disclosure that needed to be made. If a car doesn't have wheels, customers need to be told upfront.. 

In my next post, I will tell you what the SPOFs are in a single NetApp filer, and also how you can find out for yourself.

Best regards,

Jim

[#14]

So what are exactly those 6 SPOF's per site? I can only think of 1 in a metrocluster setup,.. the node itself. You have mentioned the 6 SPOF's a couple times, but haven't explained what they are. 

And manual failover in case of a site failure should always be manual since there is no automatic way to prevent a split brain scenario. 

And another question,.. if Netapp metrocluster is so bad.. why does HP position their iSCSI product (lefthand) as a cheap alternative to Netapp  Metrocluster? Would that mean that the lefthand product is a cheaper version of a bad product????

[#15]

Hi Jim,

So first, thanks for the dinner offer, the discussion will be lively.

I chose my sports analogy deliberately, because of the different rules.

To put it simply NetApp has two solutions to multiple site  design;-

 Metro/Stretched Cluster and independant replicating systems at each site. These can be HA(no SPOF) or single at either.

In the event NetApp is required to provide a HA(no SPOF) solution at both sites it can and is easily done with any FAS or V-Series cluster, which can be different models, sizes, disk or FAS to V-Series. refer to http://media.netapp.com/documents/CS_NETAPP_DISASTER_RECOVERY.pdf for an example.

HP only has this HA(no SPOF) type of solution available and in fact there is less flexibility - no V-Series virtualization equivalent for instance.

So Metro/Stretched cluster is a different and additional set of options - which you are choosing to target based on your dual HA rule mindset - which simply highlights HP lack of comparable solution hiding behind a classical - "we don't do that!" pitch. Fair enough - so pitch in why yours is better (but remember NetApp has that solution too), but do not feed in statements referring to the systems as independant, when they form a cluster.

Now for something else.

We all know that by far the largest component of system downtime/error etc is human failure. We all try and provide systems which are reliable, flexible and self or quick healing. We all examine scenarios and raise alarm at possible negative outcomes, so we can avoid or plan correction. What post event analysis reveals is that most human failures are not deliberate, or true accident, but activity which has unintended and unplanned cosequences ie leaping before properly looking. The first rule is "Do no harm".

SO

As a long term Trekkie I will take issue on your characterisation of Scottie and Captain (I assume Kirk?) as leaping to the conclusions you did. Experienced specialists like them would know better than to deliberately fail a system that is working. Scottie would also have fully analysed the system and evaluated the consequences before providing a professional report - not made a half-cocked run to the Captain generating panic.

This brings me to the reality check - not the competitive feature/rule bashing. A simple single system with lots of SPOF can and often does run for years without any problems or incident. It is always a business decision (yes I agree with proper technical input including proper diclosure) that surrounds the additional investment in reducing failure points, increasing flexibility, providing HA, DR, BC etc. The risks need to be balanced against the costs, and having more options is a good thing. If you restrain youself to simple analysis and question whether the solution is suitable it will be more usefull to your audience.

[#16]

I am from NetApp, so let's get that straight out there before I post my response and not in Zurich any time soon...

You have either chosen to mislead people here, which I'd be surprised by, or you just haven't understood this one of many features we offer for cross site availability, in which case let me quickly educate you.

A Metrocluster is NOT two separate systems as you portray, it is an HA system where we simply offer the capability to separate the two halves of the controller up to 100km, in the extremely rare event that, for example a shelf should fail in one half of the cluster then the corresponding shelf in the other half simply steps up. It is NOT two systems with SPOF's but one Active / Active system with NO SPOF's. As long as the Network in between has no SPOF then a split brain scenario would also never occur, but we make sure that this is documented should a customer’s fully redundant WAN become an issue.

This is a unique solution, which we appreciate isn't right for everyone, which is why we offer Replication between HA Pairs over FC or IP as other options where this may be more suitable.

To your point about NetApp being a proprietary OS and therefore not able to take advantage of Microsoft Clustering advances, what are you talking about? could you remind everyone on this post what proprietary OS's EVA runs? and XP? and P4000? and of course SVSP? etc. etc. etc. we integrate extremely well with all of MS Clustering technologies where this is an appropriate solution as well.

Can you please re-clarify exactly what the point is you are trying to make here? or has my explanation helped you to understand how this one specific unique NetApp feature works?

Regards

Matt

[#17]

D from NetApp here.

I think this is becoming a semantics issue.

Metrocluster shops don't see it as 2 different sites - they see it as a single datacenter entity that happens to have components geographically dispersed a bit (this is an important nuance).

Disk reads can, indeed, come from both sides of a MC setup, so we see MC actually accelerating many workloads since the other side is not dormant but rather active. Which emphasizes the above point - it's essentially a single logical entity.

A controller failure in a MC shop results in what normally happens with all NetApp controllers - a fully automatic failover occurs.

Customers that want very high availability have chosen this solution since it's extremely effective yet far less expensive than competitive solutions. If a manual failover is needed, a single command (that can also be automated) is run to make either side the primary for all operations. Far easier than failover with other products

D

[#18]

Thanks everyone, you made some excellent counter-points. I’ll respond to your comments in my next post. In this post I’ll talk about SPOFs.

A single NetApp filer has multiple single-points-of-failure. But curiously, I’ve never seen them listed in any NetApp document. I have seen NetApp documents that talk about a single controller being a SPOF – rather obvious, I would say – but nothing more concrete than that.  

However, fortunately, NetApp has an OEM partner with a different marketing philosophy – IBM. On page 463 of the IBM N Series Redbook in section 22.2.7 titled: “Eliminating Single Points of Failure with Active/Active Configurations”, you will find a table showing the NetApp single filer SPOFs. Clear, concise, understandable. They are:

1. The N Series system itself
2. NVRAM
3. CPU fan
4. Single NIC
5. FC-AL card

Now, not all of these apply to a MetroCluster filer. For example, the FC-AL card and the single NIC can be easily eliminated. That leaves NVRAM, CPU fan, and the filer itself – or rather the processors in the filer – an important point. I don’t know how NetApp calculates its 9’s capability, but HP does it by calculating the failure rate of each component in the system. So, in our 5.9s calculations we don’t look at the failure rate of a controller, but of the processors and other components inside. NetApp filers have 2-4 processors. Each one is a single point of failure. They are the same processor but they are not redundant. Lose any of the four and your filer goes down. Same with the NVRAM. Each DIMM is a SPOF. As for the CPU fan – unless someone wants to tell me they have dual CPU fans now, there is another SPOF.  And then you can throw in the OS if you wanted, as one blogger suggested.  

So, how many SPOFs are there? Whether it’s 4 SPOFs, 6 or 10, what difference is it – the point is made.

NetApp enthusiasts might say that as long as there is another filer to failover to, the SPOFs are not a problem. I respectfully disagree. First, if a customer didn’t have MetroCluster, but had just a standard two-node NetApp system at the primary site, their chances of losing that system and having to failover to a secondary site are pretty small. However, as soon as they swap that 2-node system for a single filer, their chances of having to failover to the second site increases by orders of magnitude. And in real life, you don’t want to do failovers or failbacks over distance. if at all possible. It’s an emergency procedure. It is not a trivial exercise. The failbacks in particular can be very complex, carrying with them the definite possibility of data corruption. For this reason, your first objective in constructing a mission-critical solution should always be to have a rock-solid primary site.

Here’s the second reason: In a 2-node cluster, after the first filer goes down, your mission critical data is now on a filer with multiple SPOFs and which is now handling all the mission-critical data and IO traffic from two filers. It is a departmental-caliber solution by anyone’s definition. A proper mission-critical solution should be mission-critical at both ends – in my opinion.    

Does all this mean that MetroCluster should never be sold? No. But is all this information that should be clearly understood before a customer buys? Yes? And that is the purpose of this post.

Earlier, one blogger asked me what the HP solution was. Well, we have several, but if you are talking about the mid-range, we would have a totally redundant EVA at the primary site and a totally redundant EVA on the secondary site. If your first site goes down, your mission critical data would be sitting on an EVA with all the performance and internal RAS of the EVA on the primary site – a mission-critical solution at both ends. 

Best regards,

Jim

[#19]

Hi Jim

Yes, I work for EMC.  And yes, I've been known to take a swipe or two at the various NetApp "solutions" for their tendency to be overpositioned at times :-)

That being said, I had never downloaded the NetApp pdf you linked to.  Maybe I should have.  Like you, I was a bit shocked at what was being positioned as "high availability".

 At a high-level, they've taken a standard two-node controller design, and attempted to stretch the failover logic at a distance. 

Imagine -- a network issue *outside* the data center could compromise availability *inside* the data center.   This isn't about fine grades of "better", this is more about "fit for intended purpose".

We may have had our differences in the past, but I applaud you for pulling on your asbestos suit and giving this the attention it deserves. I would bet that most of the people running this have no idea what they're potentially exposed to.

Auto manufacturers make safety recalls -- how about IT vendors?

-- Chuck

[#20]

Thanks for your excellent counter-points. I appreciate everyone’s efforts to keep this blog cordial and interesting. I think it is accomplishing what blogs between competitors are supposed to do.

First, I have to admit, Anonymous may have gotten me with his Star Trek comment about Kirk never failing over from a functioning system. The best I could come up with was failing over from impulse to warp drive but even I could see the holes in that argument. Score one for Anonymous.   

Now as to everyone's counter-arguments: I can’t answer all your points (it’s an issue of time and space and the quantity of your points) but I will address what I consider the top four.

1. MetroCluster is not two separate systems but one system.
2. NetApp offers MetroCluster and other fully redundant HA solutions. The customer gets to choose.
3. Hardware failures are rare, so why worry about the SPOFs.
4. With the P4000, HP offers the same kind of solution as MetroCluster, so who are we to throw stones.

Point one: MetroCluster is a single system. Response: I will argue that point with you later, but first, my answer is “So What”. Who cares if it’s a single system or two systems. The customer has a problem. In this case, he’s got mission-critical data that he wants to protect from a site disaster. Now, how are you going to solve it? Well, in NetApp’s case, whether you use MetroCluster or one of their other HA solutions, you are going to have either one or two NetApp filers at one site and either one or two filers at another site some distance away. To the customer, it quacks like a duck – it’s two systems.

Now, is MetroCluster really a single system?  It’s a 2-node cluster, with two separate OSs, two separate root drives, shared nothing storage, you can’t read drives from one filer through the host ports of the other filer, and you have separate controller failover software (roughly $30k for a FAS6080 if I’m not mistaken), with each node placed up to 100km apart. Sounds like two systems to me, but I’m not an engineer.

Point two: NetApp offers MC and full 2-node systems at each site. Customers can choose. Response: Good point. And this hits at the root of my argument: My contention is that NetApp is on the pricey side and sells MetroCluster into environments it really doesn’t belong to keep the price down. In other words, I’m not criticizing MetroCluster, but I’m calling into question how it is marketed and explained to customers. Now, I’m willing to give all the NetApp employees on this blog the benefit of the doubt that they would never mislead a customer about the capabilities of MetroCluster. But, it happens.

Point three: Hardware failure are uncommon, no need to unduly worry about them. Response: Not much I can say to that argument. But I know that Intel and AMD and Dell, in fact, all vendors, have been hit by hardware quality problems. I hope your luck holds.  

Point four: When I’m criticizing MetroCluster, I’m really criticizing the P4000 architecture as well. Response. Check out the size of the P4000 clusters.

Best regards,

Jim

[#21]

Ok, so let me start out by saying that I work for a major NetApp reseller, and have actually sold MetroCluster to a very large customer in the U.S.  and been involved in some smaller deals as well. Yes, that makes me a rare beast here in the U.S. where not much in the way of MetroCluster is really going on, it seems to be more popular in Europe.

So, first let me take exception to the basic premise that you put for that we don't tell NetApp customers about the "SPOF" in MetroCluster.  I always go over the MetroCluster architecture with them in detail during the pre-sales process, and a discussion of failure scenarios and failover is always part of that discussion. The customer always knows exactly what will happen if a controller fails, a disk shelf fails, etc. before we sell them the solution. 

I can tell you from experience, that those customers who bought MetroCluster from us, they were aware of everything you point out (and I think you're putting forth a completely semantic argument here), and they don't care. The way that they see it is that they have two data centers that they treat as one big data center, and to them failing over from one data center to another data center is no big deal. I would also like to point out that there is a NetApp product that will allow you to set up automatic failover between the nodes in a MetroCluster if the customer wants that to happen. Again what I can tell you from experience is that most customers don't want that, they make an informed decision that they would prefer to control the failover manually (single button).

Bottom line is that there are any number of rocks I could throw at the MetroCluster technology, but what you point out here isn't where the real issues are in the real world of customers who just want to be able to solve a problem.

Regards,

Joerg

[#22]

Jim,

I work for EMC, I read the NetApp document you posted and from what I can tell MetroCluster is an okay solution but the physical configuration is quite complex.  I can't comment on HP's solutions, but looking at EMC's portfolio it seems to me that it would be easier to deploy a pair of Celerra Unified systems with MirrorView/Sync.  Each site would have full HA hardware inherent in the EMC platform, synchronous mirroring between the systems, and automated failover between the two sites for NFS/CIFS/iSCSI/FC data.  CIFS and NFS exports will come up on the other side exactly as they were in the primary site as well as the block LUNs.  And the whole thing can be managed with a single Unisphere GUI (one window) with more configuration flexibility.  For even higher availability, using VPLEX, block data would be read/write at both sites simultaneously with any storage on the back end (HP, NetApp, HDS, EMC, etc).  No failover delay at all.

[#23]

Richard,

As someone form NetApp pointed out earlier. NetApp also has a solution like that where the customer buys two identical systems and sets up sync SnapMirror between them and they are done. MetroCluster solves a different problem. Specifically where the customer has two data centers (or just here they want to "stretch" the array to different parts of the same data center) that are close toegther and wants to save money and not have to buy 4 heads. It's a choice that the customer makes understanding how things work after we do explain it to them.

What it boils down to is choice, the customer can save some money under the right circumstances with MetroCluster, or if they perfer, they can do the same as what you are suggesting and spend a bit more. The bottom line is the customer gets to choice, and they confirm the value of the solution with their dollars.

--joerg

[#24]

On the P4000 front, you can build a similar solution by stretching the P4000 cluster. However you aren't limited to just a single node (controller) per site, so you can build a solution with both local high availability and remote failover for either site, one of the benefits of Network Raid.

As it's a single cluster you manage the two sites as a whole and the multi-site cluster can also be made site aware if you wish, ensuring host to storage traffic remains local to it's designated site during normal operation.

If you want both automated failover and failback, then there's a failover manager included to overcome the split brain scenario by acting as arbitrator whilst ensuring quorum is maintained during a site outage.

Not suggesting this competes with the larger Netapp solutions, but it does seem to have overcome many of the inherent issues of stretching a dual node cluster, best of all, this functionality is included in the base cost of the P4000.

[#25]

I have deployed a few Metrocluster configurations and Customers love it.

Its not for everyone but for those wanting a stretched DataCentre across multiple sites it is fantastice

Here are the "SPOF" and how they are dealt with

1) Disk failure  -   Normal Raid

2) Shelf Failure - Read from the shelf on the other node

3) Switch Failure - Redundant switches

4) Controller Failure - Normal Cluster Failover

5) Fibre Failure - Redundant Fibre links geographically diverse

6) Entire Site Failure - One command to fail it all over

Pretty simple, the fact is MetroCluster is a good fit with customers that require active/active datacentres with 5 x 9 availability. (My metrocluster installation has been up for 400 odd days without a reboot. How many HP and EMC people could say the same) It runs Oracle, VMware, AIX. It runs all of their mission critical applications and suprise suprise it beat out HP, Sun and EMC in the tender process.

Anyway, Its too early in the morning for me to remember all the other area's where there redundancy have, ill let someone who is on the other side of the world where its not so early comment on the other examples of redundancy

[#26]

Hi Joerg,

Thanks for your comments. Let me make it clear: I wasn’t talking about you personally in my posts. I’m sure you honestly explain MetroCluster to all your customers – maybe that explains why you’re not selling much of it. But one question: What NetApp MetroCluster documents did you use to explain the risks/SPOFs? Could you point me to them? I can’t find any.

Also, when you explain all the issues with MetroCluster, do you also tell customers that NetApp dedupe will kill their performance, that NetApp’s usable capacity is the worst in the industry, that their capacity guarantee program is bogus, that they have a draconian licensing scheme that keeps NetApp software gross margins up at 98%, and that their filer performance degrades over time. Those are important too.

Best regards,

Jim

[#27]

Hi Richard from EMC,

Thanks for clearly identifying yourself as an EMC employee – I never would have guessed. Wouldn’t it have been easier to just post the URL of the EMC website?

Best regards,

Jim

[#28]

Hi John_H,

Thanks for your comments. I'm glad you mentioned the P4000 all-inclusive software – that with the P4000 you get clustering, Network RAID, RAID-1, cluster failover, large clusters, single-management for large clusters, thin provisioning, cloning, snapshot (including restores and writes), dynamic storage tiering, and remote copy as part of the base price. The reason why that is a good point to bring up is that NetApp has the highest gross margins that I’m aware of in the industry and does it mainly on the back of its confusing and draconian software licensing schemes – a common complaint from NetApp customers, channel partners…and ex-NetApp employees.

According to NetApp’s Q4 FY10 financial results, NetApp software gross margins were 98.2%. There are several ways NetApp keeps the software gross margins so high: One way is to parse a lot of their basic software functionality into multiple software products and to charge for things no one else charges for. For example, to get the same functionality you get with a single EVA Business Copy license you would have to buy:

2 SnapRestore Licenses and 2 FlexClone Licenses.  And then two more licenses if you want the controller failover (CFO) software that comes free with every other array I’m aware of.  

Turning to the P4000 and its all-inclusive software, the story gets even worse for NetApp. The prices I’m quoting are from the CPstorage website based on FAS3140 T3C pricing at 26% discount:

1. Cluster Failover – $8,880
2. FlexClone - $8,806
3. SnapMirror - $18,648
4. MetroCluster (software only) - $5,920
5. SnapRestore - $5,880
6. Operations Manager - $1184

The total comes to $49,318 – a pretty hefty price tag. Oh wait, I forgot, NetApp charges a separate license fee for both filers in the cluster, so the real total is $98,636. And, by the way, there you have another argument for why it makes more sense to think of a NetApp MetroCluster solution as two separate systems not one. Note: On a FAS6080, NetApp’s biggest 2-node cluster the total is $292,722.

Best regards,

Jim

[#29]

Hi Paul,

Thanks for your comments. And thanks for proving my point. Even you, someone who sells NetApp and MetroCluster doesn't know what are the single-point-of-failure. But don't feel bad. In my experience very few NetApp people do. It's a well kept secret.

I'm happy for your MetroCluster customer that he has gone 400 days without having to failover. But when he does have to, he will face a daunting task when he fails back. The details can be found on pages 36-37 here: http://media.netapp.com/documents/tr-3548.pdf   

It's too long to post the whole procedure, and it can't be summarized because it is 2 pages of steps. But here is a sample: 

Turn off power to the previously failed node (disk shelves should be left on).

 Disconnect the cluster interconnect and Fibre Channel adapter cables of the node at the surviving site.

 Use network management procedures to enable the storage systems at the disaster site to be isolated from the external public network.

 Use any application-specified method that either prevents the application from restarting at the disaster site or prevents the application clients from accessing the application servers at the disaster site. Methods can include turning off the application server, removing an application server from the network, or any other method that prevents the application server from running applications.

So for your customer's sake, let's keep our fingers crossed that his good luck holds.

Best regards,

Jim

[#30]

Hey Jim,

Thanks for your kind comments.

In actual fact, the customer is able to failover between nodes within minutes for a full site failover so your comments around failover are over cooked and purely scare mongering for those who don't know Metrocluster as a solution and the area that it suits.

Rather then bagging out another solution perhaps you should do a comparison on what HP offers compared to NetApp.

As it stands all you have done is created a blog based on FUD and as you work for a competitor vendor, your credibility goes out the window much like Scott Lowe did with his blog about NetApp should see acquisition.

Cheers

Paul

[#31]

To summarize:

In NetApp's mission-critical MetroCluster site disaster solution, this is what you get:

1. A system on the primary site that has multiple SPOFs

2. You then failover to a system on the secondary site that also has multiple SPOFs.

3. Because MC is supposed to protect from site disasters, you might be on that secondary system for days.

4. When you do failback, you face a nightmare, manual process where one false step would result in data corruption.

No respondent in this blog has yet refuted those basic facts.

No other vendor in the industry defines 'mission critical solution' the way NetApp does.

Best regards,

Jim

[#32]

Hi Paul,

Thanks for your comments. In the world of competitive blogging there is one sure way to know if you are winning the argument – it’s when your competitors fall-back to accusing you of spreading FUD. And frankly, being compared to Scott Lowe, doesn't bother me. He has a good reptutation in the industry. And, by the way, he was spot on about NetApp.

The other way you can tell you’re winning is when the only way they can score points is to deliberately misunderstand what you are saying. Go back and read what I’ve said:  I’ve criticized the failback process not the failover.

And again, you still haven’t refuted my basic points. MetroCluster is a system with multiple SPOFs failing over to another system with multiple SPOFs. I think we all agree on that. If that’s your definition of mission-critical that’s okay with me. But I think NetApp should be more upfront about the reality of the solution in their literature.

By the way, since you brought up FUD. There is something even more common in the industry than FUD. It's something I call LODE. It stands for Leave-out, Deny, and Exaggerate. And it is what some vendors use on their customers.

Best regards,

Jim

[#33]

I just looked over the various comments and came to an interesting conclusion - that is, that the NetApp enthusiasts have ended up proving my point.

Here's how:  After 31 comments to my article and a 1000 twitter messages, we have now come full circle to the original point I made in my article, i.e., that MetroCluster is a system with multiple SPOFS failing over to another system with multiple SPOFS.

And look at the counter-arguments the NetApp guys have come up with!  It makes you wonder how many holes a mission-critical product would have to have before they would stop marketing it as mission-critical 

And that is my point: NetApp vigorously promotes MetroCluster as a mission critical product.  And they leave crucial details out of their literature and are relentless at devising arguments to defend even the indefensible.

Best regards,

Jim

[#34]

Hi Jim,

There a thing I don't get in starting discussions like this. If you want to make a point by starting it, make sure you have your facts straight. If other collegues (from whatever vendor) make comments on it, just respond on the facts and don't repeat the bogus points time after time.

I'm an independent storage architect guy and worked with quite some different storage systems. (EMC DMX, HP EVA, IBM DS, IBM XIV, N-Series / NetApp and I think I know where the frustration comes from I read in the blog.

Not one of 'Big' names from the past is able to provide the customers with a range of products like NetApp does. The storage admins feel themselves like in heaven because of the simplicity in administrating a NetApp storage environment. Standarized gui / cli over the whole range of storage products. And you know what, the NetApp's do the job. How weird is that? I've got many customers who use NetApp / N-Series for their storage-needs and their high-available infrastructures and they are all very pleased with what they experience. Even after site-disasters they are very impressed with the ease and speed with which everything is back to normal again.

And don't you think it's very strange that if the design of the NetApp stuff in your eyes is not ok, so many customers switch over from HP EVA to NetApp? The last three projects I did where like that.. I I think I'm not unique in that.

Cheers, Rene

[#35]

just one of my metrocluster customer , smaller FAS3020 fabric metrocluster , uptime 997 days , no takeover , no updates no failures during that time , system was setup and it runs since that time ... reliable .

Sun Jun 27 00:00:01 GMT [sxxxfxxx: kern.uptime.filer:info]:  12:00am up 997 days,  3:55 1 NFS ops, 64021575643 CIFS ops, 1378 HTTP ops, 10701213973 FCP ops, 0 iSCSI ops

P.S. the customers superdome went down twice during that time

[#36]

P4000 lefthand?! similar to equal logic really - if you want to add another array and another device to manage each time you want to add capacity that is great. How long can you keep managing that for? 

Also noticed talk about hp snapshots...are they application consistent like NetApp or EMC?! I believe not (please correct me if wrong)

So how about you tell me some good things about hp storage rather than fud about others. My personal understanding and experience is EVA = poor perf and functionality (from experience and customer feedback), left hand is far from enterprise. Wait, hp xp is pretty good! Oh yeah that is HDS.

Actually I've seen other resellers deploy some EVAs recently, but with a falconstor appliance on top so the customers actually get some functionality. EVA is picked as the back end storage only because it is cheap.

But wait EVA released a vsphere plugin a few weeks back...only a few years after EMC and NetApp.

Funnily enough about 2 years ago, when a customer rather than a reseller, someone tried to sell me a HP EVA 8000 - mouse clicks was the best whitepaper they  and selling item they provided. Funnily enough the shortlist ended up being a CX3-40 and a FAS3140

Finally what about hp and unified storage. NetApp are there already, EMC are pretty much there but improving the so they have one management platform, how about hp? - BTW EMC employees out there unisphere, when released to customers, will not truly allow all mgmt from one console, only some of the management from one console. There will still be a need to go to navisphere or Celerra Mgr for a lot of things.

I think Joerg Hallbauer explains it best. Metro clustering has its uses cases. 

Finally Jim I would love to hear your logic (illogic) behind the comments below. Please advise on your sources. I actually thing this netapp blog actually explains how dedupe improves performance http://blogs.netapp.com/dropzone/2009/06/dedup-for-speed-higher-performance-through-deduplication.ht...

"NetApp dedupe will kill their performance"

"NetApp’s usable capacity is the worst in the industry"

Ben

Reseller with a  Preference for NetApp or EMC Storage 

[#37]

Hi Jim, this is my first-ever blog comment as in my experience such blog discussions lead to nothing more than just twisting each others word (I don't know if this is the right translation for a German idiom, but I guess you get what I mean). I don't expect to satisfactory answer any of your questions or turn your oppinion about MetroCluster, but as one of NetApp's MetroCluster experts I could not resist to comment anyway. I assume you are a Techie all way through? You completely focus on technical details only. Does hardware fail? Sure, it's made by humans. Does software fail? Sure, also made by humans. Do processes fail? Well, sure. They are performed by humans. From these three areas, it's the latter the customers really care about. You could build the most sophisticated system in the world, at some point it will fail, even if it is just the process to handle it. The overall question about MetroCluster is not if it has technical weaknesses and where those are (as every single system has those and someone denying this is just overly ignorant). MetroCluster is about solving a business problem, not about building the unbreakable system nobody ever can build. Is the NetApp MetroCluster solution better than anything from HP, EMC, HDS or any other storage vendor? I won't argue with you. It's the customer who makes this decision (and yes, they are well educated about MetroCluster and the pros and cons. They know of the lack of local Controller-HA, but it's meaningless compared to the big picture and the problem they want to be solved). If a technical solution is successful in the market does not depend on how perfect it is, but solely on customer acceptance. Is VHS better than Video BETA? Is DAT better than MiniDisk? Is BlueRay better than HD-DVD? Is the Otto engine better than Wankel? Is Windows better for desktops than Linux? (these are all just examples, I don't want to start a religious war on any of these ;-)) Both solutions in all the examples above are in some kind comparable, but only one made the break-through in their market segments. It's about "Which solution is good enough to solve my problem"? The "good enough" simply refers to cost and complexity. What do I get for my money and what does it cost me to handle it? But I don't think I'm telling anybody anything new. In current IT environments and the omnipresent virtualization and cloud hype, MetroCluster is a solution to address cost and complexity in a way no other storage vendor can do today with their legacy replication-between-two-arrays approach. NetApp does not need to defend MetroCluster. It's a perfect solution for customers chosing it, otherwise they would have chosen differently (be it NetApp with SnapMirror or any other vendor with his replication solution). Also, I don't need to convince you about MetroCluster or any other HP employee. I'll convince the customer. Ask your German sales colleagues how well that works. Kind regards, Niels

[#38]

Thanks, everyone, for your comments.

A quick note: Most articles on this blog are HP product-related articles. But, I'll be honest, occasionally we focus on a competitor product.  If anyone is looking for a article on an HP product, there will be another one coming along shortly. In the meantime, if you are philosophically against talking about competitor products feel free to consider this article, as I do, under the category of public service.

My claim is that MetroCluster is a system with multiple SPOFs at one site failing over to another system with multiple SPOFs at another site. Therefore, it shouldn’t be marketed as a mission-critical, zero downtime solution.

Also, as Chuck Hollis correctly pointed out, it has the additional drawback of being a “mission-critical, zero downtime” system that allows events outside the datacenter to impact the HA of the system within the datacenter. Good point, Chuck. So, my point is that MetroCluster is marketed incorrectly. I never said it didn’t work or that it broke down regularly. As Chuck said, the problem is, ‘it’s not fit for intended purpose.’  I think that is glaringly obvious.

In a discussion such as this, NetApp enthusiasts logically have only four choices:

1. Prove my facts wrong, i.e., there are no SPOFs, no nightmare failback, no greater chances of a failover, no network hits on system HA, etc.
2. Prove that NetApp really doesn’t market MetroCluster as an HA mission-critical solution designed for zero planned and unplanned downtime.
3. Or, failing all that, at least prove that NetApp sufficiently explains to customers the hazards inherent in the system – by pointing to the literature.  
4. Or, failing all of the above, state by what possible stretched-out definition of the term ‘mission critical and zero downtime’ can a system with SPOFs failing over to SPOFs, etc., etc., be considered satisfactory.

I’m still waiting for the response that does any of those things. So, if you want to keep this tango going, that’s fine with me, but I think you’ve taken your best shot. So here is where we stand: I am more convinced than ever that MetroCluster is marketed incorrectly. It should be marketed as adding HA features to a departmental solution. Marketed that way, it is unattackable.

But, so far, here are your arguments:

1. Our filers don’t fail anyway. Note: Wow, you must save a lot of money not needing a service department.
2. Other vendors’ 16-node cluster solutions have the same problem. Huh?
3. It is a single system – you just don’t understand the NetApp technology. (Yes, I know, I know, and WAFL is not a file system)
4. Why don’t you talk about your own products? Answer: But then this wouldn’t be a Competitive article.
5. You are just spreading FUD. (Is that worse than spreading the LODE – Leave out, Deny, and Exaggerate?)
6. We always market it accurately even though we never mention the SPOFs in any of our customer literature or technical white papers. (Yes, technical details are always best handled through word of mouth)
7. And my favorite – customers have bought it therefore it fits a customer need (I won’t bother explaining the circularity of that argument – in fact, I’m tempted to start using that argument myself whenever anyone criticizes an HP product,  – “you must be wrong about the problem or customers wouldn’t have bought it”)

I really am amazed that NetApp developed the product in the first place, - at least as a mission-critical zero downtime product.  And I keep thinking that there must have been a good reason. So, here’s a question for one of the NetApp guys. MetroCluster is bidirectional. Correct? Either side fails then the other side can pick up and go. What is the NetApp bidirectional solution when there is a full 2-node cluster at each site? There is a reason I'm asking this question. I'm working on a theory.

And finally, despite Ben's request, I wouldn't advise the NetApp folks to get into a discussion with us about NetApp performance. You already tried that once and it didn't work out so well for you. After one NetApp enthusiast told us our attack was sleazy, and another one famously said, "I'll let Avanade argue my case, " we got hold of Avanade (the company that does NetApp's performance testing) and Avanade agreed with us. It's in the articles below.

WAFL Part 1

http://h30507.www3.hp.com/t5/Around-the-Storage-Block-Blog/Making-sense-of-WAFL/ba-p/41218

WAFL Part 2

http://h30507.www3.hp.com/t5/Around-the-Storage-Block-Blog/Making-Sense-of-WAFL-part-2/ba-p/41255

WAFL Part 3

http://h30507.www3.hp.com/t5/Around-the-Storage-Block-Blog/Making-sense-of-WAFL-Part-3/ba-p/41962 

WAFL Part 4

http://h30507.www3.hp.com/t5/Around-the-Storage-Block-Blog/Making-Sense-of-WAFL-Part-4/ba-p/42045

Best regards,

Jim

[#39]

Hi Jim

Thought you had fun tearing apart MetroCluster? 

Wait until you dig into NetApp's "secure multitenancy" that is arguably neither secure nor multitenant.

And there's a few more to go after that ...

Cheers!

-- Chuck

[#40]

Funny how you didn't make any mention of:

HP unified storage

HP application consistent snapshots

HP releasing Vcenter plugin a few weeks ago - over 12 months after netapp and emc 

Also if you really want to compare performance please feel free to submit something to the SPC website - love to compare an EVA 8000 to a FAS 3170 - will be a good laugh

[#41]

Jim

you seemed to of missed reply to items in my post. Is this because you don't have an answer or you don't want to answer?  It is obvious you are ignoring the questions that reflect poorly on HP so I thought this may be the case.

Application Consistent Snapshots - HP EVA sell their snaps shot capabilities but are the snapshots application consistent (VMware, Exchange, SQL, Oracle for example)? If they are not application consistent would this be "LODE" as you put it? Giving the customers the impression that they can recover from deleted or corrupted data quickly when really they may not be able to...

HP and unified storage, any comments? 

VMware integration. As already stated the EVA plugin came out a few weeks ago....a VERY long time after EMC and NetApp. Does this mean you don't think virtualisation is important? If you do think virtualisation is important why did it take so long for the plugin? Is this a reflection on the storage R&D at HP?

Finally If you truly think that NetApp performance is poor compared to HP please feel free to submit something to SPC. I would have a pretty good laugh at seeing the results of a EVA vs the FAS3170 one already there.

One thing always stands out to me about HP. When buying a blade enclosure and servers (now HP do make good servers) about 18 months ago (c7000 with some BL460 and BL680) I asked the question why the only switches available are Cisco when HP has the procurve product line. I then had the HP rep explain that the server group collaborate better with Cisco than they do internally. Pretty much the left hand not knowing what the right hand is doing...but on a much bigger scale as there are so many different divisions.

[#42]

Your logic on this baffles me. If you are failing over to the redundant site, something went wrong at the primary. You then insist that something must go wrong at the second site and repeatedly call it a SPOF. You do know that SPOF stands for SINGLE point of failure right? Maybe I can be a bit clearer on this for you: (Primary site failure) + (Secondary site failure) = 2 failures.

[#43]

It's Saturday afternoon and I'm "fortunate" to be at a service department getting my van fixed and have a bit of time to kill - so what better way than give a couple of quick replies (I'll leave it to Jim if he wants to go any deeper).

Hey Chuck - thanks for the tip on secure multitenancy.  I'm sure Jim will have a look.

Ben - you haven't addressed any of Jim's points about MetroCluster but are running a classic smoke screen to divert the attention away from the original question.  I don't think you work for NetApp so you can't defend why their marketing would claim it's "designed for zero downtime" when we've clearly showed that it isn't.  We talk about our products all the time on the blog - for now, we'll just stick to the topic at hand instead of your smoke screen.

rtpnfd - To a great extent, we agree with you. A MetroCluster solution has no single point of failure.  I think maybe you're not reading the main point or your such a NetApp enthusiasts that you don't want to hear it.  When the primary site has failed and the secondary site takes over (or shall we say becomes the primary site), there are numerous SPOF that put the customer's environment at risk.  If you understand all that and still think it's a "designed for zero downtime solution", we disagree and there's really not much more to clarify. 

The bottom line is this: if you're going to market a product as "designed for zero downtime" it should deliver on that core value proposition.  We don't see how MetroCluster can once a failover has occurred.  If NetApp would change the marketing language they used for MetroCluster, we'd probably all agree. 

[#44]

Thanks, everyone, for your comments.

For anyone wondering why I don’t answer questions in this discussion about the ground speed of North African swallows, it’s because I don’t have attention deficit syndrome, and this is an article on NetApp MetroCluster. For all those wondering why I don’t talk about HP products very much, it’s because, well, again, this is an article on MetroCluster. And for all those baffled by my logic, let me say that I never in my wildest dreams expected anyone who sells NetApp to ever say, “Oh, never thought of that. I guess you’re right. We’ll go change how we market MetroCluster now.”

And I hope no one following this MetroCluster discussion is mistaken enough to think that the NetApp enthusiasts’ comments are anything more than a combat jet releasing flares as the heat-seeking missile hones in.

Moving on, it’s important that we don’t lose sight in this discussion that the NetApp enthusiasts and I agree on almost everything in regards MetroCluster– except my conclusion.

Things NetApp and I agree on:

1. A MetroCluster is made up of a single NetApp filer on one site and a single NetApp filer on another site up to 100km away.
2. A single NetApp filer has multiple SPOFs that can only be eliminated through clustering.
3. MetroCluster has two single-nodes connected over a Network. Note: This means that if the 2 nodes are in different buildings, then MetroCluster has the unique quality of allowing the HA of a system inside the data center to be jeopardized by an event outside the data center. In other words, if you have a network problem outside the data center, your system now has no HA on either site.  
4. A single MetroCluster node, sitting by itself in the primary site, is more likely to fail than a standard NetApp dual-node cluster. Therefore, the primary site is less robust than it would be if NetApp had simply sold the customer a 2-node cluster in the first place – and, logically, though the data is protected with MetroCluster against a site disaster, the likelihood of the primary site system having to invoke failover due to some other cause, is greatly increased.
5. MetroCluster (though there are exceptions listed in the MetroCluster best practices guide) can do a one-button failover.
6. MetroCluster failback is really complex and has the risk of data corruption. Note: Some readers may think that MetroCluster failback is the same as it is for a standard NetApp 2-node cluster - it's not; it's much more complex.
7. NetApp markets MetroCluster as suitable for mission-critical environments that require zero downtime.

My position is that if you can accept points 1-6 above as being accurate, then you cannot by any reasonable, logical, or fair standard claim that MetroCluster is a mission-critical  ZDT product. My further claim is that in sales situations NetApp sales reps routinely market MetroCluster as fit competition to solutions that are orders of magnitude more robust. Moreover, I claim that NetApp does not inform customers in its sales literature of the inherent risks of MetroCluster.  

I think what is equally clear is that NetApp AGGRESSIVELY markets the product as fit for mission critical zero downtime environments. As proof, just look at how aggressively the NetApp enthusiasts have argued against my position here. Do you think, for one second, their sales force is not arguing just as aggressively in front of customers every day? Do you think that when HP sells two EVA’s connected by CA in a bi-directional solution and it becomes a price battle, that NetApp isn’t coming in and vigorously promoting MetroCluster as offering equivalent functionality? To deny otherwise, is to claim that NetApp sales reps voluntarily take it upon themselves to market MetroCluster in contradiction to their own marketing literature. We have clearly seen in this discussion that that is not the case.

Best regards,

Jim

[#45]

Hi Jim,

I think your primary misunderstanding is, that MetroCluster is *one* Active/Active system, though geographically distributed, with *no* SPOF when configured correctly.

As with any highly available system, once it's degraded it might have a SPOF. Let's consider as an example a system with 2 PSUs, like most NetApp controllers. Would you criticize it, because once one PSU failed the system has a SPOF?

In NetApp documentation 'the controller' is always thought of as *one* unit that might fail (even though there's some redundancies built into them, like fans, PSUs, ...) The IBM docs just list some of the SFOFs within a *single* controller, that e.g. someone that doesn't want to deploy a cluster should be aware of.

Looking at the MetroCluster Guide right now, I think it's very forthcoming. Please (re-)read sections 1.1 (Scope) and 2.1 & 2.2 (Features & Operation). E.g. it says: 'To many, stretch MetroCluster is simply an
active-active configuration with longer cables and mirroring.' It also explicitly mentions, which feature protects you from what (-> 2.2).

Many of my students have MetroClusters deployed and all were/are very happy with it. Like the guys from Sweden who had a fire in their DC on Saturday, yet were able to attend my course on Monday in a relaxed mood, because MC had failed over completely transparent to the users (mobile phone carrier...). Also it was one of the few systems that after the fire was put out came up again with no troubles, resynchronized automatically and was back to normal operating mode in no time.

MC is very popular here in Europe, probably because of the (relatively) smaller distances between branches/DCs compared to the states. At the end of the day, though, I guess customers prefer guaranteed 5-9s availability to marketing speak.

Sebastian

(full disclosure: freelance technical instructor, specialized on storage, NCI)

[#46]

Hi Sebastian,

Thanks for your comments. I especially appreciate your comments because you have neatly summarized the key points of disagreement between myself and my colleagues at NetApp in regards MetroCluster.

Again, I would like to reiterate: My issue is not with MetroCluster’s functionality, component reliability, distance capability, ease of failover, or fitness to be sold as a product in some environments. My criticism is over the NetApp marketing and MetroCluster’s fitness to be sold as a high availability solution for zero-downtime, mission critical environments.

A few points:

1.  When selling an HA solution for ZDT, mission-critical environments, it is my opinion that all HA exposures, including SPOFs, should be clearly spelled out. I consider this a basic, non-negotiable requirement for any product marketed as suitable for ZDT, mission-critical environments. The reality is, in the entire industry, there is no other ZDT mission-critical product that has to reveal its SPOFs except NetApp. Why? Because no other product in this category that I'm aware of has any SPOFs. In addition, MC has the rare attribute of being the only product I’m aware of that allows an event outside the data center to affect the HA of a system within the datacenter, i.e., if the network goes down then both your primary and secondary sites now have zero HA. My contention is that this makes MC not fit for its intended purpose.

2. I’m not splitting hairs here. You’ll notice that I’m not criticizing IBM for also selling MetroCluster. The reason is that IBM has at least made a reasonable effort to disclose enough of the problems that the customer has at least a fighting chance of knowing what they are getting into.

3. I reread the sections you pointed me to in the MetroCluster guide. I take it you’re looking at the older 2008 guide since the most recent verson doesn't have a section 2. 2. Section 1 states that the purpose of the guide is to discuss “implementation planning, installation, configuration, and operation” in regards a MetroCluster solution. I consider a clear understanding of the SPOFs and other HA risks to be baseline knowledge in such a discussion. In fact, I'd consider it to be at the very top of the discussion list in any ZDT mission-critical solution. And further, I think it's perfectly reasonable to ask why a 55-page technical white paper can't add 5 lines to talk about the obvious HA exposures in a solution that markets itself right up there with the most redundant HA solutions in the industry.

4. MetroCluster is a solution with multiple SPOFs on the primary site that fails over to a system on a secondary site that also has multiple SPOFs. In other words, your primary site is now more likely to fail than if it had a single NetApp 2-node cluster, and then when it does failover, it will be to a secondary site that also has multiple SPOFs. So, in the event of a site disaster your mission-critical data could be sitting for hours or days on a system with multiple SPOFs. There is no other ZDT mission critical system I’m aware of that has these risks. The redundancy at the failover site is an important consideration in any mission-critical ZDT solution.

5. In your comments, you emphasized the point that MetroCluster is *one* active/active system. In terms of its real-world deployment, I don’t believe that is exactly accurate though on some level I’m sure it could be argued.

But here is why I believe it should be thought of as one system on a primary site and one system on a secondary. Each system has its own operating system, root drives, and owned disks. It is a shared-nothing cluster. The disks are 100% owned by the filer they are attached to. Plus, each side requires the purchase of its own software licenses. You don’t buy one license and have it work for the entire solution. You must buy two – one for each side. But the important point is that customers buy MC expecting it to solve the same problems as any other geographically dispersed 2-system HA solution.

6. For anyone who considers that MetroCluster fits the bill as a HA mission-critical ZDT solution, I would ask “How then would you describe an HA system where a fully-redundant system fails over to another fully redundant system? Would you call it double mission critical of double zero downtime? The fact is NetApp markets MetroCluster with the most powerful language you can use for any mission critical solution. You really can’t say much more about a product than “mission-critical ZDT.” Is that fair to customers for a product with MetroCluster’s issues to be marketed with the same language as other far more redundant solutions. I believe not.

If you are interested, you can read the NetApp marketing in regards this product: http://www.netapp.com/us/products/protection-software/metrocluster.html   If after reading this, you decide that MetroCluster is accurately marketed, then we will just have to agree to disagree.

Best regards, Jim

[#47]

Hi Sebastian,

I almost forgot. I wanted to comment on your points in your next to last paragraph where you describe your student's relaxed and effortless experience with MetroCluster  failback. 

In the latest MetroCluster guide, starting on page 36, are the steps for a failback . They cover about 2 pages. They are complex. Here is an excerpt: Note:  I believe it is word for word the same as in the older MC white paper you refer to.  

"Once the problem at the failed site is resolved, the administrator must follow certain procedures, including restricting booting of the previously failed node. If access is not restricted, a split brain scenario might occur. This is the result of the controller at the failed site coming back up and not knowing that there is a takeover situation. It begins servicing data requests while the remote site also continues to serve requests. 

The result is the possibility of data corruption. You can restrict access to the previously failed site controller in the following ways:

 Turn off power to the previously failed node (disk shelves should be left on).

 Disconnect the cluster interconnect and Fibre Channel adapter cables of the node at the surviving site.

 Use network management procedures to enable the storage systems at the disaster site to be isolated from the external public network.

 Use any application-specified method that either prevents the application from restarting at the disaster site or prevents the application clients from accessing the application servers at the disaster site. Methods can include turning off the application server, removing an application server from the network, or any other method that prevents the application server from running applications."

Seems complicated and risky to me. And that's just a few of the steps.

Best regards,

Jim

[#48]

the Metro cluster allow you to provide active active data center cross location. In most enterprise environment, the DR require 50 mils away, which will be within the 100KM maximu range for metro cluster. Metro cluster simplified the DR management from IT perspective, with Cisco OTV technology or L2 Long range fiber, 2 physical DC can be logically into 1. Today you do primary and secondary replication at remote site is to achieve data protection while DR happen or site failure. Metro CLuster allow you to achieve both scenerio into 1. With virtualization in place, they had even talk about Vmotion and HA across WAN. You need to look for the bigger picture

[#49]

Be interesting to understand what happened with this customers implementation!

http://www.theregister.co.uk/2010/09/30/navitaire_cock_up/

[#50]

http://www.theregister.co.uk/2010/09/29/netapp_virgin_blue/

another update in details

[#51]

I am a NetApp enthusiast/CUSTOMER/whatever you want to call me.  I am responsible for designing and maintaining a stretched datacentre solution in London, with diversely DWDM and 30km distance between sites.

We went through a very lengthy research and design phase in 2008, and unfortunately came up with only one solution that would fit our design criteria at the time.  This was a NetApp Metrocluster.  This is not 100% what we wanted to find.  I would rather have a pick of many different products and have the vendors compete against each other.  I gave HP, EMC, IBM (bar NetApp sales), HDS, and many others my list of criteria, and no one else could fit my remit.  This has now changed somewhat, but HP and HDS are still not able to fit my remit.

As a savvy customer, we never listen much to marketing hype.  What I am interested in is cause and effect - If I stick this in here, what comes out there.  We have a full time test lab which replicates what we have in production to most extents to prove this.  The NetApp solution has its foibles -  I would love for it to have dual controllers per site, but it doesnt.  It still works for most of what we want it to do however.  We use VMWare and Microsoft clusters, and both of these systems are able to seamlessly migrate between our two sites without any fuss.  This has proved extremely advantageous on occasions too numerous to mention, mostly when performing maintenance at a site.

We were previously Datacore SAN Symphony customers.  This provided the same Metro-clustering ability but with none of the wide industry support and comatibility matrices of larger vendors like HP, IBM, Sun, NetApp, etc.  We had previously purchased two "lumps" of storage for this purpose, which for no reason other than we got it for a good prices are EMC CX3-80s.  We use NetApp V-series Metroclusters in front of these.

The list of requirements we had:

1.  To be able to seamlessly fail from one site to another if a component, link or power fails.  

2.  To provide block and file based storage for VMware, user files, SAN-attached servers, etc.

3.  Zero capacity and performance affecting snapshots (redirect on write rather than copy on write)

4.  Deduplication (nice to have)

5.  Others but not as important as the first 3.

To answer number 1, There were a few vendors but none as well thought out as NetApp.  There are some caveatswith this but we have mitigated most of them through careful deployment of backup links, UPS units, etc.  If an EMC array fails (yes they do fail - I have bruises to prove it) no issue - automatic failover.  If a Filer fails - no issue, automatic failover.  If the power fails - no issue, DWDM, SAN Switches, filers, arrays are all protected by UPS.  As longs as none of these things fails at exactly the same time, a graceful site takeover will occur.  We have engineered the whole solution so the likelihood of the entire site failing at once is incredibly unlikely, even if there was a fire or plane hitting (you'll most likely get some kind of dying gasp the remote site can see and respond to).  We also have monitoring and automation in third and fourth sites, which have independant links to both DCs.  We havent implemented it but we have the option of building a "tiebreaker" which will automatically arbitrate and failover in the event of complete site failure whilst reducing the risk of split brain.

2.  Yes in 2008 you could build a multi-protocol solution from HP and EMC, but not without hitting issue number 1.  We needed this to be seamless, uninterrupted failover in most situations. EMC now have this with VPLEX and VNX, but didn't at the time.  HP still cant offer this

3.  NetApp aparrently own the patent on redirect around write snapshots.  The ability to keep 250 snapshots per volume without impacting performance by a great deal is one of the single most useful features for our filers.  We dont need to keep days and days of backups, we just snapvault our metrocluster's snapshots to a standalone filer in a third location, and we have long term recovery.  Of course we still use tapes to back this up as a last step just in case.

4.  Deduplication works really well especially for our VMware NFS datastores.  This has a positive impact on performance as the cache works on the deduplicated files.  

The issues we have with NetApp are mostly around performance, but this is only because the filers we bought were underspecced and we have grown beyond their capabilities.  This happens with most IT systems - you increase one bit, you move the bottleneck to the next bit!

In summary - forget all the marketing hype, claims, fud, facts, etc, etc - the Metrocluster was our only choice.  We completely understand all it's weaknesses, but we live with them as the advantages far outweigh them.  We were prepared for the cost, and yes its expensive, but I can sleep soundly at night knowing that my systems will still be running in the morning.

A good customer NEVER takes anything at face value.  Always do your own research and come to your own conclusions.  Those who don't proceed at your peril!