Eye on Blades Blog: Trends in Infrastructure

[#1]

Many of the IT shops I’ve dealt with recently have already upgraded to a 10Gb core network, or are well into the planning stages for one.  But we as an industry need to reconsider if this is the right time for such an upgrade.  This is not a trivial task, both in terms of expense, and potential disruption.  Most shops would expect to get five to seven years out of such an upgrade.  Many would be surprised to learn that their investment could be out of date in as little as six months.

The issue revolves around Converged Enhanced Ethernet (CEE).  Many shops are planning some level of Fibre Channel over Ethernet (FCoE) adoption over the next couple of years, and FCoE relies on CEE.  CEE uses three new Data Center Bridging (DCB) protocols:

·        802.1Qbb –   Priority-based Flow Control (PFC)

·        802.1Qaz –    Enhanced Transmission Selection (ETS)
                      DCB Capability Exchange Protocol (DCBX)

·        802.1Qau –   Congestion Notification (QCN)  

The first two protocols in this list are adequate for one or two hops of FCoE traffic, but if you want an end to end FCoE solution, the third protocol (QCN) is critical to manage congestion at the points of oversubscription.  Here’s a few facts about QCN:

·        QCN requires every device in the data path to support QCN

·        QCN is implemented in hardware, not firmware

·        Very few (if any) currently shipping products support QCN

This means if you have a vision of running end to end FCoE in your datacenter, you need to wait.  Why make a significant investment that’s out of date when it’s implemented?  As an interim step we can add some 10Gb capability to existing switches, and deploy rack switches to provide incremental 10Gb capability until products are ready for these new protocols.

According to Gartner analyst Joe Skorupa we should “Plan to maintain separate data center SANs and LANs for at least the next three years.”  Solutions like HP’s upcoming FlexFabric¹ can take advantage of FCoE to reduce complexity at the network edge, without requiring a major network upgrades or changes to the LAN and SAN before the standards are finalized.

¹ http://h18000.www1.hp.com/products/solutions/converged/ff-4aa0-7725enw.pdf

[#2]

Replaying the comments that were lost in the migration.

Ken

Brad Hedlund wrote re: This is not the 10Gb network you’re looking for
on Fri, May 28 2010 4:41 PM

If QCN is so "critical" for multi-hop FCoE networks, why is it that most traditional FC networks do not have QCN enabled?

Fact is, QCN is "optional" for FC, just as it will be "optional" for FCoE.

Cheers,

Brad

============================================================================

Carl S (HP) wrote re: This is not the 10Gb network you’re looking for
on Fri, May 28 2010 6:19 PM

Last time I checked, Ethernet and Fibre Channel Specs were handled by 2 different standards bodies.

IEEE - Ethernet Standards

INCITS - FC Standards

QCN is a new Ethernet DataCenter Bridging standard (802.1Qau), and therefore has no relation to current FC standards and flow control.

Because DCB Enhanced Ethernet is NEW, and carries both Ethernet and FC traffic, QCN is needed to manage the end-to-end oversubscription/congestion points.

The response above is consistent with recent changes to Cisco product specifications for the Nexus series. All references to 802.1Qau have been removed from all Nexus product sheets. Based on this I would expect that Cisco is not going to support the full DCB IEEE standard set on Nexus.

============================================================================

Brad Hedlund wrote re: This is not the 10Gb network you’re looking for
on Fri, May 28 2010 6:34 PM

QCN has no equivalent FC standard because Congestion Notification has never been a critical element in FC networks.  You have failed to make the case why it's critical for FCoE, when its never been critical in FC.

Cisco however did implement a non-standard implementation of Congestion Notification in FC called FCC (Fabric Congestion Control).  It's a feature that's so "critical" that it's disabled by default.  Furthermore, FCC is not even supported on the very popular MDS 9124e fabric switch for HP c-Class.

You still need to make the case why Congestion Notification is so "critical" for FCoE when that's never been true of end-to-end FC networks.

============================================================================

Chris L (HP) wrote re: This is not the 10Gb network you’re looking for
on Fri, May 28 2010 9:53 PM

In the IEEE spec for DCB, I never noticed the word (optional) next to any of those standards (IEEE references below). If QCN is optional, then ETS, PFC and LLDP (802.1AB) extensions are optional as well? I don’t think we need to justify the standard. The IEEE  has already done that. However – it is increasingly apparent that the “optional” tag is coming from Cisco  because Cisco is not planning on supporting that part of the DCB standard. Either you comply or don’t, and you guarantee interoperability or you don’t. Standards compliance is not “optional”.  Current Nexus product is not in compliance with the entire DCB standard, only part of it.

With respect to FC, it is not new. It is a well-established standard set. It is irrelevant that Cisco does not enable its own non-standard congestion management protocol (FCC) in a 9124e switch. What does that have to do with an IEEE standards discussion? Non-standard “standards” speak for themselves – they are NOT IEEE or INCITS standards. However FCoE and DCB are brand new industry standards - so new in fact that they are not yet complete enough to allow end-to-end FCoE. It is the first time that FC packets are being encapsulated in Ethernet packets, so I would most definitely expect that standards would be followed.

Since Ethernet is a lossy infrastructure and FC is a lossless infrastructure, you need to make sure there is a mechanism in place to help manage PFC queues end-to-end, to provide that lossless capability.  PFC is only part of the equation.  WRT Fibre Channel , Buffer Credits are the only standard mechanism required to provide a lossless switching environment.  However, at times of severe congestion, even FC has its challenges.  Ethernet adds more traffic complexity upstream than what FC does.  Backward Congestion Notification (BCN) was originally proposed, but was found to be too inefficient and slow since it is a software-based approach.  QCN is an ASIC-based algorithm/protocol, and no switch vendor supports today as it requires new silicon.

============================================================================


Brad Hedlund wrote re: This is not the 10Gb network you’re looking for
on Fri, May 28 2010 11:29 PM

Its not up to the IEEE to decide which standards are required (or optional) to successfully implement an application on the network, such as FCoE.  That's up to the vendor selling the solution to decide. If your best argument for QCN being "critical" is simply because its been defined as a standard, you'll need to make a more convincing case.  Especially if you are telling your customers to wait on reaping the TCO benefits FCoE brings to the server access layer today.  Which, by the way, can be accomplished in 1 hop.

You're right, buffer-to-buffer credits is the only mechanism required in FC to provide end-to-end lossless behavior, and 802.1Qbb (PFC) is the equivalent link-by-link flow control mechanism used by FCoE.  Yes, congestion happens even in FC, and it's been a very successful protocol without an equivalent implementation of QCN.  Hence, one can make a very good case that if it wasn't required in FC, it might be only be optional at best in FCoE.

ETS provides the link-by-link guaranteed bandwidth for FCoE, leaving standard Ethernet traffic free operate in a lossy fashion with TCP based flow control.  The "extra complexity upstream" as you describe is orthogonal to FCoE which running on its own virtual lane provided by PFC and ETS.

The IEEE DCB standards have a broader purpose than just FCoE.  QCN may someday have a valid "critical" use case for other apps or workloads, but thats simply not the case for FCoE.

============================================================================


Carl S (HP) wrote re: This is not the 10Gb network you’re looking for
on Fri, May 28 2010 11:34 PM

Just for reference - here are the DCB standards straight from the IEEE website:

IEEE Data Center Bridging Task Force Active projects

- P802.1Qau: IEEE Standard for Local and Metropolitan Area Networks: Virtual Bridged Local Area Networks - Amendment 10: Congestion Notification

- P802.1Qaz: IEEE Standard for Local and Metropolitan Area Networks: Virtual Bridged Local Area Networks - Amendment: Enhanced Transmission Selection

- P802.1Qbb: IEEE Standard for Local and Metropolitan Area Networks: Virtual Bridged Local Area Networks - Amendment: Priority-based Flow Control

In addition there will likely be an expansion of 801.1AB (LLDP) so that devices can negotiate/convey DCB settings properly throughout the network. However 802.1AB is not called out as a separate line item of the DCB standards on the IEEE site.

Again, considering that all of the Nexus product specifications ominously omit 802.1Qau (QCN), I would have to assume that Nexus products only support 2 out of the 3 DCB standards.

This is taken directly from the Nexus 5000 Data Sheet:

www.cisco.com/.../data_sheet_c78-461802.html

Cisco NX-OS Software Features and Benefits

IEEE Data Center Bridging

IEEE 802.1Qbb PFC (per-priority pause frame support)

IEEE 802.1AB DCBX Protocol

IEEE 802.1Qaz Enhanced Transmission Selection

IEEE 802.1Qau is missing from this list. Can Nexus support 802.1Qau, one of the core DCB standards? I think that if it would/could, the standard would be listed in the product spec sheet.

============================================================================


Ken Henault wrote re: This is not the 10Gb network you’re looking for
on Sat, May 29 2010 3:23 AM

Brad, in your view of FCoE what takes the role of end to end buffer credits?  This is the technology that does end to end congestion management in Fibre Channel.  Totally different technology than QCN, but the same end result.

============================================================================


Dave Alexander wrote re: This is not the 10Gb network you’re looking for
on Sat, May 29 2010 4:48 AM

Ken -

Given that end-to-end buffer credits are used with Class 2 traffic, while most FC traffic is Class 3, it's really not worth considering.   If we're going to look at FCoE as a replacement for traditional FC networks, the vast majority of that traffic will be Class 3.

QCN would provide similar behavior for those limited environments that needed it.   The current PFC implementations, operating on a hop-by-hop basis, implement the same type of flow control used in B2B, Class 2 FC networks.

- Dave

============================================================================


Brad Hedlund wrote re: This is not the 10Gb network you’re looking for
on Sat, May 29 2010 7:24 AM

Carl,

The Cisco Nexus 5000 not having IEEE 802.1Qau is not a secret, and it's not a shocker.  IEEE DCB has a larger role than just FCoE. Not every standard should be assumed to be implicity "critical" or even required for FCoE.  There is no equivalent of 802.1Qau in traditional FC networks today so its not necessary for FCoE either.

Ken,

Happy to shed to some light on this. In traditional FC networks end-to-end buffer credits are managed soley by the Initiator and Target.  The FC swtiches have no awareness or role to play in end-to-end flow control.  The same holds true for FCoE.  An FCoE Initiator can use the exact same end-to-end credits or SCSI flow control mechanisms with a Target and vice versa.  The Initiator and Target have no awareness of FCoE and still view the link as plain FC using all the same traffic management mechanisms.

Cheers,

Brad

[#3]

============================================================================

Blue Thunder Somogyi wrote re: This is not the 10Gb network you’re looking for
on Thu, Jun 3 2010 3:47 PM

In my initial reading of the standards, I had made the assumption that QCN was to manage the interaction of multiple layers of priority traffic (ie multiple priorities).  Whether that case exists today or not is I guess questionable.

Obviously you can map multiple VSANs to a single PFC class, which will treat the entire aggregate FC flow as a single priority layer, thus QCN is not needed.  But what if you have multiple disparate SAN environments that are going to ride over the same converged network (perhaps a multi-tenancy situation)?  These would need to be configured into separate parallel lossless PFC priorities (not sure if that is possible with shipping product, but I imagine it will be a supported feature at some point).  In this case QCN is needed to deal with communicating congestion events to the various PFC classes that PFC cannot deal with alone.

Another case would be if in the future, another form of lossless traffic is supported using DCB (infiniband? perhaps far fetched).  Again, QCN is needed to manage the interplay between the two disparate lossless traffic classes.

Obviously the IEEE is attempting to create a logically consistent standard that will scale beyond just the use cases that exist today (if they weren't, they wouldn't be very good at what they do).  However, this does not mean that QCN is something that is required for the seminal case of FCoE.  Indeed, if a solution can be deployed that functions without QCN, the fact that the QCN standard exists does not mean that it is 'incomplete' from a standards perspective (just a feature selection decision).

============================================================================


Ken Henault wrote re: This is not the 10Gb network you’re looking for
on Thu, Jun 3 2010 4:09 PM

See http://bladeguy.blogspot.com for my earlier reply...

Blue, The purpose of QCN is to push congestion to the edge of the network, and keep it from propagating through the core.  A typical Ethernet network has many points of over subscription, and each one of them can become a point of congestion. If we rely on PFC and ETS to manage the congestion, it can propagate to every link on the data path.  The role of congestion notification is to have the sender slow down until the congestion is resolved.

We don't see these issues on today's Fibre Channel SANs because they tend to be very flat, in most cases 2 to 3 hops.  Ethernet networks tend to be hierarchical, 5 or more hops is not uncommon.

============================================================================

Frank D'Agostino wrote re: This is not the 10Gb network you’re looking for
on Sun, Jun 6 2010 10:39 PM

These blogs are always interesting to me.  The real issue here why HP engineers are blogging here and putting out interesting, possibly silly, videos, is you do not have a product.  All except this one:

h18006.www1.hp.com/.../index.html

Here is the quote from the HP website - "This is pushing demand for 10 Gigabit Ethernet and consolidated I/O; applications for which the HP Nexus 5000 Converged Network Switches are the perfect match."

In addition, you have been spending your time on the side doing tech road shows validating what you say is coming, what your website says is already here, and in many cases - incorrectly stating the technology.

B2B has nothing to do with end to end traffic and the post above correctly indicates this around Class 3 FC traffic.  You guys haven't forgotten your SNIA exams - have you.  Here is a good book in case you need a refresher - www.ciscopress.com/.../article.asp

One last thing.  I imagine after the 21 - 24 of June, something magical will happen and FCoE will suddenly be a really good thing.

============================================================================

Ken Henault wrote re: This is not the 10Gb network you’re looking for
on Mon, Jun 7 2010 4:51 AM

Frank, Nowhere in the blog does it say FCoE is a bad thing.  In fact the whole point of the blog is to suggest that people might want to wait until all the DCB protocols are supported before making a significant investment in 10Gb switching at the core.  This will ensure the best possible support of FCoE.

In addition to the product you mentioned we also have these switches there are non-Cisco switches and CNAs.

As the PDF linked from the original post states, our road map does include DCB and FCoE.  While some vendors feel compelled to offer products before the technology is ready to differentiate themselves in the market, there has been no mass adoption.  I believe IT shops are waiting for proven multi-vendor solutions.  As the DCB standards near completion, and 2nd and 3rd generation CNAs become available more shops will be willing to evaluate FCoE solutions.  Even still it will be at least a couple of years before true multi-vendor end to end FCoE solutions are practical.

============================================================================

David Lawrence, FC Director Product Manager wrote re: This is not the 10Gb network you’re looking for
on Thu, Jun 10 2010 9:17 PM

Frank,

You're missing the entire main point of the first entry:  The fact there is still debate over standards for FCoE and CEE and few products on the market that adhere to 100% of them really point to the need to reduce the risk of implementation for early adopters of the converged networking.  FlexFabric provides many of  the same benefits of a converged network while reducing the risk inherent in un-finalized standards. Another way to reduce the risk is to offer the market choice.  Besides the Nexus 5000 switch, HP offers the B-series portfolio of FCoE switches and director blades as described in following article:  :www.communities.hp.com/.../where-to-get-started-with-fcoe.aspx

============================================================================

EtherealMind wrote re: This is not the 10Gb network you’re looking for
on Thu, Jun 10 2010 10:16 PM

FCoE, too complicated for mortals when experts argue over semantics and interpretations.

Confident that iSCSI and NFS will win out over FC in the near term. Lets face it, FC is so complicated and needs so much money to make it work that no one is going to implement it.

Oh, BTW, the reason no one uses QCN in FC is that the FC switches are so over specification and over sold that they can't drop a frame. Why bother with features to control flow when you don't need it.

============================================================================

Ken Henault wrote re: This is not the 10Gb network you’re looking for
on Thu, Jun 10 2010 11:21 PM

Greg, Thanks for stopping by.  I do enjoy your blog.

I have to agree with you, that none of the storage protocols we have today (or in the near future) are ideal, but they're all we have.  I'm sure most shops will get by with two or three, because they don't have another choice.  FCoE will gain traction because there's too much investment in FC to throw it a way, and FCoE will reduce some costs.  iSCSI is getting another look in many circles, and deservedly so.  It does have some good use cases, and is very cost effective.  Same goes for NFS.

I will disagree with you on congestion management in a traditional fibre channel SAN.  In most implementations it's managed by deploying very flat fabrics.  Most of the shops I'm familiar with try to keep it to two hops, a few will push it to three.  If you were to just add FCoE to most enterprise data center networks, you'd have 3 to 5 hops as a starting point.  This would make end to end congestion management a requirement.

============================================================================

Dave Alexander wrote re: This is not the 10Gb network you’re looking for
on Thu, Jun 10 2010 11:23 PM

EtherealMind -

The death of fibre channel has been greatly exaggerated.   Various parties have been predicting the replacement of FC with iSCSI (not so much NFS) for years and years, but it hasn't happened.   Why not?

NFS is a file-based protocol, and as such isn't suitable for a lot of applications.   It also has the drawback (from a storage perspective) of riding on top of TCP (usually) and IP, which simply weren't designed for high speed, bulk data transfers.

iSCSI does have the advantage of being block-based, and carries the exact same SCSI protocol as FC.   Unfortunately, it too is saddled with TCP and IP.   As a low-cost, routable storage protocol, it absolutely has its place in the data center.   Just not high volume, high performance workloads.   Now, to be fair - many existing FC deployments don't meet the high volume, high performance threshold and would be adequately served with an IP-based solution.   But that doesn't make IP-based storage "better" - it's solving a different problem in a different way.

And for the record, the reason that FC switches don't drop frames isn't because they're over-sold or over-specified.   The protocol itself and the flow control mechanisms in use (buffer to buffer credits, specifically) provide a network where only data that is capable of being processed by the receiver is ever sent - sender and receiver agree to this in advance.  

FC as a protocol isn't going anywhere.   There's simply nothing else currently available that handles storage as well as it does.   FC as a transport's days are numbered, given the now-standardized FCoE and associated performance.   There very well may remain dedicated storage networks far into the future (recommended for iSCSI as well), but I see physical-FC phasing out in the next 7-10 years.   Won't be completely gone, but I think most mainstream implementations will be FCoE by then.

Just my USD$0.02.

- Dave

[#4]

Brad,

If congestion notification isn't useful why was it originally included in the features of the Nexus 5000?

Then when it became apparent that IEEE was moving away from BCN to QCN, congestion notification was dropped.

It seems as though Nexus was too early with it's hardware design, at didn't have the hardware to support the standard as it got closer to it's final form. 

So I believe your agrument boils down to  If Cisco hardware can't support it, it must not be important.

[#5]

Ken,

ETS + PFC provides the exact same flow control available to Fibre Channel networks today, with the reduced protocol overhead of 64b/66b encoding vs 8b/10b.  That being said where do you see a need for the optional QCN standard?  If you insist it's a weakness of the Cisco devices that they don't support it, do you have any actual data to show the potential benefits of using QCN or are you basing QCN requirement on the fact that a standards body made a standard?

If you feel that every standard produced by a major standards organization must be followed take a look at the pro-curve line and virtual connect product lines and the standards they choose not to support.

Some standards are required for proper FCoE performance, some are optional.  QCN is optional because it is an additional congestion management tool that is beyond what current Fibre Channel networks provided.

Joe

[#6]

Joe,

That's the whole point.  FCoE is not likely to be deployed in the same manner as the existing Fibre Channel infrastructure.  Most IT shops have a carefully designed flat SAN design today, and workloads are very static.   If you're planning on staying with this old school design, PFC and ETS are probably adequate.

But whether you call it Converged Infrastructure, or DataCenter 3.0,workloads are going to be come more dynamic.  Moving around the datacenter to provide availability, workload optimization or to allow for easier maintenance.  In this environment you no longer have the luxury of being able to carefully optimize the SAN connectivity.  You need to be able to trust the infrastructure to do the right thing.  Without end to end congestion management, that won't be possible.

Ken

[#7]

Ken,

It comes back to quantifying your percieved value of QCN.  If you insist that QCN is required post a technical article quantifying it's benefits and back it up with real data.  If you can't make a strong argument for why it's required than all your doing is spreading FUD, that's not the job of an engineer.  If HP has chosen to build hardware that's suports QCN there must have been a technical decision making process that got there.  Post those arguments, and back them up.

Joe

[#8]

Ken,

I could take your arguments more seriously if HP's own FCoE product, HP FlexFabric, included QCN capabiliies.  But it doesn't, isn't that right?

Accordning to your logic, HP must not believe QCN is importnat either, becasue your hardware doesn't support it.

Isn't that right?

Cheers,

Brad

[#9]

Looking for an update to Brad Hedlund's question...

Does HP's FlexFabric device support QCN?

Thanks in advance,

-sean

[#10]

Brad,

HP's not out there touting FCoE because it's too early in the development of the DCB standards and supporting products.  Congestion management is critical to large scale, end-to-end deployments of DCB/FCoE across the datacenter.  Even Cisco thinks it's important, that's why they’re the company that sponsored congestion notification as part of the DCB standards, and why they baked Backward Congestion Notification (BCN) into the Nexus product line.  Then a funny thing happened on the way to ratification within IEEE.  BCN evolved into QCN and Cisco realized they jumped the gun.   QCN is done I hardware, not firmware , so Cisco had built the wrong product.  Cisco stripped all references of BCN from the marketing literature(see above), and declared congestion management was unimportant.

But the technical documents still acknowledge the value of QCN.  Take a look at Set up FCoE Connectivity for a Cisco UCS Blade  Document ID: 110434 dated April 14 2010.

The Ethernet links on the system support these Ethernet enhancements to ensure lossless transport for the FCoE traffic:

Priority Flow Control (PFC) IEEE 802.1Qbb is an extension of the PAUSE (802.3x) mechanism. PFC creates eight virtual links in each physical link and allows any of these links to be paused individually without affecting the flow of traffic in the other links.
 
 Enhanced Transmission Selection (ETS) IEEE 802.1Qaz is a scheduling mechanism in hardware that allows a two−level Deficit Weighted Round Robin (DWRR) with strict priority support. This allows control not only of bandwidth, but also of latency.

Congestion Notification IEEE 802.1.Qau is a form of traffic management that shapes traffic as close to the edge as possible to reduce/eliminate congestion. Provides end−to−end congestion management for protocols that do not already have built−in congestion control mechanisms.
·
Data Center Bridge eXchange (DCBX) is a discovery and capability exchange protocol to verify that both ends are configured properly to support the DCE traffic. It can provide basic configuration if one of the two sides is not configured properly.

For an example of what a difference QCN can make, look at these graphs from a Stanford Research Paper.  Notice how the Innocent Flow(in red) is affected by pause based flow control just as much as the Victim of congestion.  Note what happens to the queues within the switches.  Adding QCN corrects the problem, alleviates the excessive queue length, and doesn't impede the Innocent Flow. .

                                              WITHOUT QCN                                                                         WITH QCN

My opinion is that data center networking is going to change more in the next 15 months than it's changed in the last 15 years.  Anyone making a major upgrade to any vendors 10Gb infrastructure should make sure that vendor has a roadmap to support the protocols of tomorrow before making a huge investment today.

Late addition, here's another good blog on QCN  http://jdsu-fcoe-blog.com/?s=qcn&x=25&y=16

As always, the opinions expressed in these blogs are mine, and do not necessarily reflect the views of HP.

Follow me on Twitter:  htth://twitter.com/bladeguy

[#11]

So does FlexFabric support QCN?

[#12]

FlexFabric does not support QCN.  That's why HP has positioned FlexFabric as an edge solution.  Using FlexFabric in this manner can save as much as US$1000 per server over a comparable solution with discrete Ethernet and Fibre Channel connectivity.

[#13]

But why not let the customer decide of QCN is make-or-break for FCoE and support FCoE on the FlexFabric uplinks? If customers don't think QCN is a requirement they can 'flip on' the FCoE uplink ability. If they disagree with Cisco and think QCN is required then they chose to rip out the FlexModules and replace them when HP does offer QCN-enabled products.

As it stand now HP is forcing customers to replace nearly $36K per-chass of switches if they want to use end-to-end FCoE in the future. That's not investment protection or choice, its a forklift upgrade and money down the drain.

[#14]

Robert88

Thanks for stopping by.  QCN isn't the only reason the FlexFabric module doesn't support FCoE on the uplinks.  There are a number of technical hurdles yet to that desired end goal.  To my knowledge there are no mainstream products out there that support FCoE on the uplinksyet, not even Cisco's blade offering.

Scott Lowe has a blog on what some of the issues are.  http://blog.scottlowe.org/2009/08/11/why-no-multi-hop-fcoe/

Ken

[#15]

In case you haven’t been following the related discussions on Twitter, you really should.  I learned a lot about Cisco’s views on DCB/FCoE in Twitter.  The view I have, and wrote this article to, was that DCB/FCoE is best implemented as an end to end protocol.  That we could have a Fibre Channel Forwarder (FCF) to connect the FCoE network to the legacy SAN, then use DCB/FCoE the rest of the way to the server.  This design solves a number of issues with existing FC SANs.

The limited network diameter is one issue solved by end-to-end DCB/FCoE.  Most FC SANs are limited to two, maybe three hops.  One of the key reasons is that FC has no end-to-end congestion management.  FC relies on buffer credits to manage congestion on each link.  By using DCB and QCN, we can create a SAN with a much larger diameter because QCN manages eend-to-end congestion.  Then, instead of carefully designing and controlling the SAN, we can rely on the DCB network to manage the traffic.  See my blog You don’t want to micromanage the cloud for some related thoughts.

Vendor Lock-in is another issue solved by end-to-end DCB/FCoE.  Joe has an interesting blog on lock-in.  The example he used for vendor lock-in was a half full blade chassis.  I think that’s a poor example because I’ve installed c-Class enclosures in data centers that had half-full enclosures from another vendor.  If you feel your current vendor isn’t meeting the need, you don’t have to fill the enclosure.  Fibre Channel SANs are a much better example.  Once you have a SAN from one vendor, you are very unlikely to add a second vendor.  If you do, you usually have to dumb the whole SAN down to Compatibility Mode, and lose all the cool management and performance features that are vendor specific.

The vision that I’ve heard from Cisco is that they want to use DCB to connect FCFs, and have an FCF at every hop.  I like to think of this as “hop-to-hop” FCoE.  This means that every switch in the path has both Ethernet and Fiber Chanel components.  At each hop, the Ethernet packet will be stripped off, the FC switch will do it’s magic, then a new Ethernet pack with a new MAC address will be created, and sent along it’s way.  The advantage of this method is that all the tools, processes and controls built into today’s FC SANs could be made to work.  The downside is that we propagate the vendor lock-in that we’re stuck with in Fibre Channel to FCoE, and we add the expense to the FC logic to each switch.

Another key issue with hop-to-hop FCoE is the SAN diameter.  As discussed above, Fiber Chanel has no end-to-end congestion management.  That’s why SANs are typically limited to 2-3 hops, what’s referred to as core-edge and edge-core-edge designs.  If you just replace one of the FC edge switches with an FCoE switches, no problem.  But if you want to extend beyond that, a whole host of problems come up.

With an end-to-end DCB network we can continue build networks like we always have.  The five hop edge-distribution-core-distribution-edge is a fairly common data center network design.  When implementing a hop-to-hop FCF network, we add the SAN design constraints on top of the network design constraints. 

So this brings up the subject of training.  Network design is already complex enough that the average CCIE earns more than an MBA.  Now the network architect will have to be trained in SAN design, and will have to take SAN constraints into consideration to build a hop-to-hop FCF network.   If instead we move forward with a true end-to-end DCB/FCoE SAN we can simplify the design constraints, and work with existing datacenter design principles.  This design implies an infrastructure that has more self-healing and self-management capabilities.

Standards are important to our industry.  To be meaningful and valuable standards should require a large degree of interoperability yet still allow for vendor differentiation.  By this standard I view legacy fibre channel as a failure.  There is plenty of vendor differentiation, but little interoperability within the fabric. 

So we’re at a crossroads as an industry.   We can allow Fibre Channel with all it’s warts to be ported to FCoE, or we can take this opportunity to build a new infrastructure that is smarter, more flexible and open.  This will be fun to watch.

[#16]

Ken, would you please point to a public URL showing where HP supports what you have above.  HP is selling a product now and it would be great to have a validation for you customers what you suggest is critically important.

I do not want to speak for everyone on this thread, but you have previously made a comment about "Cisco's Views".  Cisco's views are published on www.cisco.com.  I would hazard a guess that the brilliant engineers from Cisco on this thread are merely stating "their views" and not "Cisco's Views".

[#17]

Ken,

I like a lot of what you had to say in you last comment here especially: 'Standards are important to our industry.  To be meaningful and valuable standards should require a large degree of interoperability yet still allow for vendor differentiation.' 

I'd also say you're pretty dead on with the failures of Fibre Channel, they just never got interoperability right because the standards were too loose and everyone was more worried about selling a product and locking competition out then making things interoperable.  I'd have to say Cisco is unique in SAN due to the built in interop modes on the full line of MDS products.  It was a great feature to seamlessly move Cisco MDS into existing Brocade and McData shops, but could just as easily be used to migrate from Cisco to a competing SAN switch if a customer chose to do so.

Anyway enough off topic commentary.  I think one of the key things your missing here is one of my favorite statements, 'Just because we can doesn't mean we should.'  So let's take a hypothetical and plug FCoE end-to-end on your five tier network example above with QCN managing congestion.  What did we gain?  Not much FCoE is still just connecting disk to servers nothing glamorous.  What we lost in this example is the big catch.  We now have big fat loss sensitive 2242 Byte FCoE frames traversing 5 tiers of aggregation, core, and edge LAN network for no reason.  Sure QCN will push congestion to the edge if we're talking DCB switches, sure ETS will guarantee bandwidth, and our good friend PFC will unsure lossless behavior but did we actually gain anything from all that complexity?  I vote no.  Take a look at my extended thoughts here: http://www.definethecloud.net/?p=468.  Either way we discuss it over a beer next week at VMworld, or we can just drink and babble about Jarhead days ;-)

Joe

[#18]

Joe,

I edited your post, but only to make the font larger.  The 7pt font was too hard on the eyes.

Frank,

Most of the people I've been talking to assumed it would be done with DCB, including QCN.  Here's a couple of relevant documents that speak to DCB/FCoF:

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c02044591/c02044591.pdf

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c02475134/c02475134.pdf

Thank you for reminding me ti be careful on assigning credit for views and opinions where it belongs, point taken.

Ken

[#19]

Ken,  the Stanford study that you put the graphs in this post are indeed something worth the enginering discussion, as they do point out that TCP flows can in certain situaitons benefit from a QCN.  My suggestion here is that posts on this thread support that proposition for DCB in certain situations.  Since FCoE is fundamentally different and TCP windowing programmatics involving the TCP state machine do not come into play, this particular study would not apply here.

The fundamental idea that I think we all (as professionals in this field) could have is really boiling down to (IMHO) allowing FCoE as an integrated part of the DC infrastructure, vs an over-the-top technology.

Did the contributors to FC-BB-5 have an opinion either way?  My reading led me to believe it could go either way, but my personal view is this would need to be integrated vs. over-the-top.  Like you say, it will be fun to see where the market takes this.

[#20]

I understand the marketing - Sean McGee asked the same from above.  Where is HP's support for this technology in the products you are selling.  This has been a stall tactic by HP due to an absence of the product.  Now that you have one - where is the support?

We are educating customers on the true architecture issues around these topics, not the hype.  Each company wants to influence customers on the purchase of their technology (duh), but I enjoyed your posting about what customers have to build FC topologies the way they do. 

As a manufacturer of that technology (not a reseller or OEM), it is clear why Cisco is leading with products and innovation and HP is forced to regurgitate those ramblings from the QLogic or Brocade OEM'd or pupose built platforms and not provide an architecture.  I guess that is also reflected in Cisco's R&D budget versus HP as well.

Let's not go down the proprietary path either as many of the people on this thread are clear about what is and is not proprietary.  The key is invests, innovates, and provides customers an architecture - not excuses for why things are not delivered.

[#21]

Hey Guys,

Great conversation. First off, at the risk of pointing out the obvious, ethernet carries more traffic than just FC. Any argument that starts with "We used to do it this way..." just doesn't hold water.  We also used to build large circuit switched networks with inefficient use of bandwidth ( Greg F.'s overprovisioned networks ) over TDM which people called the PSTN and we didn't have quality issues. Cisco has had a lot of succes helping their customers move to IP Telephony and VoIP installations that required a much more robust use of Quality of Service tools. "The old PSTN didn't need those tools and it worked just fine!" the old timers cried.  But sadly, their time had passed.

     To extend the analogy, I remember in the early days of IP Tel, Avaya failed 100% of their installations based on there own internal metrics. Why? Because they didn't bother with configuring the appropriate QoS features in the customers networks. They didn't need it before, so why would they need it now? 

My point is that just because FC didn't require QCN, that doesn't mean that FCoE can get by without it. The SAN was overprovisioned and ran one application.  If there's no congestion, you don't need congestion notification. Now, we live in a Virtualized world where people are aiming for 70% or greater utilization of the data center resources ( CPU/Memory/Bandwidth/etc..) which means that in an ideal state, the environment is going to be close to capacity ( ie congestion ) at all times. To me, this means that the HP approach of single-hop FCoE and then handing of to the SAN is based on sound engineering with a healthy does of skepticism for unproven technology. 

 In a couple of years, we will know one way or another. But until then; can I recommend that customers implement FCoE and look them straight in the eye and tell them it's going to be alright? How much data is going to get lost before we say "Wow... I guess we really DID need QCN".

On the QoS note; one of the other things I see applied all the time is QoS strategies for enterprise LANs applied to DataCenters and it just doesnt' work. Classification and Queuing (Prioritization) mechanisms make sense in a bursty LAN environment where most of the traffic is not mission criticial. The ability to apply backpressure ( QCB ) to be able to throttle offending traffic is desirable. To prioritize one type of traffic, we need to DE-prioritize another type of traffic.

I don't know of many applications running in my customers datacenters which they would elect for deprioritization. In this case, using a congestion avoidence technique, such as would be provided by QCN, makes a LOT more sense.

And until that is available, I don't see any point in moving to FCoE when my current FC SAN is working just fine. 

But that's just me.

Chris