Eye on Blades Blog: Trends in Infrastructure

[#1]

The Virtual Connect FlexFabric module for HP BladeSystem was officially announced today.  This brings two new capabilities to the Virtual Connect family, iSCSI offload and FCoE.  There are two key components; an I/O module for the BladeSystem enclosure, and a FlexFabric Converged Network Adapter (CNA) that is integrated on the G7 blade and available as a mezzanine card for G6 and later blades.  The FlexFabric adapter combines the functionality of a 10Gb NIC and a Fibre Channel HBA into a single device that also support full iSCSI offload.

These new offerings are part of HP’s FlexFabric network architecture. HP FlexFabric is the networking ‘pillar’ of HP’s Converged Infrastructure strategy.  HP’s FlexFabric describes an end-to-end data center network architecture that includes network infrastructure, management, and security.  Virtual Connect is a key network server edge component of that architecture.  According to my friends over at HP Networking “The HP FlexFabric architecture combines advanced, standards-based networking technologies with a new modular architecture that optimizes connectivity in server virtualization-focused environments.” ¹  I expect a blog from them very soon with more details on FlexFabric.

Back to the Virtual Connect FlexFabric Module.  FCoE is grabbing many headlines today, but iSCSI shouldn’t be overlooked.  iSCSI is a mature technology that offers performance required for most applications at a very cost effective price point. For a proof point on how iSCSI can scale to meet the toughest enterprise class workloads, check out our reference architecture for VDI http://h20195.www2.hp.com/v2/GetPDF.aspx/4AA1-9256ENW.pdf.  This VDI story will only get better as the architecture is updated to take advantage of the FlexFabric adapter’s iSCSI offload capabilities.  This places all the iSCSI processing on the FlexFabric adapter, freeing up CPU cycles for applications like VDI. 

In addition to full iSCSI offload, the Virtual Connect FlexFabric module supports and manages iSCSI boot.  The iSCSI boot parameters can be managed as part of the Virtual Connect server profile, creating a stateless server using a cost effective iSCSI boot device.  This capability was only available on Native Fibre Channel in previous versions of Virtual Connect, now it’s available on all storage protocols supported by the Virtual Connect FlexFabric module; Fibre Channel, FCoE and iSCSI.

The Virtual Connect FlexFabric module’s FCoE capability has some unique advantages too.  By combining Ethernet and Fibre Channel into a single unit, it reduces the number of I/O modules within the blade infrastructure, reducing complexity and cost.  As HP offers the only mainstream blade infrastructure that can support 10GB on the system board, it is the only blade server family that can embed the CNA on the system board.  This again has the advantages in simplicity and cost, no extra components to buy. 

A third advantage is how the FCoE capabilities are contained within a blade enclosure.  While the FCoE standards are complete and just waiting for final ratification, that isn’t the whole story.  Ethernet by it’s very nature is a very lossy protocol.  Fibre Channel requires lossless delivery.  While Ethernet can retransmit just the lost frames, Fibre Channel must retransmit the entire transaction, which could be several megabits.  So Ethernet had to be modified to provide a lossless transport capability to support FCoE.  These modifications are defined in a new set of standards called Converged Enhanced Ethernet (CEE).  Without getting into the detail, only one out of the three CEE standards are complete.  This creates risk for any products implementing pre-production standards.  By keeping the FCoE within the blade enclosure, early adopters can take advantage of these emerging standards but still be protected while the standards are still in flux. 

It’s important to note that the Virtual Connect FlexFabric module doesn’t replace the existing Virtual Connect modules.  It is possible to mix and match pairs of Virtual Connect modules as needed.  For applications with high I/O requirements, traditional Ethernet and Fibre Channel might still be the preferred option.  So whether you plan on sticking with conventional Ethernet and Fibre Channel, need better support for iSCSI, or you’re looking to move to FCoE, Virtual Connect has a solution to fill the need.

For more information on HP FlexFabric see: http://www.hp.com/go/flexfabric

¹ http://h10144.www1.hp.com/solutions/enterprise/datacenter/flex-fabric.htm

Follow me on Twitter;  http://twitter.com/bladeguy

As always, these opinions are mine and don't necessarily represent the views of HP.

[#2]

I see limited value in this particular module. While I agree using a CNA on the server side can cut costs (no more separate FC cards and FC module), the module would be vastly more future proof if the uplinks also supported FCoE. Since the module does not appear to support FCoE uplinks, I will need to rip and replace these FlexFabric modules when I want to utilize FCoE to the blade chassis. That's not much investment protection.

I would have preferred to see each uplink support FC, 10GbE, and FCoE.

[#3]

Tom,

Running multi-hop FCoE is risky while the DCB standards are still in flux.  FlexFabric provides the opportunity to take advantage of the cost savings FCoE provides with out exposing the infrastructure to that risk.  FlexFabric can reduce the cost by ten to sixteen thousand dollars per enclosure.

As with any networking device, much of the functionality of the VC FlexFabric module is controlled by firmware.  It is likely that the functionality of future firmware will evolve as FCoE/DCB technology matures.  What that specific functionality will be can't be known until the standards are finalized.

As always, these opinions are mine and don't necessarily represent the views of HP.

Ken

[#4]

While Tom is correct that an end-to-end multi-hop FCoE network is desirable, the fact is very few are even considering it today.  FCoE, if anything, is a way to reduce costs, both in equipment and operations.  Given that that the vast majority of FC ports are at the edge, CNAs and the FlexFabric Module allow customers to capture this cost reduction immediately using a well-known approach while waiting for the standards to complete.

Customers don't move to a new architecture overnight and this edge-to-core migration over time makes sense for many of them.  The ability to mix and match FC and 10GbE ports on the FlexFabric Module is another feature that gives customers the flexibility in how quickly they want to adopt this technology.

Deepak

[#5]

One of the fundamental "flaws" of the previous Flex-10 modules is inflexibility of configuring the NIC bandwidths. By that I mean I can only set maximum bandwidth per flex-NIC, I can't set minimums. So that results in wasted bandwidth because operations like vMotion are not constant and require large bandwith so I have to allocate a fixed bandwidth to the vMotion NICs. If HP would let me set minimum bandwidth, I could preserve general server VM traffic while letting vMotion use all other available traffic. Likewise, if I wasn't doing any vMotions the server VMs could use all the bandwidth if I was doing something like backups or large file copies. Furthermore, I understand the Flex technology only caps egress traffic, not ingress. And one feature that is really needed is the ability to group NIC server configurations so that I could, for instance, modify the bandwidth or network assignment to a group of ESX servers at the same time with a couple of clicks. As it stands now I must make the same tedious configuration change on every server in a serial manner.

I'm not trying to bash HP, as I have a couple of C-7000 chassis with Flex technology. However, other vendors are surpassing the technology with a more flexible configuration, QoS support, and features that make it more dynamic which is extremely important.  Plus the new flash interface for VC has resulted in the inability to launch VC on our servers since Flash it banned from all servers due to constant and significant security vulnerabilities it has on a very frequent basis.

[#6]

Hello

i have connect blade server (bl460c) to cisco nexus 5020 with fcoe connection.what I need  adapter and switch on server blade and blade enclosure c7000?

tanks for all.

please help me

[#7]

According to Cisco there's a lot of FUD around which standards are in flux and what is really needed to support multi-hop FCoE. Seems to me that Cisco's stance is that everything that is needed for FCoE is ready to go so there's no reason to support it in shipping products. http://blogs.cisco.com/datacenter/comments/will_the_real_fcoe_standards_please_stand_up/ What is HP's response to Cisco's claims? Exactly which standards does HP feel are not yet solid enough to base shipping products on?

[#8]

Can you provide us some details on what QoS features the FlexFabric module has? One of our requirements is to support end to end QoS, from the Nexus 1000v through to our external switches. The Cisco UCS solution fully supports QoS so unless things have changed from the Flex-10 days, we can't consider HP's solution since it has no QoS support.

[#9]

Brad,

It is common to overestimate the bandwidth requirements for various functions.  In a typical ESX implementation I configure 3 redundant networks; management, production and VMotion.  Management has very low bandwidth requirements so I configure 100Mb.  Production seldom requires more than 1Gb.  This leaves 8.9 Gb for VMotion if you wanted to configure for maximum VMotion bandwidth. 

As for reconfiguring bandwidth for various functions like backup, I would want these processes to be fully autoamated, and not rely on someone configuring through a GUI.  In the scenario you presented I would write a simple Virtual Connect script to automate the bandwidth adjustements.

Saloam and Brad,

You both asked about QOS.  First off, the Nexus 1000v does not enforce QOS, it only supports QOS tagging.  If you need end to end QOS, the 1000v is not the right solution.  If you do need to guarantee bandwidth for a specific  network, you can configure a dedicated uplink for that connection to meet the application's requirements without the management complexity of QOS.

So while on paper other technologies might look better, Virtual Connect can be configured to meet and exceed most real world requirements.  If not, the c-Class portfolio has other interconnect options like the ProCurve 6120XG(which does support QOS).

David,

It is my opinion that Congestion Notification is the biggest hole in the current DCB/FCoE offerrings.  While the FCoE protocol itself is complete, it relies on the DCB enhancements to Ethernet to create the lossless network.  See my blog This is not the 10Gb network you’re looking for for an in depth discussion.  I'll be updating that blog later this morning.

Ken

[#10]

I as a HP customer would be perfectly happy deploying hardware today without QCN in the ASIC. We do not tax our networks, and see the benefits of end-to-end FCoE. So HP, let the customer decided what is or is not important and let them choose. Deliver FCoE products today that supports everything but QCN, that is if you didn't build QCN into the FlexFabric module.

By not supporting FCoE uplinks on your FlexFabric module you have removed choice from being in the customer's hands and letting HP dictate what we should be using. Give us choice!

[#11]

There's a good debate going on here, http://blogs.cisco.com/datacenter/comments/when_are_fcoe_standards_done/, about QCN and if its actually relevant to FCoE. It seems to me that HP's view that QCN is a hard requirement for an enterprise FCoE deployment is in the minority. I would question HP and wonder if they are using QCN as a smoke screen or an excuse to delay shipping their FCoE products when their competitors have been shipping them for months if not longer.

[#12]

I find your comments about QoS troubling. First, in a blade chassis with possibly hundreds or even thousands of VM's if you think I'm going to try and dedicate one uplink to a single application to ensure QoS, that's just not sane. Second, having to script VC to change bandwidths dynamically is also not very real world. Why can't HP just give me the ability to define guaranteed minimums for the FlexNICs? And there's still the problem that the Flex architecture only rate limits egress traffic, not ingress.

You are right the Nexus 1000v is not the total QoS solution, and that's my point. The hardware at each hope needs to support intelligent QoS and the Flex modules do not support it. In an ideal world I'd use the 1000v, a QoS aware flex module, and QoS aware external switches and routers. That architecture provides me a flexible, scalable, and enforcable QoS infrastructure. The in-flexibility of the 'Flex' modules really has me confounded.

Plus as another comment said, you can't group server profiles and make mass changes to all of them at once. If I have 16 ESX servers and I want to change the VLAN configuration I'm forced to make the tedious change 16 times, increasing the chance of a mis configuration, increases operating costs, and takes time away from doing more important tasks. Both IBM and Cisco seem now be ahead of HP in terms of the "FlexNIC" technology and the management thereof. I hope HP can come out with a new firmware release to address these shortcomings and catch up to competitors. Yes IBM was late to market with a multi-NIC virtualization product, but they have a more flexible product than what HP can offer.

[#13]

David,

In my experience QOS is not required for the vast majority of network connections.  In a properly designed network you only need QOS for voice or video.  QOS everywhere seems to be a fairly new phenomenon, and I'm not sure what's driving it.  QOS only helps in a narrow band of network utilization.  On a lightly loaded network QOS isn't required.  On a completely congested network traditional QOS won't help.  So QOS can only help on a congested network that isn't saturated for most applications.

When it comes to an individual server, it is nearly impossible for  a single server to flood one 10Gb interface let alone two in normal use cases.  A properly configured Flex-10 network is more than up to the vast majority of datacenter workloads.

As for making mass configuration changes with Virtual Connect, I have long been an advocate of scripting.  In a scripted environment the changes you're looking to make are trivial.

As for IBM's offering, it falls far short of what Virtual Connect offers.  IBM lacks the concept of a server profile.  With Virtual Connect profiles you define the connectivity as well as the personality, and both are moved as one.  This allows you to move a personality to a different server in a single step to support hardware maintenance or failure recovery.  IBM only defines the personality, so you'd have two additional tasks to move Ethernet and SAN connectivity (and hope you got it all right).

Ken

[#14]

LE77,

First off I don't speak for HP.  The opinions expressed by me and all the other bloggers in these blogs re our personal opinions.  Second, QCN has nothing to do with FlexFabric.

My previous blog on the state of DCB protocols, wasn't about FCoE, it was about when to make a major upgrade to the core networking infrastructure.  If at all possible, why not put off the upgrade until you know what will be required, and what's supported.  

You may be considering a purchase from Vendor-X today, and still end up with Vendor-X in another 12 - 18 months.  But the advantage is that all these new protocols will have settled down, and you'll have a higher degree of certainty you bought the right product.  What would you do if you made a $1M network upgrade today, and found out it didn't support the protocols you need in 15 months?

Like I've said before, "the DataCenter network will change more in the next 15 months than it's changed in the last 15 years."  Why rush into a buying decision before the dust settles?  

Ken

[#15]

Ken,

I'm not worried about one server flooding a 10Gb NIC on egress. But how about this situation: 16 ESX hosts, each is allotted 4Gb for vMotion. If I vMotion 3-4 VMs on different hosts to one host, I could easily flood the target ESX host with vMotion traffic because the Flex-NICs have no ingress rate limiting capabilties. Or if I have a DRS cluster and one or two hosts go into maintenance mode and there's a mass vMotion migration. ESX 4.1 has really optimized vMotion traffic and it can utilize more bandwidth than previous versions.

I would hate to be the customer that you tell him or her that you expect your typical sysadmin to try and script VC to make multi-server changes. The storage company HP is trying to by, 3PAR, introducted a storage feature earlier this year called autonomic provisioning. The whole theory behind this feature is to automate one-to-many and many-to-one configuration tasks related to LUNs and hosts. Zero scripting is needed..just a few point and clicks in a GUI. The same ease of use, low TCO, reduction in human error, and high-efficiency thought should be baked into the VC interface. It would be very trivial to add 'server groups' that one could apply a sub-set of a port profile to, such as the bandwidth allocations or VLANs.

Whenever a company comes back to me with a 'script it' answer, I really question their understanding of real-world IT and the level of people handing day-to-day operations. While I agree some tasks are best suited to a script, 3PAR has realized common tasks that other companies require multiple steps or scripts to complete are best accomplished by back-end intelligence and a slick GUI.

[#16]

David-

In your regards to your situation, "I'm not worried about one server flooding a 10Gb NIC on egress. But how about this situation: 16 ESX hosts, each is allotted 4Gb for vMotion. If I vMotion 3-4 VMs on different hosts to one host, I could easily flood the target ESX host with vMotion traffic because the Flex-NICs have no ingress rate limiting capabilties."

Why not just trunk the whole 10gb down two links then take network i/o and then carve out the bandwidth since it is Dynamic shared based instead of dedicated.  You dont have to map your VLANS inside of Flex-10 and use all four nics on each LOM that is your choice.

Honestly, I dont work for HP....Not a huge fan of HP storage but I can honestly say my clustered C7000 chassis with stacked Flex-10 fell short and was a truly great experience from a customer prospective.