I saw a question from our field pre-sales team this week regarding issues at a customer when they were connecting a 10GbE pipe to a 1GbE pipe through Virtual Connect. Richard Jones replied to the request for some help and I thought it was great info for all of us to remember in these situations.
Here's the request for help:
**********************
"Can you help us with our problem where we see huge amounts of retransmissions from the 10GbE Virtual Connect uplink to 1GbE clients? All the switch ports outside are clean up to the 10G Virtual Connect uplink to 1GbE clients. What has to be set on the Virtual Connect interface to stop the dropped frames (ending with the segment loss and retransmission)? Is it a problem to have 10GbE Virtual Connect with 1GbE clients?"
*************************
And Rick gave an indepth reply.
********************************
A few things:
*) Define "huge." In something like a percentage of total traffic
rather than an absolute number of retransmissions.
*) The VC uplink in and of itself does not source traffic nor
retransmit it. These retransmissions are from hosts upstream of
the 10 Gbit/s VC uplink to blades with downlinks operating at 1
Gbit/s?
*) Please say more about the nature of the traffic - number of
connections, if they are TCP, what they are doing, etc.
Generally speaking, if you have a source which operates at 10 Gbit/s and a destination which operates at 1 Gbit/s, a queue will build-up just in front of the 1 Gbit/s link. How full that queue will be will depend on how much faster than 1 Gbit/s the data arrives off the 10 Gbit/s link, and for how long. If fast and long enough, the queue will fill, and packets will be dropped. The only way to avoid having the queue fill is if the sustained rate to the 1 Gbit/s destination stays below 1 Gbit/s, and any "bursts" do not exceed the size of the queue.
Since "retransmissions" implies TCP, the sending TCP will back-off on something it calls the "congestion window" and retransmit. However, TCP always probes for additional bandwidth, so will grow the congestion window again, and it may grow to be larger than the queue.
The cycle will continue for the life of the TCP connection(s). The congestion window is something a sending TCP computes and is distict from, but capped by the receivers advertised window (the "classic"
window which appears in the TCP header)
The only way to "guarantee" no retransmissions in such a speed mismatch case is to "guarantee" that at no time is there more data outstanding going towards the 1 Gbit/s destination than the size of the queue in front of that 1 Gbit/s link. That means capping the TCP window advertised by the receiver(s) on the far side of the 1 Gbit/s link to less than the size of the queue in front of that 1 Gbit/s link.
There are "wrinkles" - things like queues employing Random Early Drop (aka RED) or other forms of Active/Adaptive Queue Management, but I do not know if the VC module(s) imploy RED et al or are just plain "drop-tail" (aka drop packets when the queue is full). Another wrikle is that Linux at least will "grow" the classic TCP window too if allowed.
But basically, short of something pathological, when there is a speed mismatch such as you seem to have described, queuing is inevitable.
Thus dropped packets and retransmissions are virtually inevitable.
********************
As always, great info and insight from Mr. Jones. Let me know what you think and othe thoughts about mismatched bandwidth issues.
