Richard is known as "Mr. NetPerf" to all the people that seek his advice on networking standards, performance and all the nuiances that it entails. Recently he did a great reply to a question about Network Performance testing and I wanted to share it with you.
The original question came from Glen:
I am mucking about with some lab gear, trying to show server Virtual Connect to IRF throughput. So far I have managed to generate about 3 Gbps of traffic with just three VMs on a single BL460c Gen8 server using iPerf. I am curious to know what tools folks have used for server traffic generation that might be better than iPerf for lab testing?
And here was Richard's reply:
Actually, if one's goal is strictly bits per second iperf is probably no worse than anything else. I will point-out there is more to networking than bits per second.
If the goal is to show the throughput achievable by the combination of server, VC and IRF, why the VMs? It may not be iperf holding things back...
Now, having said that, some boilerplate I trot-out from time to time on the topic of networking performance, which may have some useful tidbits. You may have to "translate" some of the Linux utilities to your OS of choice:
Some of my checklist items when presented with assertions of poor network performance, in no particular order, numbered only for convenience of reference:
1) Is *any one* CPU on either end of the transfer at or close to 100%
utilization? A given TCP connection cannot really take advantage
of more than the services of a single core in the system, so
average CPU utilization being low does not a priori mean things are
2) Are there TCP retransmissions being registered in netstat
statistics on the sending system? Take a snapshot of netstat -s -t
from just before the transfer, and one from just after and run it
through beforeafter tools:
netstat -s -t > before
transfer or wait 60 or so seconds if the transfer was already going
netstat -s -t > after
beforeafter before after > delta
3) Are there packet drops registered in ethtool -S statistics on
either side of the transfer? Take snapshots in a manner similar to
that with netstat.
4) Are there packet drops registered in the stats for the switch(es)
being traversed by the transfer? These would be retrieved via
5) What is the latency between the two end points. Install netperf on
both sides, start netserver on one side and on the other side run:
netperf -t TCP_RR -l 30 -H <remote>
and invert the transaction/s rate to get the RTT latency. There
are caveats involving NIC interrupt coalescing settings defaulting
in favor of throughput/CPU util over latency but when the connections are over a WAN latency is important and
may not be clouded as much by NIC settings.
This all leads into:
6) What is the *effective* TCP (or other) window size for the
connection. One limit to the performance of a TCP bulk transfer
Tput <= W(eff)/RTT
The effective window size will be the lesser of:
a) The classic TCP window advertised by the receiver. This is the
value in the TCP header's window field shifted by the window
scaling factor which was exchanged during connection
establishment. The window scale factor is why one wants to get
traces including the connection establishment.
The size of the classic window will depend on whether/what the
receiving application has requested via a setsockopt(SO_RCVBUF)
call and the sysctl limits set in the OS. If the receiving
application does not call setsockopt(SO_RCVBUF) then under Linux
the stack will "autotune" the advertised window based on other
sysctl limits in the OS. Other stacks may or may not autotune.
b) The computed congestion window on the sender - this will be
affected by the packet loss rate over the connection, hence the
interest in the netstat and ethtool stats.
c) The quantity of data to which the sending TCP can maintain a
reference while waiting for it to be ACKnowledged by the
receiver - this will be akin to the classic TCP window case
above, but on the sending side, and concerning
setsockopt(SO_SNDBUF) and sysctl settings.
d) The quantity of data the sending application is willing/able to
send at any one time before waiting for some sort of
application-level acknowledgement. FTP and rcp will just blast
all the data of the file into the socket as fast as the socket
will take it. Scp has some application-layer "windowing" which
may cause it to put less data out onto the connection than TCP
might otherwise have permitted. NFS has the maximum number of
outstanding requests it will allow at one time acting as a
defacto "window" etc etc etc
7) Another magic forumla for TCP bulk transfer performance comes from
Mathis, Semke, Mahdavi & Ott
Tput <= (MSS/RTT) * (1/sqrt(p))
MSS is Maximum Segment Size
RTT is Round Trip Time
p is the packet loss rate as a probability (eg values of 0 to 1.0)
Which assumes a few things about the congestion control algorithm
being used and that there is no classic TCP window limitation as
mentioned in item 6.
8) Is the link/path between the sender and the receiver composed of
single-link hops, or might some be aggregated link hops? If the
latter, does traffic from a single flow (eg TCP connection) get
striped across each link, or does it stay on just one link in the
aggregation(s)? Striping across multiple links can lead to packet
re-ordering which will affect TCP performance. If there are
aggregated links and no striping then the advertised "N Gbit/s" may
really be "1/n Gbit/s" per flow.
Thanks to Richard and to get more info on Virtual Connect go to: www.hp.com/go/vitrtualconnect