Eye on Blades Blog: Trends in Infrastructure
Get HP BladeSystem news, upcoming event information, technology trends, and product information to stay up to date with what is happening in the world of blades.

Visiting the Blades Lab at HP Blades Tech Day (#hpbladesday)

By Dan Bowers


Up in the BladeSystem SWAT lab for a mid-afternoon session.   (For some reason, there's a big jar of Bit-o-Honey candy in here;
everyone is digging in.)  Two cool things in this lab: plenty of cover-off hardware (including
a special "chopped in half" c7000 enclosure so you can see the innards); and Dell, IBM, and other
blades up and running, which is great for comparisons.



James Singer and Gary Thome from HP discussing BladeSystem hardware in detail.  
Simon Seagrave (http://www.techhead.co.uk/) got top marks for knowing that the true name of the purple-ish color
used on BladeSystem to indicate hot-pluggable components is "port wine".


Power supply efficiency, says James, is usually quoted as a single number, and refers to the efficiency
when the supply is running at around 80% of capacity.  However, it's important to look at the
efficiency at the whole range of the power supply.


THe newest power supplies are 2400W and "a lot more efficient than their predecessors in the lower
utilization zone."  James used the whiteboard to explain how the 2400W power supplies intro'd
last year for HP BladeSystem have their efficiency improved greatly in the low- to mid- utilization;
my pitiful attempt to recreate his drawing in MS Paint below:



Quiz: Why did HP choose 6 power supplies for the c7000 enclosure?   THe storage bloggers
struggled with this question :smileyhappy:  but eventually folks came down to the right answer.   Generally,
enclosures are fed with two sepearte power feeds (for redundancy); and data center power feeds
tend to be 3-phase power.   3 phases, two feeds, equals the need for six supplies.  If you have
something not divisible by 6, you get unbalanced demand on the power phases at the data
center level, which can mean bigger power bills, irate power companies, or both.



Simon Seagrave (http://www.techhead.co.uk/) gave his own quiz to the HP engineers.
"Were the BladeSystem power supplies invented by someone in the model airplane industry?  Or is this
an urban myth?"  Well, says Gary, the truth is close: engineers (including HP's Wade Vincent)
who were developing cooling systems for HP BladeSystem uses concepts they learned as model airplane
enthusiasts to develop the BladeSystem fans.

Applications Matter - What Affects Server Power Consumption: Part 2

 How does the application you are using and what it is doing affect the power consumption of system.


The first thing that everyone looks at when talking about power consumption is CPU utilization.  Unfortunately CPU utilization is not a good proxy for power consumption and the reason why goes right down to the instruction level. Modern CPUs like the Intel Nehalem and AMD Istanbul processors have 100s of millions of transistors on the die.  What really drives power consumption is how many of those transistors are actually active.  At the most basic level an instruction will activate a number of transistors on the CPU, depending on what the instruction is actually doing a different number of transistors will be activated. So a simple register add, for example, might integer add the values in two registers and place the result in a third register.  A relatively small number of transistors will be active during this sequence.  The opposite would be a complex instruction that streams data from memory to the cache and feeds it to the floating point unit activating millions of transistors simultaneously.


Further to this modern CPU architectures allow some instruction level parallelization so you can, if the code sequence supports it, run multiple operations simultaneously. Then on top of that we have multiple threads and multiple cores.  So depending on how the code is written you can have a single linear sequence of instructions running or multiple parallel streams running on multiple ALUs and FPUs in the processor simultaneously


Add to that the fact that in modern CPUs the power load drops dramatically when the CPU is not actively working, idle circuitry in the CPU is placed in sleep modes, standby or switched off to reduce power consumption.  So if you're not running any floating point code, for example, huge numbers of transistors are not active and not consuming much power. 


This means that application power utilization varies depending on what the application is actually doing and how it is written.   Therefore depending on the application you run you will see massively different power consumption even if they all report 100% CPU utilization.  You can even see differences running the same benchmark depending on which compiler is used and whether the benchmark was optimized for a specific platform or not and the exact instruction sequence that is run.


The data in graph below shows the relative power consumption of an HP BladeSystem c7000 Enclosure with 32 BL2x220c Servers.  We ran a bunch of applications and also had a couple of customers with the same configuration who wre able to give us power measurements off their enclosures.  One key thing to note is that the CPU was pegged at 100% for all of these tests, (except the idle measurement obviously).



As you can see there is a significant difference between idle and the highest power application, Linpack running across 8 cores in each blade.  Another point to look at is that two customer applications, Rendering and Monte Carlo, don't get anywhere close to the Prime95 and Linpack benchmarks in terms of power consumption.


It is therefore impossible to say what is the power consumption of server X and comparing it to server Y unless they are both running the same application under the same conditions.  This why both SPEC  and the TPC have been developing power consumption benchmarks that look at both the workload and power consumption to give an comparable value between different systems.


SPEC in fact just added Power Consumption metrics to the new SPECweb2009 and interesting enoughly the two results that are up there have the same performance per watt number, but they have wildy different configurations, absolute performance numbers and absolute wattage numbers. So there's more to performance per watt than meets the eye.


The first part of this series was Configuration Matters

Is Power Capping ready For Prime Time

 


Mike Manos responded to my post about power capping being
ready for prime time with a very well thought out and argued post that really
looks at this from a datacenter manager's perspective, rather than just my
technology focused perspective.


I'm going to try and summarize some of the key issues that
he brings up and try to respond as best I can.


Critical Mass


This one spans a number points that Mike brings up,  but I think the key thing here is that you
must have a critical mass of devices in the datacenter that support power
capping otherwise there is no compelling value. 
I don't believe it is necessary, however, to have 100% of devices in the
datacenter that support power capping.  There
are two reasons why:


1.     
In most Enterprise datacenters the vast majority
of the power for the IT load is going to the servers.  I've seen numbers around 66% servers, 22%
storage and 12% networking.  This is a
limited sample so if you have other numbers let me know I would be interested.


2.     
Most of the power variation comes from the
server load. A server at full load can use 2x - 3x the power of a server at
idle.  Network switch load variation is
minimal based on some quick Web research (see Extreme Networks power consumption test or Miercom power consumption testing). Storage power consumption variation also seems to
fairly light at no more than 30% more than idle. See Power Provisioning for a
Warehouse-sized Computer
by Google 


So if our Datacenter manager, Howard, can power cap the
servers then he's got control of the largest and most variable chunk of IT
power.  Would he like to have control of
everything, absolutely yes, but being able to control the servers is more than
half of the problem.


Been there done that,
got the T-Shirt


The other thing that we get told by the many Howards that
are out there is that they're stuck. 
They've been round and round the loop Mike describes and they've hit the
wall.  They don't dare decrease the
budgeted power per server any more as they have to allow for the fact the
servers could spike up in load, and if that blows a breaker taking down a rack
then all hell is going to break lose. 
With a server power cap in place Howard can safely drop the budgeted
power per server and fit more into his existing datacenter.  Will this cost him, sure, both time to
install and configure and money for the licenses to enable the feature. But I
guarantee you that when you compare this to cost of new datacenter facilities
or leasing space in another DC this will be trivial.


The heterogeneous
datacenter


I agree most datacenters are in fact heterogeneous at the
server level either; they will have a mix of server generations and
manufacturers.  This again comes down to
critical mass, so what we did was enable this feature on the two of the best
selling servers of the previous generation, DL360 G5 and DL380 G5 and pretty
much all of the BladeSystem blades to help create that critical mass of servers
that are already out there, then add on with the new G6 servers.  We would of course love for everyone with
other manufacturer's product to upgrade immediately to HP G6 ProLiant Servers
and Blades, but it's probably not going to happen.  This will delay the point at which power
capping can be enabled and for those customers that use other vendors systems
they may not be able to enable power capping until those vendors support it.


Power Cap Management


There's a bunch of issues around power cap management that definitely
do need to get sorted out.  The HP
products do come from an IT perspective and they are not the same tools that facilities
managers typically use.  Clearly there
needs to be some kind convergence between these two toolsets even if it's just
the ability to transfer data between them. 
Wouldn't it be great if something like the Systems Insight Manager/Insight
Power Manager combination that collects power and server data could feed into something
like say Aperture (http://www.aperture.com/)
then you'd have the same information in both sets of tools.


The other question that we have had from customers is who
owns and therefore can change the power cap on the server, the
facility/datacenter team or IT Server Admin team.  This is more of a political question than
anything else, and I don't have a simple answer, but if you are really using
power caps to their full potential changing the power cap on a server is
something that both teams will need to be involved in.


I would like to know what are the other barriers you see to
implementing power capping - let me know in the comments and be assured that
your feedback is going into the development teams.


SNMP Access.


Just to make Mike happy I thought I'd let you know that we
do have SNMP access to the enclosure power consumption.


If you collect all six SNMP MIB power supply current output
power values and add them together, you will have calculated the Enclosure
Present Power.


In the CPQRACK.MIB file, which you can get from here http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?swItem=MTX-a7f532d82b3847188d6a7fc60b&lang=en&cc=us&mode=3&


There are some values


cpqRackPowerSupplyCurPwrOutput which is MIB item:
enterprises.232.22.2.5.1.1.1.10.1 through enterprises.232.22.2.5.1.1.1.10.6
gives you the Input Power of each Power Supply, I know the MIB name says output
but it's actually input - sum these together then you have the Enclosure Input Power.


Power supplies placed in standby for Dynamic Power Savings
will be reporting 0 watts.


And for enclosure ambient temp - read:


CPQRACKCOMMONENCLOSURETEMPCURRENT


Tony

Datacenter Power Allocation

 


I was going to add some more details to Chuck's post on how a blade server powers on, but I got sidetracked by a brilliant post from Mike Manos of Digital Realty on the real basics of what is going with power in your datacenter.


What Mike is explaining, far better than I could, is how power gets used up and reserved in your datacenter by breaker sizes, redundancy and natural tendency of the facility management to be conservative when allocating power to servers, and as he says they have good reason to be.  If they plug in a device that causes a breaker to trip taking down multiple servers - it's their butts that are on the line.


He raised a good question about why the faceplate label, the label on power supply that indicates the max power input, is so high that most facilities managers are comfortable de-rating it by 20% - 30%.  Well the reason is explained in part by my post on how configuration affects power consumption; the power supply is designed to deal with maximum configured load.  The range from a minimum configured load for a 2 socket server e.g. 1 Low Power CPU, 1 or 2 DIMMs, 1 x SSD drive and no PCI cards, to a maximum configured load e.g. 2 x 120W or 130W CPUs, 12 or 18 DIMMs, 8 x 15K RPM Drives, 3 x PCI Cards including a 200W graphics card, is huge and that’s just one server.  The example I use in the Configuration Matters post shows a difference of over 1kW across an enclosure.  Talk to any power supply designer and you'll find out that they are just as conservative as any facility manager (and unappreciated) and for pretty much the same reasons.  Who gets blamed when you run a high power program like Prime95 or Linpack and the server shuts down because the power supply couldn’t deliver enough juice.


That’s why HP came up with the common slot power supply design for rack mount servers. It allows you to size the power supply for the actual configuration you will be using rather than just stuffing a 1200W power supply in every server.


 This has two great consequences:




  1. It reduces the amount of trapped or stranded power by reducing the amount the power that the facility manager has to allocate to a given server.



  2. It increases your power supply efficiency, reducing energy wasted.  All power supplies have an efficiency curve that for servers at low outputs has a low efficiency and gets to peak efficiency at about 35% - 50% 65% load (Got corrected by one of the engineering team on this.  Must remember in future to check my numbers). Remember most servers have redundant power supplies and in the HP case they load share so the PSU can only ever exceed 50% load in the event of a redundancy failure.



This does add complexity to your buying decision, now you have to pick the power supply you need based on your configuration.  That's why we created the HP Power Advisor to help with that decision.  Of course you can still just use a 750W or 1200W PSU for every server if you want to, but you won't be running as efficiently as you could.


One area though where I must respectfully disagree with Mike is in his comments on Power Capping. I agree that is a technology that has huge potential in the datacenter to allow your facilities team to recover that trapped capacity, but I disagree that it is not ready for prime-time.


HP delivered our first version of power capping in 2007. This was relatively slow acting and was really only good for controlling the average power consumption of a server. This was great if you had a cooling issue on your datacenter and wanted to control the heat output of your servers as heat is largely related to average power of the server, but you couldn’t use it to protect circuit breakers.


In November 2008 HP introduced Dynamic Power Capping with circuit breaker protection.  This is a hardware based solution that can respond to changes in power consumption in less than 500ms and because it’s a hardware solution it’s operating system and application independent.  This is supported on all G6 servers, most blade servers and selected G5 rack-mount servers.  When run on an HP Blade Enclosure you gain additional capabilities; the Onboard Administrator can manage the blade server caps to optimize the performance of the enclosure. It will change the blade level power caps so that busier blades get more power and less busy blades will get less power while maintaining the enclosure level power cap so you can protect your breakers.


For a demonstration of this on the rack mount servers showing how we deliver circuit breaker protection see this video with “Professor” Alan Goodrum and for more information Dynamic Power Capping go to http://www.hp.com/go/powercapping


 

Configuration Matters - What Affects Server Power Consumption: Part 1



Following on from my first post I'll take a look at the affect hardware configuration will have on the power consumption of the enclosure.


To do this I went into the Blade Power Sizer and configured up two equivalent systems. I kept the enclosure configuration constant with just 2 x Virtual Connect 1/10 Ethernet Modules and 2 x Virtual Connect Fibre Modules just to simplify the example.


The Blade configuration was BL460cG1, 2 x 2.66GHz CPUs, 16GB RAM, 2x 1Gbit Ethernet, 2x 4Gbit Fibre Channel, 2x 72GB 10K SAS Drives.


Why did I pick BL460cG1, to be honest the actual server doesn't matter, what I'm trying to show here is that the hardware configuration can have a very significant effect on the enclosure power consumption.  The nice thing about the BL460cG1 for this purpose is that it shows this really clearly.


 






















































































 



Configuration A



Configuration B



System



BL460cG1 x 16



BL460cG1 x 16



CPU



E5430 2.66GHz



L5430 2.66GHz



Memory



8 x 2GB FB-DIMMs



4 x 4GB LP FB-DIMMs



Base Ethernet



1GBit Dual-Port Multi-Function



1Gbit Dual-Port Multi-Function



Additional Ethernets



None



None



Fibre Channel



Qlogic 4Gbit



Qlogic 4Gbit



Drives



2 x 72GB 10K SAS



2 x 72GB 10K SAS



Enclosure



c7000



c7000



Ethernet Switches



Virtual Connect 1/10



Virtual Connect 1/10



Fibre Channel



Virtual Connect 4Gbit Fibre Channel



Virtual Connect 4Gbit Fibre Channel



Fans



10



10



Power Supply



HP 2250W x 6



HP 2400W HE x 4



Results



 



 



Idle Power



3698W



2900W



100% Load



5855W



4238W



 


The difference between the two configurations is 798W at idle and 1,617W at high load, which is a huge difference.


Where is most of the power difference coming from, well there are 3 differences in the configuration:



  1. CPU - E5450 (80W TDP) versus L5450 (50W TDP)

  2. Memory - 8 x 2GB FB-DIMMs versus 4 x 4GB Low Power FB-DIMMs

  3. Power Supply - HP 2250W versus HP 2400W High Efficiency


The power supply is worth about 200W (25%) at Idle and 300W (18%) at full load on this configuration. So it's a significant proportion of the difference at the idle load.  At the high loads, though, the majority of the difference between the two configurations is coming from the CPU and Memory.  A standard FB-DIMM takes approximately 10W per DIMM so the difference between 4 and 8 physical DIMMs is roughly 40W per server, additionally a Low Power DIMM uses 2W - 3W less than a standard DIMM.


So what can I take from this example? 



  • System configuration matters. A lot.

  • At high loads the server power consumption is the main factor

  • At low server loads the enclosure becomes a larger proportion of power consumption.


 What are the practical steps I can use to reduce power consumption.



  • Use the lowest power cost effective processor

  • Use smallest number of largest physical DIMMs that are practical and cost effective.

  • Use the highest efficiency power supply that is available


Comments as always are welcome. Let me know where you want me to go as I continue on with this series.


Part 2 of this series is Applications Matter


 

Search
Follow Us


About the Author(s)
  • More than 25 years in the IT industry developing and managing marketing programs. Focused in emerging technologies like Virtualization, cloud and big data.
  • I work within EMEA ISS Central team and a launch manager for new products and general communications manager for EMEA ISS specific information.
  • Hello! I am a social media manager for servers, so my posts will be geared towards HP server-related news & info.
  • HP Servers, Converged Infrastructure, Converged Systems and ExpertOne
  • WW responsibility for development of ROI and TCO tools for the entire ISS portfolio. Technical expertise with a financial spin to help IT show the business value of their projects.
  • I am a member of the HP BladeSystem Portfolio Marketing team, so my posts will focus on all things blades and blade infrastructure. Enjoy!
  • Luke Oda is a member of the HP's BCS Marketing team. With a primary focus on marketing programs that support HP's BCS portfolio. His interests include all things mission-critical and the continuing innovation that HP demonstrates across the globe.
  • Global Marketing Manager with 15 years experience in the high-tech industry.
  • Network industry experience for more than 20 years - Data Center, Voice over IP, security, remote access, routing, switching and wireless, with companies such as HP, Cisco, Juniper Networks and Novell.
  • 20 years of marketing experience in semiconductors, networking and servers. Focused on HP BladeSystem networking supporting Virtual Connect, interconnects and network adapters.
  • Greetings! I am on the HP Enterprise Group marketing team. Topics I am interested in include Converged Infrastructure, Converged Systems and Management, and HP BladeSystem.
Labels
The opinions expressed above are the personal opinions of the authors, not of HP. By using this site, you accept the Terms of Use and Rules of Participation