power - Does the choice of CPU affect the energy budget of a commodity server farm?

How does the choice of CPU affect the energy budget of a commodity server farm? I'm wondering how much it costs per year to power the kinds of commodity servers Facebook, Twitter et al. use.

Let's take one such typical server – an Intel Xeon processor X5650 – and a typical load with all 12 cores each running at 90% capacity.

What would the yearly power consumption be? What assumptions need to be made other than the above scenario? And how much would this cost a company operating at the scale of the above? How about cooling. How much to cool such a server?

If the load was reduced to, say, 45%, by what proportion would power and cooling costs go down?

Your original question is entirely too broad, as it is impossible to make even a rough estimate of actual dollars, because there are just way too many variables involved (hundreds of thousands). I'm not even going to attempt to come up with a number. Instead, I'm just going to discuss one particular aspect of hosting costs: the CPU.

CPUs!

CPU operating costs depends heavily on the generation of the server.

By generation I mean, approximately when the processor was released, but more importantly, what architecture it is based upon.

As processors have become more complicated and more advanced, they have also developed more sophisticated power management features. Let's take a look at the evolution of Intel processor power management, and keep in mind that all features are cumulative over time, which means that if a feature existed in an older processor, it probably exists in newer ones, but it may be improved or more efficient in newer processor generations.

For an extreme example, if you were to look at a very old server chip, something like a Pentium II Xeon, you might notice that the spec sheet does not mention anything about power saving technology. This chip will use its maximum TDP pretty much all the time.

Something a little bit newer (but still very old - ~2005), like a Xeon 3.80E, which is somewhere newer than Pentium IV but older than "Core" architecture, starts to show signs that Intel is conscious of power savings: "Intel SpeedStep Technology" is listed in the product data sheet.

Jump ahead to a Core2 microarchitecture based Xeon (~2008), like this X5365, and you notice several things:

• The CPU has Idle States power saving functionality, which means that it can drop down into a lower power mode, some point in between being "powered off" and "fully running", when it is not actively doing anything. It can switch idle states tens or hundreds of times per second, for lots of very "fine-grained" power savings.

• The SpeedStep technology we saw before is now Enhanced SpeedStep technology, which is finer-grained, meaning that the CPU can drop to a specific voltage that exactly fits the current workload, and if the workload increases or decreases, it can scale both the fan speed as well as the CPU's power consumption based on the workload changes (even if the workload is fluctuating more than once per second).

• Starting at the Core Microarchitecture, we start to see Intel selling Xeon processors with the "L" prefix, for "Low-voltage". These processors have a lower Thermal Design Power (TDP), which means that they are designed to consistently operate at lower amounts of energy consumption (primarily driven by operating at lower voltage) than their more power-hungry counterparts. These parts are offered as an option, because you can get slightly more performance if you use the ordinary power-hungry parts, which are prefixed with an "E" or "X".

• Intel Demand-Based Switching technology is used. Quoting the Intel website:

  Intel® Demand Based Switching is a power-management technology in which the applied voltage and clock speed of a microprocessor are kept at the minimum necessary levels until more processing power is required. This technology was introduced as Intel SpeedStep® Technology in the server marketplace.

Jump ahead to a Nehalem microarchitecture based Xeon, like this X3480, and you notice several things:

• Intel Turbo Boost Technology; this CPU will run at very power-efficient (great performance per watt) speeds most of the time, but if the CPU gets very high utilization spikes, it is able to exceed normal TDP (for increased power consumption and less efficiency) to deliver more performance in "Turbo" mode.

• Hyper-threading; this means that you can run four cores with performance near what the performance would be with eight cores, but with very high efficiency. Hyperthreading is both a cost-saving and a power-saving mechanism, allowing you to get better performance per watt (you put the same energy in, but get more performance out, than you would on processors without hyperthreading).

Jump ahead to a Westmere (Nehalem-C a.k.a. die shrink of Nehalem) Xeon X5650, which is what you asked about in your question, and the situation is basically the same power-wise as the original Nehalem above, except that you will have slightly lower power usage across the board, because the fabrication size is smaller.

Now, after Westmere/Nehalem, we have three more microarchitectures to go to get us up to the present day:

• Sandy Bridge, 2011, a 32nm processor family with a new microarchitecture (aka "tock");
• Ivy Bridge, 2012, a 22nm processor family based on the Sandy Bridge microarchitecture but with lower power consumption and better power efficiency (aka "tick-plus");
• Haswell, 2013, a 22nm processor family with a new microarchitecture (aka "tock").

Each of these successive generations of processors has given us better power management features, as this is one of Intel's primary focus areas right now, for several reasons:

• They're shipping x86 tablets with very small battery capacities, which need power-efficient processors.
• Datacenters want to reduce energy and heating and cooling costs.
• As normal desktop users stop needing more and more computing power, they can start to reduce their energy costs, heat output, etc. by going with more efficient processors that offer more modest performance improvements but dramatically cut down on power use.

• As processors get more complicated internally, there is more room for complicated logic and circuitry that calculates exactly how much power is needed to provide the current workload efficiently, while keeping waste to an absolute minimum. Intel has invested heavily in this technology in recent CPU generations.

• A Sandy Bridge low-voltage Xeon, such as the E3-1260L, is quite efficient, while also being quad-core with hyperthreading. Its 45 Watt TDP shouldn't be taken as an indicator that it's slow; far from it -- it's much faster than even 105 W TDP processors from just a few years earlier.

• An Ivy Bridge low-voltage Xeon, such as the E3-1265L v2, is even more efficient than the 1260L, with still a 45 Watt TDP, but significantly better performance and an on-CPU voltage regulator for extremely fast voltage change response.

• A Haswell low-voltage Xeon, such as the E3-1265L v3, is the pinnacle of what is currently available on the market in terms of power efficiency, with still a 45 Watt TDP, but yet better performance and more power saving goodness.

Of course, outside of the low-voltage category, there are also higher performance Xeons in the recent series, such as the beastly 15-core E7-8890 v2, due for release in Q1 2014, which is incredibly high-end, with a 155 Watt TDP (very large for a CPU) -- the huge difference being that, with all those high-end cores, this processor can do a lot more than a low-voltage chip.

Overall, your X5650, being four generations removed from the current gen (it only has 6 cores with hyperthreading, by the way, not 12 cores), is going to rival something like a commodity-level, quad-core "E3" branded Xeon from the Ivy Bridge or Haswell generations, even though it has more cores. The newer CPUs have a higher clock rate, more L3 cache, support faster RAM, and are more power-efficient than an older X5650, which means that they will be able to keep up with it even though they have fewer cores.

Load adjustment's impact on cost

If the load was reduced to, say, 45%, by what proportion would power and cooling costs go down?

Well, based on what we know above, if the CPUs were modern CPUs (Sandy Bridge, Ivy Bridge, or Haswell), chances are that power and cooling costs for the CPU alone (to say nothing of the motherboard, hard drive, RAM, etc) would probably go down approximately linearly with reduced load. That's the ultimate goal, anyway: if you want X number of instructions per second, it'll cost $YYY; if you want X*10 number of instructions per second, it'll cost $YYY*10. Linear scales make for very predictable economics, so that's what Intel is aiming for.

Of course, the older generation CPUs didn't even come close to being linear, because they'd waste a lot of power simply being idle, and even when they were fully utilized, they weren't making the best use of available resources, because they lacked features such as HyperThreading.

Attempting (very vaguely) to sketch out how to approach an answer to your original question

Now that you know all of that detail about CPUs, I'm going to let you in on a secret: CPUs don't make up the majority of the operating costs of major websites. The biggest costs are employees, facilities (real estate, land, datacenters, etc), and cooling.

In order to come up with a "back of the envelope" operating cost for something like Facebook or Twitter, you have to consider the following:

• Even if they use the newest CPUs, they're still paying for electricity for cooling (it costs less, obviously, if the outside environment is very cold, compared to in the summer when it can cost a great deal to keep servers cooled). Cooling costs vary depending on whether they're using "off-grid" power (wind, solar) or buying electricity from the utility company, which could be making it from coal or nuclear, etc. The costs also vary greatly from country to country, as energy costs are variable from region to region, depending on where they get their energy from, how much demand there is, etc.

• Motherboards, hard drives, SSDs, RAM, cooling fans, networking equipment, lighting, safety equipment, offices for employees, etc. all consume additional energy, and the costs incurred here can vary wildly depending on how efficiently the operation is being run.

• Facilities costs will depend on how much emphasis is placed on safety and security. For instance, backup diesel generators incur a significant cost, both in terms of fuel, regular testing of the correct operation of the generators, battery packs (to maintain consistent power while the diesel generators start up), etc. Without these costs, your datacenter is more apt to experience total outages in the event of a main power failure, but day-to-day operating costs will go down significantly. Also, costs go up with things like security cameras, armed guards, badge readers, etc., which can also be considered "optional extras" if you're really trying to go barebones.

• You also have to refine the definition of the question you're asking in order to come up with accurate numbers. For instance, are IT support specialists, who maintain the server hardware and network, considered to be part of the cost of a datacenter operation? Are the programmers who write the software factored into the cost? How about system administrators? How about managers? How about custodians who clean the floors and replace light bulbs? How about taxes they owe to the government? Where do you stop measuring the costs? This is all part of the definition of your problem, and because I'm not psychic and don't know what exactly you are asking, I am not going to attempt to answer any of these questions. Most of them are off-topic for SuperUser, anyway.

Having said all that, I'm still not going to make any rough estimates in terms of dollars. You'll have to figure that out on your own, based on whatever assumptions you care to make about the company's operating procedures, cost of electricity, outside temperature, cost of labor, etc.