Cluster Computing John Kelleher

Cluster Computing
HA Linux Project
Sys Admin
Tsunami Technologies
ARS Technica
Force10 Networks
Microsoft TechNet
Load Balancing in Server 2003
University of Buffalo
Who Cares?
Science Field (SETI)
Medical Field (Protein Folding/ VR)
Hollywood (SFX)
Anyone who needs massive amounts of
computing power – on a “budget”
Types of computer clusters
Who uses them?
Benefits of each
3 Types of Clusters
• High Availability (HA)
• Load Balancing / (LB)
• High Performance/Parallel (HP)
High Availability Clusters
• Two or more servers setup so that if one
fails, the other takes over
– Share a common data source
– Only one accesses data at any time
– Monitor each-other by sending “heartbeat” packets
between themselves
– When one server fails, the work is migrated over to
the working one
– Users experience a brief break in transmission
(typically a few seconds) but not a total loss
HA Clusters ( cont )
Load Balancing Cluster
• A main server distributes data/application
requests to any one of many other
– A web server could distribute http requests to
appropriate servers based on proximity to
client, etc
– Often called a “server farm”
– Improve performance by splitting up requests
Load Balancing ( cont )
High Performance Cluster
• Divides computational tasks amongst
many different nodes, allowing their
shared processing power to be utilized
– 296 of the world’s fastest computers are HP
clusters (CNET News)
– Nodes can communicate with each-other
while processing
– Processor intensive calculations that would
take years on a typical computer, can be
completed in a small fraction of the time
HP Clusters ( cont )
• The price per gigaflop when using a
supercomputer is about 5x more
expensive than with a cluster
• Clusters have brought the price per GFlop
down to around $85
• Supercomputer prices have come down as
well though a high-end Sun Fire 15K
server with 16 1.0Ghz processors and 64
gigabytes of ram costs $320,000
Costs ( cont )
• Cost of ownership
– Important to remember you are not just
buying the hardware inside the computer case
– Switches, cabling, and interface hardware
must be purchased as well
– Power consumption of a large cluster (for
example 20 nodes) will carry a price tag in
excess of $3000 per year
– Cost to free up space for a large cluster
– Cost to pay employees to set it up and
monitor it
Case Study
University of Buffalo, New York
800 Dell nodes each housing dual 3.2 GHz Xeon Processors
Case Study
• The University of Buffalo cluster cost $2.3 million
to implement
– Runs at a peak of around 10 TeraFlops (10 Trillion
Floating Point Operations Per Second)
– Put them around #40 on the Top500 most powerful
computer list
– Costs $150,000+ per year in electric costs
– For now can only be run at 60% power without
causing a power outage
– Rendered a video that would have taken a typical
home PC 125 days to render, in 8 hours
Cost Effective Alternative
• Many companies offer “cluster on demand”
– Rent time on clusters from 3rd party companies
– Great for companies that do not need that much
computing power, all the time
– Typically pay per CPU hour:
60 cents per CPU hour
40 node cluster
72 hours
$1700 (most companies have minimum hour requirements)
HP Cluster Construction
• MPI (Message Passing Interface)
– Most widely used standard
– Sought to unify parallel programming
– Gives programmers a starting point
– Works best when programmed with Fortran,
but C++ works as well
Cluster Construction ( cont )
• Install Fortran ( or C ) Compiler
• Configure SSH (need to be able to
execute commands at root without asking
for a password)
• Setup a network file system
• Install MPI (from
• Edit MPI file to add each node’s hostname
• Run test programs!
(most steps were taken from
• Create a HP Cluster in the computer lab
• CCSU audio/video server into a high
availability setup