CLUSTER BOOT ISSUES
Shawn Marriott
COSC 3P93
What is a Cluster?
A cluster is a collection of individual autonomous
nodes working in concert to create the illusion of a
single system.
The trouble with nodes
If you must consider the state, and configuration of
each node in a cluster the illusion of the single
system is lost. As the size of a cluster grows the
management of individual nodes becomes tedious,
and error prone.
A Paradox
How does a node get the information it needs to
boot, if it needs to boot to get the information it
needs ?
Booting a computer
Part of the startup procedure for most computers is to
initialize a bootstrap, which in turn starts the operating
system. Where to find the bootstrap is generally
configure on a per machine basis in the bios. Common
places that are searched for the bootstrap are the
hard drive(s), optical drives, solid state devices, and
floppies.
How do you boot a cluster?
Or at least convince the nodes to boot…….
A Sub Ideal Solution:
Manually configure every node in the cluster…
An ideal solution:




Unpack the computer.
Plug in the computer.
Turn on the computer.
Done.
How could this be done?
If the nodes in our cluster are computers connected
to a network. Then when a computer starts up have
it use the network to broadcast information about
itself to a server, and have the server provide
configuration information and a boot strap.
In the beginning….
There was the Reverse Address Resolution Protocol
(RARP). RFC 903, 1984.
Next try……
Bootstrap protocol (BOOTP). RFC 951, 1985
Sample BootP config
node01:ht=ether:ha=080009030166:ip=15.19.8.2:sm=255.255.248.0:gw=15.19.8.1:bf=/bootloader
node02:ht=ether:ha=080009030176:ip=15.19.8.3:sm=255.255.248.0:gw=15.19.8.1:bf=/bootloader
node03:ht=ether:ha=080009030186:ip=15.19.8.4:sm=255.255.248.0:gw=15.19.8.1:bf=/bootloader
node04:ht=ether:ha=080009030196:ip=15.19.8.5:sm=255.255.248.0:gw=15.19.8.1:bf=/bootloader
Another go around
Dynamic Host Configuration Protocol (DHCP).
RFC 1531, 1993.
Sample DHCP Configuration
subnet 192.168.0.0 netmask 255.255.255.0 {
range 192.168.0.10 192.168.0.49;
option subnet-mask 255.255.255.0;
option broadcast-address 192.168.0.255;
option routers 192.168.0.1;
filename "pxelinux.0";
next-server 192.168.0.100;
}
One more try?
In 1999 Intel releases the Wired for Management
Framework (WMF)
The framework was mostly ignored, but two
interesting parts of the specification endure.
Preboot Execution Environment (PXE).
Boot Integrity Services (BIS).
What is PXE?
PXE is a client designed to work with DHCP and
TFTP to retrieve boot strap information, it also
defines APIs to allow a loaded boot strap to query
a local host for additional information
What is BIS?
BIS enables a PXE client to examine a digitally
signed boot image. This provides a mechanisms to
verify the integrity of a supplied boot strap image.
How is PXE implemented?
If you want to use a network to provide
configuration information for a node, then you
better have a network card (NIC).
Since you must have a network card, it seems
reasonable to put the PXE client on the network
card, and have the bios treat network cards as a
bootable device.
Intel’s vision
What is a Network Bootstrap
Program(NBP)?



NBP is a binary executable file, specific to a give
CPU’s architecture.
They are small, usually less than 512KB in size.
What an NBP does is up to whoever creates it. The
PXE specification does not go into any detail on
NBP.
PXE Work flow with an NLB
What is involved in setting up a PXE
environment?



A Node with a PXE client set as the boot device.
A DHCP server
TFTP server
Sample DHCP Configuration
subnet 192.168.0.0 netmask 255.255.255.0 {
range 192.168.0.10 192.168.0.49;
option subnet-mask 255.255.255.0;
option broadcast-address 192.168.0.255;
option routers 192.168.0.1;
}
Sample proxy DHCP config
subnet 192.168.0.0 255.255.0.0
{
vendor pxe
{
bootstrapserver
192.168.0.100
# TFTP server ip address.
#Type, SystemArch, MajorVers
pxebootfile
1
2
1 window.one
1 0
pxebootfile
2
2
1 linux.one
2 3
pxebootfile
1
2
1 hello.one
3 4
client 6 10005a8ad14d
{
pxebootfile
1
2
1 aix.one
pxebootfile
2
2
1 window.one 6 7
}
}
}
5 6
What PXE does right




It is a standard feature on modern NICS, and
mother boards with integrated NICS
A PXE client knows the architecture of the node it is
running on, and can request an appropriate NBP
It builds upon existing technologies (DHCP, TFTP)
It has extensions to authenticate the boot server,
and verify the integrity of a downloaded NBP.
The problems with PXE




PXE has limited knowledge of it’s host.
PXE relies on DHCP, which is inefficient on large
networks
PXE relies on TFTP, which is impractical for large
files, or many concurrent file transfers
If you need different NBPs for different nodes you
must uniquely identify each node(By MAC address)
and group them accordingly in DHCP or Use a
DHCP proxy server and separate the management
of network address and images
Questions and Discussion
References
Thomas L. Sterling , 1998. Beowulf Cluster Computing with Linux.
Cambridge, Massachusetts: The MIT Press
Intel Corporation, 1998. Boot Integrity Services Application Programming. Version 1.0.
ftp://download.intel.com/design/archives/wfm/downloads/bisspec.pdf
Intel Corporation, 1999. Preboot Execution Environment (PXE) Specification. Version 2.1.
http://download.intel.com/design/archives/wfm/downloads/pxespec.pdf
http://www.ietf.org/rfc/rfc903.txt
http://www.ietf.org/rfc/rfc951.txt
http://www.ietf.org/rfc/rfc1531.txt
http://www.beowulf.org
http://en.wikipedia.org/wiki/Preboot_Execution_Environment
http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp