An Overview of Myrinet
By:
Ralph Zajac
What is Myrinet?
LAN designed for clusters
 Based on USCD’s ATOMIC LAN
 Has many characteristics of MPP
message-passing networks

MPP Network Characteristics
High data rate
 Regular topology and scalability
 Very low error rate
 Cut-through routing
 Flow control

ATOMIC LAN: Overview
Based on technology from Caltech’s
Mosaic project
 Used TCP/IP at burst rates of 400 Mb/s
 Error rate less than 10-15

ATOMIC LAN: Limitations


Asynchronous communication could lead to
deadlock
Network topology
– Complex
– Hidden computing power


Performance limited by lack of DMA in host
interface
End-to-end data rates limited by TCP/IP stack
of OS
Myrinet
Designed to address the limitations of
ATOMIC LAN
 Not limited to an existing MPP network
 Specifications published in ANSI/VITA
26-1998

Myrinet: Links




Can be copper (up
to 25m) or opticalfiber
Full duplex
Uses non-return-tozero encoding
Flow control run by
a slack buffer
STOP
GO
Slack Buffer
Myrinet: Packets and Routing
Leading byte determines the outgoing
port
 MSB of each header byte distinguishes
between “host” and “switch” packets
 Variable payload length
 8-bit CRC

Myrinet: Switches
Use blocking-cut-through routing
 Each contains two custom VLSI chips

– crossbar-switch
– dual interface

Topology is arbitrary
Myrinet: Host Interface
Available for PCI and Sbus
 32-bit SRAM for MCP
 Can act as a slave or a bus master

Myrinet: Software

MCP
– Handles all interfacing with the network
– Continuously mapping and monitoring the
network

Host Software
– TCP/IP and UDP/IP
– Myrinet API
High Performance Virtual Machine




Windows NT 4.0 cluster built at UCSD and
UIUC for the NCSA
96 dual processor nodes
Used a fat-tree network with 8-port Myrinet
switches
Was a 56 GFlops system with:
– 45 GB of DRAM
– 1.6 MB/s bisection bandwidth
– 400 GB of disk storage
HPVM: Testing

Tested on four programs
– ZeusMP: astrophysical phenomena in 3D
– Cactus: modular 3D numerical relativity
– AS-PCG kernel: linear systems solver
– QMC kernel: Quantum Monte Carlo
simulator

Compared with a Cray T3E and an
Origin 2000
HPVM: Results

Performed from 2 to 4 times slower
– Slower floating point calculations
– Flow control limitations of Myrinet

Much better price/performance ratio
Conclusions

Myrinet is:
– Flexible
– Fast

Affordable systems with acceptable
performance