Presented by: Quinn Gaumer
CPS 221




16,384 Processing
Nodes (32 MHz)
30 m x 30 m
Teraflop
1992


With 16,384 processors the interconnect
plays a large role
3 Types of Networks
◦ Data
◦ Control
◦ Diagnostic







Easily Attainable High Performance
Scaling
Data Parallel Programming
High Reliability and Availability
Space/Time Shared
Fast Time to Market
Modular
Include
 Control Processor
 Processing Nodes
 Slices of Data and Control Networks
◦ Privileged vs. Non-Privileged


Program Isolation
Time Sharing



Provide Simple View of Network to Processors
Sharing and Fault Tolerance
Decouple Network/Processor by Providing
Contract
◦ Software -> ISA -> Hardware

“The data network promises to eventually
accept and deliver all messages injected into
the network by the processors as long as the
processors promise to eventually eject all
messages from the network when they are
delivered to the processors. ”

Collection of Memory Mapped FIFOs
◦ Outgoing/Ingoing

Restricted Operations
◦ Implemented with protected pages

Physical/Relative(Virtual) Address
◦ Programs use only relative addresses

Network Independent of User
◦ Delivery guaranteed by network not processing
node
◦ Requires network diagnostics

Fat Tree Structure
◦ Closer to the root, thicker the tree
◦ Ensures no bottlenecks at root

User Partitions and I/O are
Sub-trees
◦ Guarantees network
independence
◦ Messages in partition stay within
partition

Many Optimal Node to Node
Paths
◦ Choose randomly among open
links



Data can be only 1-5 Words
Wormhole Routing
CRC Checking done at every Link
◦ Additional !CRC sent when error first found

Primary Errors allow Diagnostic Network to
Determine location



Message Counters at every Link
Kirchoff’s Law to Determine Missing
Messages
What to do with a Bad Chip or Link?
◦ Route Messages Away from Failure
◦ Map Out Nearby Processors
◦ Which is better?
 Both.

Solution: Virtual Channels
◦ One channel for request and response
◦ 4 channels per chip (Incoming and Outgoing)

Deadlock still possible!
◦ User sends but never attempts to receive messages
◦ Higher level languages to implement
communication protocol

Objectives
◦ Clear all messages for new user
◦ Allow all messages in transit to
eventually finish

“All Fall Down” Method
◦ Evenly misroute all messages in
transit to nodes
◦ Message saved at node
◦ Resent when swapped in

Control Processor broadcasts program
◦ Not instructions(SIMD)


Each Processor runs program on data set
Inter-Processor Communication
◦ Hardware Barriers allow for processes to
communicate without shared semaphores

Program smaller than instructions
◦ Easier to deliver

Local fetch allows commodity processors
◦ Fast new RISC processors, less R & D.


Control system useful for other problems
Execution of generic MIMD code
◦ Message passing

Broadcasting
◦ User/Supervisor
◦ Interrupt
◦ Utility

Combining
◦ Reduction
◦ Forward/Backward Scan
◦ Router Done

Global Operations
◦ Synchronous/Asynchronous OR


Binary Tree
Four Types of Packets
◦
◦
◦
◦

Single Source : Broadcasting
Multiple Source: Combining
Idle: Filler
Abstain: Allow control node to skip waiting
Collisions on Network
◦ Multiple/Multiple: Buffering based on arrival time
◦ Multiple/Single: Single Source Packets Prioritized
◦ Single/Single: Error

Control Processor for each Partition
◦ Executes scalar code while processing nodes
execute parallel code

Connect any Control Processor to any
Partition
◦ Problems can occur in control networks too
◦ Diagnostics may show part of control network must
be mapped out

Binary Network
◦ Pods(physical subsystem)
are leaves

JTAG
◦ Designed for Multichip…but
serial


Do JTAG for each Pod
Combine Responses with
OR/AND