Interconnect simulation
Different levels for Evaluating an
architecture
• Numerical models
– Mathematic formulations to obtain performance
characteristics (close form or non-close form)
• Simulation
– Simulating the hardware/software in a program
• Emulation
• Prototype
• Operational system
Simulation
system boundary
exogenous inputs
to system
(the environment)
system under study
(has deterministic rules
governing its behavior)
“real” life
observer
program boundary
psuedo random inputs
to system
(models environment)
computer program
simulates deterministic
rules governing behavior
“simulated” life
observer
*Jim Kurose, University of Massachusets,
Amherst
Why simulation?
• Real-system not available, is complex/costly or
dangerous (eg: space simulations, flight
simulations)
• Quickly evaluate design alternatives (eg:
different system configurations)
• Evaluate complex functions for which closed
form formulas or numerical techniques not
available
Simulation advantages and limitations
• Advantages:
–
–
–
–
sometimes cheaper
find bugs (in design) in advance
generality: over analytic/numerical techniques
detail: can simulate system details at an arbitrary level
• Drawbacks:
– caution: does the simulation reflect reality?
– large scale systems: lots of resources to simulate (especially to
accurately simulate)
– may be slow (computationally expensive – 1 min real time could be
hours of simulated time)
– art: determining right level of model complexity
– statistical uncertainty in results
Types of simulators
• Time driven simulation
– Cycle/instruction based simulation
– Stop and observe the system after a fixed time
interval (N cycles, N instructions, etc)
– Events are assumed to happen at the time interval
boundary
• Event driven simulation
– Move from one event to another
– Not at a fixed interval
– Always simulate the first event in the future – this
event may trigger new events before the second
event.
Types of simulators
• For network simulation, it is more common to
use event driven simulation technique.
• Flow-level simulation (fluid model)
– How each flow (message) is transmitted
– Events: flow start, flow finish
• Both can affect the rates of the current flows.
• When there is no event, the system state does not
change, flows just follow the fluid model, which is
computed in the previous event.
– Topology and routing issues
Types of simulators
• Packet-level simulation
– How each packet is sent from the source to the destination
– Messages are chopped into packets
– The network is modeled as a network of queues; and packets move
from queue to queue.
• Processing node: send queue, receive queue
• Routers: queues for each port (input queue and output queue).
– Sample events: message arrival, message departure, packet arrival
(departure) to(from) a queue.
– Besides the topology and routing issues, packet level simulation can
also be used to study more detailed packet level issues such as packet
scheduling, congestion control mechanisms, switch latency, etc.
• Flit-level simulation
– How each flit goes through the network
– Similar to the packet-level simulation except the unit is flit: there are a
lot more flits than packets.
– Needed to study cut-through routing and wormhole routing.
Programming an event driven
interconnect simulator
• Define a set of events
– E.g. Message arrival, message departure, etc
• Each event will trigger the change of the state in the system
• The simulator simulates the system activities for
all events
– Change system states
• E.g. In a flow-level simulator, when a message starts
communicating, the throughput of other messages may be
affected because of sharing.
– Adding future events if necessary
• A message arrival will trigger packets being generated.
Programming an event-driven
simulator
• An event driven simulator must maintain the
following:
– simulated time: internal (to simulation program) variable that keeps
track of simulated time
– system “state”: variables maintained by simulation program define
system “state”
• e.g., may track number (possibly order) of packets in queue,
current value of retransmission timer
– events: points in time when system changes state
• each event has associate event time
– e.g., arrival of packet to queue, departure from queue
– precisely at these points in time that simulation must take
action (change state and may cause new future events)
• model for time between events (probabilistic) caused by external
environment
Event driven simulator block diagram
initialize event list
get next (nearest future)
event from event list
Advance time to event time
process event
(change state values, add/delete
future events from event list), etc
update statistics
n
done?
An example: a flow level network
simulator
• Input:
– a list of messages as well as their arrival time
• src_id, dst_id, msg_size, arrival_time
• Output
– the finishing time for each message
• Assumption: The topology and the routing scheme is
given.
– Topology: Link between switches and its bandwidth
– Routing: given a pair of src_id and dst_id, routing givens
the list of links along a path from source to destination.
An example flow level simulator
• Events
– Message arrival and message departure
• Modeling network activities
– Given a set of active flows, we must model the
transmission rates for each active flow.
• Need to make assumptions
• One possible assumption: the flow rate is equal to the
equal share of the link bandwidth for the most loaded
link along its path (remember Method 1 in the
modeling)
• Another possible assumption: MAX-MIN fairness.
An example flow level simulator
• Assumption 1: the flow rate is equal to the equal share of
the link bandwidth for the most loaded link along its path.
• Assumption 2: MAX-MIN fairness
• Active flows: 1->5, 2->3, 3->5, 4->5, link bandwidth 1Mbps
– Assumption 1: 1->5, 3->5, 4->5 333Mbps, 2->3, 500Mbps
– Assumption 2: 1->5, 3->5, 4->5 333Mbps, 2->3, 667Mbps
6
1
7
8
9
10
2
3
4
5
An example flow level simulator
• Network state:
– Active messages, sizes of remaining active messages,
rate for each active message, may be the finishing
time for each active message based on the current
data rate.
– When a new message arrives, the number of active
messages increases: the network state (flow rates)
needs to be recomputed
– When a message finishes, the number of active
message decreases: the network state (flow rates)
needs to be recomputed.
An example flow level simulator
initialize event list
get next (nearest future)
event from event list
time = event time
process event
(change state values, add/delete
future events from event list)
update statistics
n
done?
put all messages in a priority queue
get next (nearest future)
(first arrival time or finishing time)
time = event time
Change the set of active flows
Recompute the remaining message
size, new rate, and new finishing time
If message finish, output time/msg
n
done?
An example flow level simulation
• How accurate if the simulator?
– A lot of assumptions – this is common for all simulation type of work.
– If the assumption or the model is off, simulation produces junk results.
– Implementation details have significant impacts on the results
(numbers reported) – unknown simulator results are less credible.
• How to offset this?
– Use existing (well known) simulator, or simulation infrastructure.
– Be very careful in developing a new simulator.
Another example: a packet level
simulator
• Objective: to investigate different packet
scheduling algorithms used in routers.
• Input:
– a list of messages as well as their arrival time
• src_id, dst_id, msg_size, arrival_time
– The messages can be generated following some
distribution – to evaluate the performance under
different traffic conditions.
• Output
– the finishing time for each message
An example packet level simulator
• Events:
– Message arrives a compute node, packet at a
compute node, packet arrives at an input port,
packet moves to an output port, etc.
• Model system activities
– Modeling the compute node behavior
– Modeling router behavior
An example packet level simulator
• Model system activities
– Modeling the compute node behavior
• When a message arrives, packetize the message, insert
packets into a host packet queue (with ready time) to
be injected to the network.
• When a packet is received, if it is the last packet of a
message, keep the statistics.
An example packet level simulator
– Modeling router behavior
• Each input port can be modeled by an input queue
• Each output port can be modeled by an output queue
• Packet scheduling algorithm: when multiple packets are
ready at the input queue, the algorithm determines which
packet gets to be scheduled to the output queue.
• Modeling router latency: after a packet is removed from an
input queue, insert the packet to the output queue with a
time stamp equal to current_time + latency
– Modeling link latency?
An example packet level simulator
• Network states:
– Queues on routers and end hosts, message at end
hosts
• The simulator basically figures out the next future
event, and perform the event
– A message arrives at end host: packetize the message
– A packet is ready on end host: move the packet
– A packet is ready on a queue: move the packet
An example packet level simulator
initialize event list
get next (nearest future)
event from event list
time = event time
process event
(change state values, add/delete
future events from event list)
update statistics
n
done?
Packet level .vs. flow level simulation
• Flow level simulation has higher level of
abstraction – more details are lost.
• Is flow level simulation always more efficient?
– Not necessary, rate calculation can be quite
compute-intensive.
Traffic generation issue
• Random uniform traffic, traffic arrival follows a
Poisson process.
– In one time slot, each SD pair has equal probability to
have a message.
– Cycle level simulation: use a random number trying to
generate traffic in every cycle.
• Each cycle each SD pair has 0.1% change of having a traffic.
– Event driven simulation:
• Determine the time when a message is supposed to be
generated.
– On average 1 message in 1000 cycles. The next message time is
exponentially distributed with an average of 1000.
» Once the time is determined, generate a random SD pair.
Network simulation packages
• For network simulation, the driver is quite standard.
– Difference in the modeling techniques
• Quite some network simulators are available:
– NS2, OpNet, Mininet, Booksim, etc
– Provide many common library funcationality
– Allow build the network topology and routing schemes.
• When to use the existing simulators and when to write
one from scratch?
– Decide whether existing one can do what you want to do.