Generic Warfighter’s Information Network (WIN) Components Parallel Simulation of Large-Scale Heterogeneous Communication Systems UAV Network PI: Rajive Bagrodia rajive@cs.ucla.edu How does the network perform as it is scaled to 100,000+ heterogeneous devices? Senior Dev Engr: Dr. Mineo Takai mineo@cs.ucla.edu Computer Science Department UCLA OSPF, LANDMARK, or DAWN, routing? Partial support from DARPA DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu • • • • • • • DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu Project Accomplishments Technology Transfer Design & development of GloMoSim framework with rich protocol stack Demonstrated substantially superior sequential performance compared to existing alternatives (2-5x faster) Demonstrated further improvement with parallel execution (up to 10x) Demonstrated scalability of GloMoSim using very highfidelity models with a complete protocol stack to networks with 50,000+ devices; Demonstrated feasibility of real-time simulation of networks with 100s of nodes Demonstrated hybrid simulations with integration of real applications running with virtual protocol stack. Direct comparison of alternative unicast and multicast wireless protocols for GloMo scenarios DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu • • • • DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu Scalable Simulation Technology • GloMoSim and PARSEC integrated into SEAM-LSS, a DARPA-funded M&S environment developed by SAIC GloMoSim commercialized by Scalable Simulation Solutions Commercial version of GloMoSim being used in M&S study for the JTRS program Wide distribution (close to 3000 downloads) of public domain simulation software GloMoSim Library Efficient and high-fidelity simulations via parallel execution on diverse parallel architectures (PARSEC ) • PARSEC PARSEC(C-Based) (C-Based)Front-End Front-End Portable PortableMulti-threaded Multi-threadedCommunication CommunicationLibrary Library (xsend, (xsend,xrec, xrec,etc...) etc...) MPI/AIX MPI/AIX MPI MPICH/ CH/ BSD BSDUnix Unix Pthreads Pthreadson onWindows WindowsNT, NT, Linux, Linux,Solaris, Solaris,IRIX IRIX Data Plane Linux, Linux, Windows Windows NT, NT, Unix Unix • • • • IBM SP • PC Network Dell SMP, Sun Sparc 1000; SGI Origin 2000 Uniprocessor Machine • Modular, extensible library for network models Model each layer using abstract or detailed model Built-in statistics collection at each layer Customizable GUI Large and growing model library worldwide installed base of users Modular and composable library of parallelized models with standard APIs for end-end models (GloMoSim) DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu Page 1 Application Processing Application RTP Wrapper TCP, UDP, RSVP IP OSPF, AODV, … Transport IP Network IEEE 802.11, 802.3, … Link Layer MAC Layer EPLRS, WaveLAN, ... Radio Free space, TIREM Propagation model Packet Store/Forward Glomosim Standalone • • • • • • • • Application: Replicated file system, ftp, telnet, cbr, web caching, NetMeeting, WebPhone, synthetic traffic generators Transport : TCP(FreeBSD), NS TCP (Tahoe), UDP, DBS satellite models, Multicasting: ODMRP, CAMP, AMRIS, AMRoute, AST, DVMRP Routing: Distributed Bellman-Ford, Flooding, Fisheye, DSR, DSDV, WRP, LAR, NS-DSDV, DREAM, MMWN MAC: CSMA, IEEE 802.11, MACA-W, Radio: DS SS with and without capture Propagation: analytical (free space, Rayleigh, Ricean), 2-ray ground reflection model, path loss trace files Mobility: random waypoint, trace files DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu GloMoSim Path Loss Models Critical for accurate wireless network simulations • • • • • • • Memory consumption [MB] 500 Scalability 400 • 300 200 100 8 16 No. of processors 0 10000 20000 30000 40000 50000 Real-Time 2.5 50 2 100 1.5 200 1 0.5 0 50 nodes 100 nodes 500 nodes 1000 nodes 0 1 2 3 4 5 6 -40 -50 -60 NS RX limit (-64.3) -70 GloMoSim prop limit (-76.0) -80 Backgroundnoise Ambient noise -100 Thermal noise (-100.9) New GloMoSim prop limit -110 -120 NS GloMoSim 1.x Opnet / GloMoSim 2.x 0 10000 20000 30000 40000 50000 • The packet delivery ratio decreases gradually as the CBR traffic increases. The end-to-end delay is more adversely affected by heavier traffic than the packet delivery ratio due to many retransmission, but the major loss of packets is derived from the network queue overflow (50 tail drop), not from IEEE 802.11 retransmission limits. 4 0.6 0.4 0.2 0 Packet drops per session 0.8 End-to-end delay [s] Packet delivery ratio 300 5 What causes this increase? 3 2 1 150 200 250 300 Number of CBR sources DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu 250 200 802.11 retx limit Queue overflow 150 100 50 0 0 100 NS CS limit (-78.0) -90 Large-scale Simulation Results (2) • 1,000 network nodes on a flat terrain (density of 20,000m2/node) 376m boundary radio model (from the WaveLAN specification) with detailed SIR (signal to interference) calculation IEEE 802.11 DCF with RTS / CTS option; LAR (Location Aided Routing Protocol) scheme 1 ad hoc wireless routing 100 to 300 CBR sources with 4 packets/s for randomly selected destinations (about 6 hops away) 1 50 100000 DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu Network analysis using large-scale Simulations • 10000 Number Network Nodes DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu • 5000 1200 1000 800 600 400 200 0 Packets processed per node [/s] • • 1000 Simulation of wireless networks with full protocol stack (density of 20,000m 2 per node, free space, 250m boundary radio model, IEEE 802.11 DCF, AODV, UDP, 10% nodes have CBR traffic with 4 packet per second) Memory Consumption [MB] 1 4 550 Simulation Scalability 600 3 2 250 -30 SIRCIM (topography, building type) GloMoSim 2.x includes all the above. 3.5 3 1 100 Number of network nodes Parallel Execution 5 86.1 DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu 0 Sim / real time ratio Speedup 7 10 -20 ? : std dev for log normal shadowing GloMoSim Unique Features: • Scalability to very large (wireless) networks • Efficiency via transparent support for parallel execution • Potential for real-time simulation of networks Distance [m] Free space Abstracted two-ray ground reflection(NS-2) Trace based (path loss - distance) Generic (n, ? ) • n: path loss exponent Rx Power [dBm] Models currently available in GloMoSim 50 100 150 200 250 50 300 100 150 200 Number of CBR sources Number of CBR sources DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu Page 2 250 300 Models available in QualNet 1.0 GloMoSim & QualNet • • wireless • GloMoSim: library for mobile ad hoc networks developed as a research tool at UCLA QualNet: Wired & wireless network modeling library commercialized by Scalable Simulation Solutions (SSS) • GUI for experiment design, animation, protocol model design • Larger model library: wired, wireless, QoS • Built in statistics collection and analysis capabilities • Application level performance prediction • Technical support, maintenance & training • For information on QualNet: info@scalable-solutions.com • • • • • • • DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu 32.3705 SEAM-LSS Integration 32.37 Developing a complete analysis capability for military comm needs (in partnership with Telcordia/SAIC) Latitude (degrees) • Application: ftp,telnet,cbr, Tcplib, NetMeeting, WebPhone, MODSAF, SEAM-LSS, synthetic traffic, self-similar traffic with long range dependency Transport : TCP (FreeBSD), UDP, RTP, RSVP, MPLS, DiffServ Multicasting : ODMRP, PIM Routing: Distributed Bellman-Ford, OSPFv2, RIPv2, BGP, Flooding, Fisheye, DSR , DSDV, WRP, LAR, AODV MAC: CSMA, IEEE 802.11, IEEE 802.3 Physical : point-point link, wired bus, IEEE 802.11 DSSS radio Propagation : analytical(free space, Rayleigh, Ricean), TIREM, 2-ray ground reflection model, path loss trace files Mobility : random waypoint, MODSAF, SEAM-LSS, trace files SEAMLSS Results 32.3695 Completed PARSEC McKenna 11-node DAWN Latitude vs. Simulation Time 32.369 32.3685 32.368 32.3675 0 100 300 400 500 600 700 100 Thread Completion Time (s, log scale) QualNet Models 200 800 900 1000 Simulation Time (seconds) Mobility Scenarios Scenarios Simulation Realistic Propagation Models Communication Threads PARSEC McKenna 11-node DAWN Thread Completion Time vs. Thread Start Time Number of Completions: 371 10 1 Thread Instance 0.1 0.01 0 100 200 DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu • 100 10 Min Mean Max • 0.01 0.001 0.0001 • SIT RE P SL SIT to PL RE P PL LO GR to CC EP LO S GR Lt oP EP S PS Se to ns 1S itiv GT e Re Se p ns SL itiv to e Re PL p PL Sta tus to CC Re Sta pS tus Lt oP Re G pP Off G to en Co 1S siv nd GT euc Off on ta tta en the c siv kc m e ov om -o eP Off p n en L to the siv SL em ov on Off e the SL en siv m to ov e FT -o eF Off L n TL en the siv to m SL e ov -o e n PL De the fen to m siv SL ov ee 2 De sta RT fen tion O siv ary to e PL PL -s to tati RT on O ar yP L to RT O 2 Min, Mean, & Max Thread Comp. Time (sec, log scale) PARSEC GloMo 78-node DAWN Min, Mean, & Max Thread Completion Time vs. Thread Type 1 500 600 700 800 900 ModSAF (Modular Semi-Automated Forces) models munitions, group movement behavior • ModSAF supports HLA through a DIS/HLA gateway • GloMoSim, being written in PARSEC, supports HLA extensions HLA Interactions between MODSAF & GloMoSim: • ModSAF sends unit positions through HLA • GloMoSim receives position updates, computes signal transmission based on new positions HLA and sfdsimulator interfaces from GloMoSim have been integrated with MODSAF 5.1 Thread Type DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu 400 Thread Start Time (seconds) GloMoSim and ModSAF 5.0 Cosimulation SEAMLSS Results 0.1 300 DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu Page 3 1000 ModSAF, DIS/HLA, Intermediate Federate, GloMoSim Execution Constraints • ModSAF position updates are real-time, while GloMoSim/PARSEC is a DES • an intermediate PARSEC federate was created between the gateway and GloMoSim DIS-HLA Gateway RO MODSAF Real time DIS-HLA IF Intermediate Federate (IF) Time Regulated MODSAF GloMoSim GloMoSim Time Constrained RTI DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu Co-Simulation • Validation Using Emulation • Interfaces to support interoperability of OPNET and GloMoSim models using HLA and modified RPR-FOM Heavy traffic using FTP transferring a 10MByte file in a wireless Wavelan network over 802.11 (with RTS/CTS) using a 2Mbit/s link Same scenarios in both real network and hybrid network with a real FTP client and server 0 Ftp 1 Distance between nodes is 1m • • OPNET 80 0.5 40 0.4 1 2 3 4 5 Scenario No. Gateway 2 Scenario5 120 0 Gateway Ftp 3 160 Real Network Hybrid Network Loss Rate(%) Throughput(Kbyte/sec) DAWN subnets in PARSEC/SEAMLSS Ftp Ftp 200 0.3 0.2 0.1 0 1 2 3 4 5 Scenario No. DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu Technology Transfer Selected Users Users by Platform 19% 2% 2% 1% 2% 36% 21% 12% • • • • 0% 1% 0% 0% Redhat Linux Other Linux Solaris Windows 95/98 Windows NT SunOS FreeBSD • Other PC Solaris HPUX Irix Macintosh OSF • 1% 5% Over 1775 PARSEC and/or GloMoSim downloads Mar 00-July 00 • Over 900 PARSEC/GloMoSim downloads Nov ‘99-- Feb 00 • http//pcl .cs.ucla.edu/projects/parsec Second Parsec workshop held Nov 11 & 12, 1999 • • http//pcl .cs.ucla.edu/projects/parsec/workshop99 Over 50 attendees including commercial, military, universities Integrated into SEAM LSS: http://www.seamlss .com Commercialization via Scalable Simulation Solutions DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu Government/Military: MITRE, Lawrence Livermore National Labs*, FAA, Jet Propulsion Lab, NASA *, MIT Lincoln Laboratory, Space and Naval Warfare System Center (SPAWAR)*, … Corporations : Cisco Systems, Fujitsu Laboratories, General Dynamics*, Philips Research, Lockheed Martin, Lucent Technologies*, Motorola*, NEC*, Nortel Networks, Nokia • Research Center*, Oracle Telecomputing, Primeon Inc.*, SAIC*, SRI International*, … Universities (US): Boston University*, Caltech*, Cornell, • International Sites: AT&T (UK), CSIRO (Australia), NATO Dartmouth, UC Berkeley, UCLA*, University of Texas*, USC*, … SACLANT Undersea Research Centre, Italy; Technion, Israel; University of Aizu , Japan; VTT Electronics, Finland; … DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu Page 4 Multicast Protocol Performance • ODMRP (UCLA) • Creates a mesh of nodes (the forwarding group) to provide redundant multicast routes • on-demand technique to establish route/membership CAMP (UCSC) • Creates a shared mesh • requires underlying unicast protocols (e.g., WRP) AMROUTE (Telcordia) • Creates bidirectional shared multicast tree • Uses virtual mesh links to establish the multicast tree AMRIS (NUS, Singapore) • Creates a shared tree and uses ranking to direct the flow of multicast data Flooding • Selected Case Studies • • • DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu Forwarding Group Concept Multicast Protocol Comparisons A set of nodes in charge of forwarding multicast packets Supports shortest paths between any member pairs Mesh topology and flooding help overcome displacements and channel fading • Configuration: • 50 nodes placed randomly in 1000m x 1000m area • Capture Radio; power of 250 m; Bandwidth: 2 Mbps • MAC: IEEE 802.11 DCF • Traffic: CBR with payload size 512 bytes Metrics: • Packet delivery ratio; • control overhead Independent variables: • Mobility • Network traffic load • Multicast group size • No. of senders Paper presented at Infocomm 2000 (Lee et al): http://pcl .cs.ucla.edu/papers • • • DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu Multicast Performance with Mobility Multicast Performance with Mobility Packet Delivery Ratio (PDR) • • • • 20 multicast members 5 sources transmit packets at the rate of 2 pkt/sec each Mobility Speed: 0-72 km/hr PDR : fraction of packets actually received by intended recipients. Control Overhead 1.0 35 0.9 30 0.8 25 0.7 ODMRP FLOOD CAMP 0.6 0.5 15 AMROUTE AMRIS 0.4 10 0.3 5 0.2 0 0.1 0 0.0 10 20 30 40 50 60 70 Mobility Speed (kph) 0 10 20 30 40 50 60 70 1.2 1.0 0.8 0.6 0.4 0.2 0.0 0 10 20 30 40 50 Mobility Speed (kph ) 60 70 Mobility Speed (kph) • Mesh-based (CAMP, ODMRP, flooding ) do better than tree based (AMRIS, AMROUTE) • Good delivery ratio in ODMRP due to multiple redundant routes •CAMP degrades due to poor pkt delivery to distant routers (these have fewer redundant paths); WRP loop detection can temporarily mark node subsets as unreachable, postponing rote updates for mesh maintenance. DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu ODMRP FLOOD CAMP AMROUTE AMRIS 20 Control Bytes Transmitted • • • DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu • • • Control overhead: no. of control pkt bytes + header size in data packets AMRIS is low due to very low delivery ratio; AMROUTE high due to loops CAMP has higher overhead than ODMRP due to trigerred updates in WRP, particularly with high mobility. DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu Page 5 TCP and MAC Interactions Evaluate MAC interaction with TCP in presence of mobility. 0 10 m 1 2 10 11 17 18 19 20 26 72 73 74 • • with mobility: Bellman-Ford with routing table updates every second. 3 horizontal (18-26; 36-44; 54-62) and 3 vertical (2-74; 4-76; 6-78) end-end FTP connections. • • without mobility : static routing • WMSCA ‘99 (Gerla, Bagrodia, Tang): http:// pcl.cs.ucla.edu/papers DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu • • • Study performance issues of (ad hoc) wireless networks using real applications Importance of abstract vs. detailed network models Efficient simulation of large scale models via parallel execution • DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu Replicated File System When are detailed models of the protocol stack necessary for studying application performance? FAMA 802.11 Throughput (bps) 36-44 100000 54-62 50000 0 6-78 2-74 CSMA FAMA 802.11 MAC Protocol 4-76 6-78 • • Link level ACKs help recover from loss caused by transient nodes. Capture exists. Conclusion • Link-level ACKs important to combat packet loss in wireless ad-hoc environment. Faster update dissemination Better adaptation to dynamic network topologies Replicated File System: Results Reconcilitaion behavior as a function of MAC protocol & mobility speed Abstract models may be used only in absence of mobility Globecomm ’99: Ahuja et al: http://pcl .cs.ucla.edu/papers Reconciliation time [milli sec] Stale read/write rate: No. of read/write access to data that has since been modified by another replica • CSMA 4-76 18-26 150000 Without mobility • CSMA performs poorly due to interference by neighboring and intersecting streams. • FAMA fair due to RTS/CTS and less aggressive yield time. • 802.11 exhibits capture. With mobility • CSMA and FAMA collapse due to lack of fast loss recovery facilities. • 802.11 still operational. • • • • replica generates a reconciliation request to when the reconciliation completes. Frequency of reconciliation? 2-74 RRFS shares data through peer replication • Every unit gets its own copy of the data • Every unit can make updates to its copy Use periodic update propagation for data reconciliation Use opportunistic update propagation between any replicas Contrast with client server architecture • • Average Reconciliation time: Time from when a • • 200000 DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu Application performance Metrics: Scalability of design with no. of replicas, nodes, traffic, deployment area, …? DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu 54-62 36-44 The Replicated File System (RRFS): Distributed Data Replication Application Performance in AdHoc Networks • 100000 50000 DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu • • • • 18-26 MAC Protocol 80 Mobility: 10 meters per second in a random direction with a probability of 0.5. Routing: • 200000 150000 0 81 nodes; radio range : 30m; bandwidth: 2Mbps • • • 250000 8 9 9 X 9 Grid Experiments with Mobility 9 X 9 Grid Experiment with No Mobility Throughput (bps) • TCP/MAC Performance 800000 600000 CSMA MACA 400000 FAMA 802.11 200000 0 0 2 4 Mobility [kmph] DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu Page 6 6 8 Replicated File System: Results Replicated File System: Results Impact of transmit power on recon time • • • Simulation of replication service with a detailed stack model: TCP, Bellman Ford, CSMA, radio • • Topology: 20 mobile nodes; 6 Rumor nodes; ring topology. Reconcilitation interval: 4 hours • Impact of varyingTCP window size from 1 to 32 packets Increasing window size causes more collissions between data packets and ACKs travelling in opposite directions Again, difference with mobility is much more than no mobility Abstract models may have errors upto 400% in presence of mobility. Reconciliation time(s) • Reconciliation time 2500000 2000000 50m power 1500000 40m power 1000000 30m power 800 600 1 packet 400 32 packets 200 0 500000 0 2 4 6 8 Mobility(km/hr) 0 0 2 4 6 8 Mobility (kmph) DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu Scalability via Parallel Execution 1 5 n Rumor Servers in Ring Topology 7 8 3 4 Ring Topology 9 60 Servers 120 Servers 4 Speedup 4 3 2 1 3 2 2 3 4 5 6 7 8 1 Number of Processors 2 3 4 5 6 7 Accomplishments • • • 0.02 0.01 Design & development of GloMoSim framework for detailed simulation of networks with tens of thousands of nodes. Demonstrated hybrid simulations with integration of real applications running with virtual protocol stack. Direct comparison of alternative unicast and multicast wireless protocols for GloMo scenarios Design of scaleable unicast & multicast wireless protocols GloMoSim and PARSEC integrated into SEAM-LSS GloMoSim commercialized by Scalable Simulation Solutions Commercial version of GloMoSim being used in M&S study for JTRS program Wide distribution (close to 3000 downloads) of public domain simulation software DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu Page 7 0.01 0 40 60 80 100 120 DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu Conclusion 0.02 0.015 0.005 Number of Processors Technology Transfer: • • • 0.03 Number of Servers 8 DOMAINS; July 2000; R. Bagrodia; rajive@cs.ucla.edu • 0.04 0 1 0 1 • 0.025 20 0 Ring Toplogy 0.05 Rumor Servers in Tree Topology 120 Servers 5 Speedup 6 2 Scaling Replicas Consider a set of servers in ring topology with reconciliation interval of four hours. Stale Write Rate 0 60 Servers • 0 3 Stale Read Rate 2 1 20 40 60 80 100 Number of Servers 120