On-line Automated Performance Diagnosis on Thousands of Processors Philip C. Roth Future Technologies Group Computer Science and Mathematics Division Oak Ridge National Laboratory Paradyn Research Group Computer Sciences Department University of Wisconsin-Madison OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 1 High Performance Computing Today Large parallel computing resources Tightly coupled systems (Earth Simulator, BlueGene/L, XT3) Clusters (LANL Lightning, LLNL Thunder) Grid Large, complex applications ASCI Blue Mountain job sizes (2001) 512 cpus: 17.8% 1024 cpus: 34.9% 2048 cpus: 19.9% Small fraction of peak performance is the rule OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 2 Achieving Good Performance Need to know what and where to tune Diagnosis and tuning tools are critical for realizing potential of large-scale systems On-line automated tools are especially desirable Manual tuning is difficult Finding interesting data in large data volume Understanding application, OS, hardware interactions Automated tools require minimal user involvement; expertise is built into the tool On-line automated tools can adapt dynamically Dynamic control over data volume Useful results from a single run But: tools that work well in small-scale environments often don’t scale OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 3 Barriers to Large-Scale Performance Diagnosis • Managing performance data volume • Communicating efficiently between distributed tool components • Making scalable presentation of data and analysis results Tool Front End Tool Daemons d0 d1 d2 d3 dP-4 dP-3 dP-2 dP-1 App Processes a0 a1 a2 a3 aP-4 aP-3 aP-2 aP-1 OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 4 Our Approach for Addressing These Scalability Barriers MRNet: multicast/reduction infrastructure for scalable tools Distributed Performance Consultant: strategy for efficiently finding performance bottlenecks in large-scale applications Sub-Graph Folding Algorithm: algorithm for effectively presenting bottleneck diagnosis results for large-scale applications OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 5 Outline Performance Consultant MRNet Distributed Performance Consultant Sub-Graph Folding Algorithm Evaluation Summary OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 6 Performance Consultant Automated performance diagnosis Search for application performance problems Start with global, general experiments (e.g., test CPUbound across all processes) Collect performance data using dynamic instrumentation Collect only the data desired Remove the instrumentation when no longer needed Make decisions about truth of each experiment Refine search: create more specific experiments based on “true” experiments (those whose data is above userconfigurable threshold) OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 7 Performance Consultant c001.cs.wisc.edu c002.cs.wisc.edu c128.cs.wisc.edu myapp367 myapp4287 myapp27549 OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 8 Performance Consultant CPUbound main Do_row Do_col c001.cs.wisc.edu c002.cs.wisc.edu c128.cs.wisc.edu myapp367 myapp4287 myapp27549 … c001.cs.wisc.edu c002.cs.wisc.edu myapp{367} myapp{4287} myapp{27549} main main main Do_mult … Do_row Do_col Do_mult Do_col Do_mult … Do_row … Do_col Do_mult … … … OAK RIDGE NATIONAL LABORATORY Do_row … c128.cs.wisc.edu U. S. DEPARTMENT OF ENERGY 9 Performance Consultant cham.cs.wisc.edu CPUbound main Do_row Do_col c001.cs.wisc.edu c002.cs.wisc.edu c128.cs.wisc.edu myapp367 myapp4287 myapp27549 … c001.cs.wisc.edu c002.cs.wisc.edu myapp{367} myapp{4287} myapp{27549} main main main Do_mult … Do_row Do_col Do_mult Do_col Do_mult … Do_row … Do_col Do_mult … … … OAK RIDGE NATIONAL LABORATORY Do_row … c128.cs.wisc.edu U. S. DEPARTMENT OF ENERGY 10 Outline Performance Consultant MRNet Distributed Performance Consultant Sub-Graph Folding Algorithm Evaluation Summary OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 11 MRNet: Multicast/Reduction Overlay Network Parallel tool infrastructure providing: Scalable multicast Scalable data synchronization and transformation Network of processes between tool front-end and back-ends Useful for parallelizing and distributing tool activities Reduce latency Reduce computation and communication load at tool front-end Joint work with Dorian Arnold (University of Wisconsin-Madison) OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 12 Typical Parallel Tool Organization Tool Front End Tool Daemons d0 d1 d2 d3 dP-4 dP-3 dP-2 dP-1 App Processes a0 a1 a2 a3 aP-4 aP-3 aP-2 aP-1 OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 13 MRNet-based Parallel Tool Organization Tool Front End Internal Process Filter Multicast/ Reduction Network Tool Daemons d0 d1 d2 d3 dP-4 dP-3 dP-2 dP-1 App Processes a0 a1 a2 a3 aP-4 aP-3 aP-2 aP-1 OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 14 Outline Performance Consultant MRNet Distributed Performance Consultant Sub-Graph Folding Algorithm Evaluation Summary OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 15 Performance Consultant: Scalability Barriers MRNet can alleviate scalability problem for global performance data (e.g., CPU utilization across all processes) But front-end still processes local performance data (e.g., utilization of process 5247 on host mcr398.llnl.gov) OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 16 Performance Consultant cham.cs.wisc.edu CPUbound main Do_row Do_col c001.cs.wisc.edu c002.cs.wisc.edu c128.cs.wisc.edu myapp367 myapp4287 myapp27549 … c001.cs.wisc.edu c002.cs.wisc.edu myapp{367} myapp{4287} myapp{27549} main main main Do_mult … Do_row Do_col Do_mult Do_col Do_mult … Do_row … Do_col Do_mult … … … OAK RIDGE NATIONAL LABORATORY Do_row … c128.cs.wisc.edu U. S. DEPARTMENT OF ENERGY 17 Distributed Performance Consultant cham.cs.wisc.edu CPUbound main Do_row Do_col c001.cs.wisc.edu c002.cs.wisc.edu c128.cs.wisc.edu myapp367 myapp4287 myapp27549 … c001.cs.wisc.edu c002.cs.wisc.edu myapp{367} myapp{4287} myapp{27549} main main main Do_mult … Do_row Do_col Do_mult Do_col Do_mult … Do_row … Do_col Do_mult … … … OAK RIDGE NATIONAL LABORATORY Do_row … c128.cs.wisc.edu U. S. DEPARTMENT OF ENERGY 18 Distributed Performance Consultant: Variants Natural steps from traditional centralized approach (CA) Partially Distributed Approach (PDA) Distributed local searches, centralized global search Requires complex instrumentation management Truly Distributed Approach (TDA) Distributed local searches only Insight into global behavior from combining local search results (e.g., using Sub-Graph Folding Algorithm) Simpler tool design than PDA OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 19 Distributed Performance Consultant: PDA cham.cs.wisc.edu CPUbound main Do_row Do_col c001.cs.wisc.edu c002.cs.wisc.edu c128.cs.wisc.edu myapp367 myapp4287 myapp27549 … c001.cs.wisc.edu c002.cs.wisc.edu myapp{367} myapp{4287} myapp{27549} main main main Do_mult … Do_row Do_col Do_mult Do_col Do_mult … Do_row … Do_col Do_mult … … … OAK RIDGE NATIONAL LABORATORY Do_row … c128.cs.wisc.edu U. S. DEPARTMENT OF ENERGY 20 Distributed Performance Consultant: TDA cham.cs.wisc.edu c001.cs.wisc.edu c002.cs.wisc.edu c128.cs.wisc.edu myapp367 myapp4287 myapp27549 … c001.cs.wisc.edu c002.cs.wisc.edu myapp{367} myapp{4287} myapp{27549} main main main Do_row Do_col Do_mult Do_col Do_mult … Do_row … Do_col Do_mult … … … OAK RIDGE NATIONAL LABORATORY Do_row … c128.cs.wisc.edu U. S. DEPARTMENT OF ENERGY 21 Distributed Performance Consultant: TDA cham.cs.wisc.edu c001.cs.wisc.edu c002.cs.wisc.edu c128.cs.wisc.edu myapp367 myapp4287 myapp27549 … c001.cs.wisc.edu c002.cs.wisc.edu myapp{367} myapp{4287} myapp{27549} main main main … c128.cs.wisc.edu Sub-Graph Folding Algorithm Do_row Do_col Do_mult Do_col Do_mult … Do_row … Do_col Do_mult … … … OAK RIDGE NATIONAL LABORATORY Do_row U. S. DEPARTMENT OF ENERGY 22 Outline Paradyn and the Performance Consultant MRNet Distributed Performance Consultant Sub-Graph Folding Algorithm Evaluation Summary OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 23 Search History Graph Example CPUbound c33.cs.wisc.edu c34.cs.wisc.edu main myapp{1272} myapp{1273} myapp{7624} myapp{7625} main main main main A B A C D A B C D E B A A B C B C C D D D OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 24 Search History Graphs Search History Graph is effective for presenting search-based performance diagnosis results… …but it does not scale to a large number of processes because it shows one sub-graph per process OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 25 Sub-Graph Folding Algorithm Combines host-specific sub-graphs into composite sub-graphs Each composite sub-graph represents a behavioral category among application processes Dynamic clustering of processes by qualitative behavior OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 26 SGFA: Example CPUbound c33.cs.wisc.edu c34.cs.wisc.edu c*.cs.wisc.edu main myapp{1272} myapp{1273} myapp{7624} myapp{*} myapp{7625} main main main main A B A C D A B B C C D A E D D A B C B D C E D OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 27 SGFA: Implementation Custom MRNet filter Filter in each MRNet process keeps folded graph of search results from all reachable daemons Updates periodically sent upstream By induction, filter in front-end holds entire folded graph Optimization for unchanged graphs OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 28 Outline Performance Consultant MRNet Distributed Performance Consultant Sub-Graph Folding Algorithm Evaluation Summary OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 29 DPC + SGFA: Evaluation Modified Paradyn to perform bottleneck searches using CA, PDA, or TDA approach Modified instrumentation cost tracking to support PDA Track global, per-process instrumentation cost separately Simple fixed-partition policy for scheduling global and local instrumentation Implemented Sub-Graph Folding Algorithm as custom MRNet filter to support TDA (used by all) Instrumented front-end, daemons, and MRNet internal processes to collect CPU, I/O load information OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 30 DPC + SGFA: Evaluation su3_rmd QCD pure lattice gauge theory code C, MPI Weak scaling scalability study LLNL MCR cluster 1152 nodes (1048 compute nodes) Two 2.4 GHz Intel Xeons per node 4 GB memory per node Quadrics Elan3 interconnect (fat tree) Lustre parallel file system OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 31 DPC + SGFA: Evaluation PDA and TDA: bottleneck searches with up to 1024 processes so far, limited by partition size CA: scalability limit at less than 64 processes Similar qualitative results from all approaches OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 32 DPC: Evaluation OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 33 DPC: Evaluation OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 34 DPC: Evaluation OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 35 DPC: Evaluation OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 36 DPC: Evaluation OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 37 DPC: Evaluation OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 38 DPC: Evaluation OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 39 DPC: Evaluation OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 40 DPC: Evaluation OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 41 SGFA: Evaluation OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 42 Summary Tool scalability is critical for effective use of large-scale computing resources On-line automated performance tools are especially important at large scale Our approach: MRNet Distributed Performance Consultant (TDA) plus Sub-Graph Folding Algorithm OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 43 References P.C. Roth, D.C. Arnold, and B.P. Miller, “MRNet: a Software-Based Multicast/Reduction Network for Scalable Tools,” SC 2003, Phoenix, Arizona, November 2003 P.C. Roth and B.P. Miller, “The Distributed Performance Consultant and the Sub-Graph Folding Algorithm: On-line Automated Performance Diagnosis on Thousands of Processes,” in submission Publications available from http://www.paradyn.org MRNet software available from http://www.paradyn.org/mrnet OAK RIDGE NATIONAL LABORATORY U. S. DEPARTMENT OF ENERGY 44