Supercomputers Special Course of Computer Architecture H.Amano Contents • • • • What are supercomputers? Architecture of Supercomputers Representative supercomputers Exa-Scale supercomputer project Defining Supercomputers • High performance computers mainly for scientific computation. – Huge amount of computation for Biochemistry, Physics, Astronomy, Meteorology and etc. – Very expensive: developed and managed by national fund. – High level techniques are required to develop and manage them. – USA, Japan and China compete the top 1 supercomputer. – A large amount of national fund is used, and tends to be political news→ In Japan, the supercomputer project became the target of budget review in Dec. 2009 「K」 achieved 10PFLOPS, and became the top 1 in the last year, but Sequoia got back in the last month. FLOPS • Floating Point Operation Per Second • Floating Point number – (Mantissa) × ２ （index） – Double precision 64bit, Single precision 32bit. – IEEE Standard defines the format and rounding sign Single Double index 8 mantissa 23 11 52 The range of performance 106 109 1012 1015 1018 100万 10億 1兆 1000兆 100京 M（Mega） G（Giga） T（Tera） P（Peta） E（Exa） iPhone4S 140MFLOPS Supercomputers 10TFLOPS-16PFLOPS High-end PC 50-80GFLOPS Powerful GPU Tera-FLOPS 10PFLOPS = 1京回 in Japanese → The name 「K」 comes from it. growing ratio: 1.9times/year How to select top 1？ • Top500/Green500: Performance of executing Linpack – Linpack is a kernel for matrix computation. – Scale free – Performance centric. • Godon Bell Prize – Peak Performance, Price/Performance, Special Achievement • HPC Challenge – Global HPL Matrix computation: Computation – Global Random Access： random memory access： Communication – EP stream per system: heavy load memory access： Memory performance – Global FFT: Complicated problem requiring both memory and communication performance. • Nov. ACM/IEEE Supercomputing Conference – Top500、Gordon Bell Prize、HPC Challenge、Green500 • Jun. International Supercomputing Conference – Top500、Green500 10 Rmax: Peta FLOPS Top 5 K Japan Sequoia USA 16PFLOPS 9 8 3 Tianhe(天河) China 2 Jaguar USA Nebulae China １ Kraken USA Roadrunner USA Tsubame Japan Jugene Germany 2010.6 2010.11 2011.6 2011.11 From SACSIS2012 Invited Speech. Top 500 2011 11月 Name Developmen t Hardware Cores Performanc e TFLOPS Power （KW) K (京) (Japan) RIKEN AICS SPARC VIIIfx 2.0GHz Tofu Interconnect Fujitsu 705024 10510 (11280) 12659.9 Tianhe1A( 天河) (China) National Supercompute r Center Tenjien NUDT YH MPPXeon X5670 6C 2.93GHz,NVIDIA 2050 NUDT 186368 2566 (4701) 4040 Jaguar (USA) DOE/SC/Oak Ridge National Lab. Cray XT5-HE Opteron 6-Core 2.6GHz, Cray Inc. 224162 1759 (2331) 6950 Nebulae( China) National Supercomputi ng Centre in Shenzhen Dawning TC3600 Blade, Xeon X5650 6C 2.66GHz, Infiniband QDR, NVIDIA 2050,Dawing 120640 1271 (2974) 2580 TSUBAM E2.0(Jap an) GSIC,Tokyo Inst. of Technology HP ProLiant 73238 SL390s G7 Xeon 6C X5670, NVIDIA GPU,NEC/HP 1192 (2287) 1398.6 Green 500 2011 11月 1 Machine Place FLOPS/W Total kW BlueGene/Q, Power BQC 16C 1.60 GHz, Custom IBM - Rochester 2026.48 85.12 IBM Blue Gene/Q got 1-5 2－5 BlueGene/Q, Power BQC 16C 1.60 GHz, Custom BlueGene/Q Prototype IBM – Thomas J. Watson Research Center /Rochester 1689.86 2026.48 6 DEGIMA Cluster, Intel i5, ATI Radeon GPU, Infiniband QDR Nagasaki Univ. 1378.32 47.05 7 Bullx B505, Xeon E5649 6C 2.53GHz, Infiniband QDR, NVIDIA 2090 Barcelona Supercomputing Center 1266.26 81.50 8 Curie Hybrid Nodes - Bullx B505, TGCC / GENCI Nvidia M2090, Xeon E5640 2.67 GHz, Infiniband QDR 1010.11 108.80 10位はTsubame-2．0（東工大） Why Top1? • Top1 is just a measure of matrix computation. • Top1 of Green500, Gordon Bell Prize, Top1 of each HPC Challenge program → All machines are valuable. TV or newspapers are too much focus on Top 500. • However, most top 1 computer also got Gordon Bell Prize and HPC Challenge top1. – K and Sequoia • Impact of Top 1 is great! Why supercomputers so fast? × Because they use high freq. clock Freq. Pentium4 3.2GHz Clock freq. of High end PC Nehalem 3.3GHz K 2GHz Sequoia 1.6GHz 1GHｚ 40％ / year Alpha21064 150MHz 100MHｚ 1992 The speed up of the clock is saturated in 2003. Power and heat dissipation The clock frequency of K and Sequoia is lower than that of common PCs 2000 2008 年 Major 3 methods of parallel processing in supercomputers Supercomputer = massively parallel computers – SIMD (Single Instruction Stream Multiple Data Streams) • Most accelerators – Pipelined processing • Vector computers – MIMD(Multiple Instruction Streams Multiple Data Streams): • Homogeneous (vs. Accelerators), Scalar (vs. Vector machines) – Although all supercomputers use three methods in various level, it can be classified by its usage. Key issues other than computational nodes Large high bandwidth memory Large disk High speed Interconnection Networks. SIMD (Single Instruction Stream Multiple Data Streams Instruction Memory Instruction Processing Unit Data memory •All Processing Units executes the same instruction •Low degree of flexibility •Illiac-IV/MMX instructions/ClearSpeed/IMAP /GP-GPU（coarse grain） •CM-2,（fine grain） GPGPU(General-Purpose computing on Graphic ProcessingUnit) – TSUBAME2.0(Xeon+Tesla,Top500 2010/11 4th ) – 天河一号(Xeon+FireStream,2009/11 5th ) ※()内は開発環境 GeForce GTX280 240 cores Host Input Assembler Thread Execution Manager Thread Processors Thread Processors Thread Processors Thread Processors Thread Processors … PBSM PBSM PBSM PBSM PBSM PBSM Load/Store Global Memory PBSM PBSM PBSM PBSM GPU (NVIDIA’s GTX580) 128 Cores 128 Cores L2 Cache 128 Cores 128 Cores 512 GPU cores ( 128 X 4 ) 768 KB L2 cache 40nm CMOS 550 mm^2 Cell Broadband Engine PS3 PPE L2 C SPE L1 C SXU SXU SXU SXU IBM Roadrunner Common platform for supercomputers and games PXU LS LS LS LS IOIF1 DMA DMA DMA DMA 1.6GHz / 4 X 16B data rings BIF/ MIC SXU SXU SXU SXU IOIF0 LS LS LS LS DMA DMA DMA DMA Peta FLOPS 11 K Japan Peak performance vs Linpack Performance 10 5 4 The difference is large in machines with accelerators Tianhe(天河) China Homogeneous Using GPU 3 Nebulae China 2 Jaguar USA Tsubame Japan １ Accelerator type is energy efficient. Pipeline processing １ ２ ３ ４ ５ ６ Stage Each stage sends the result/receives the input every clock cycle. N stages = N times performance Data dependency makes RAW hazards and degrades the performance. If the large array is treated, a lot of stages can work efficiently. Vector computers vector registers a0a1a2….. adder Y=Y+X[i] multiplier X[i]=A[i]＊B[i] b0b1b2…. The classic style supercomputers since Cray-1. Earth simulator may be the last vector supercomputer. Vector computers vector registers a1a2….. adder multiplier a0 b0 Y=Y+X[i] X[i]=A[i]＊B[i] b1b2…. Vector computers vector registers a2….. adder multiplier a1 a0 b0 b1 Y=Y+X[i] X[i]=A[i]＊B[i] b2…. Vector computers vector registers a11….. multiplier a10 adder x1 x0 a9 b9 b10 Y=Y+X[i] X[i]=A[i]＊B[i] b11…. MIMD（Multipe-Instruction Streams/ Multiple-Data Streams) • Multiple processors (cores) can work independently. – Synchronization mechanism – Data communication: Shared memory • All supercomputers are MIMD with multiple cores. • However, K and Sequoia (BlueGene Q) are typical massively parallel MIMD machines. – homogeneous computers – scalar processors MIMD（Multipe-Instruction Streams/ Multiple-Data Streams) Node ０ ０ Node １ １ ２ Ｉｎｔｅｒｃｏｎｎｅｃｔiｏｎ Ｎｅｔｗｏｒｋ Node 2 ３ Node ３ Shared Memory Processors which can work independently. Multi-Core (Intel’s Nehalem-EX) CPU CPU L3 Cache CPU CPU CPU CPU L3 Cache CPU CPU 8 CPU cores 24MB L3 cache 45nm CMOS 600 mm^2 Intel 80-Core Chip Intel 80-core chip [Vangal,ISSCC’07] How to program them？ • Can the common programs for PC be accelerated on supercomputers? – Yes, a certain degree by parallel compilers. • However, in order to efficient use of many cores, specialists must optimize programs. – Multithread using MPIs – Open MP – Open CL/CUDA → GPU accelerator type The fastest computer Also simple NUMA From IBM web site IBM’s BlueGene Q • Successor of Blue Gene L and Blue Gene P. • Sequoia is consisting of BlueGene Q • 18 Power processors (16 computational, 1 control and 1 redundant) and network interfaces are provided in a chip. • Inner-chip interconnection is a cross-bar switch. • 5 dimensional Mesh/Torus • 1.6GHz clock. Japanese supercomputers • K-Supercomputer – Homogeneous scalar type massively parallel computers. • Earth simulator – Vector computers – The difference between peak and Linpack performance is small. • TIT’s Tsubame – A lot of GPUs are used. Energy efficient supercomputer. • Nagasaki University’s DEGIMA – A lot of GPUs are used. Hand made supercomputer. High costperformance. Gordon Bell prize cost performance winner • GRAPE projects – For astronomy, dedicated supercomputers. SIMD、Various version won the Gordon Bell prize. SACSIS2012 Invited Speech Supercomputer 「K」 L2 C Memory Core Core Core Core Core Core Core Core Tofu Interconnect 6-D Torus/Mesh Inter Connect Controller SPARC64 VIIIfx Chip 4 nodes/board 96nodes/Lack 24boards/Lack RDMA mechanism NUMA or UMA+NORMA SACSIS2012 Invited speech SACSIS2012 invited speech water cooling system Lacks of K 6 dimensional torus Tofu 3-dimensional mesh ２ ００ ０ ００ ０１０ ０ ２０ １ ００ １０１ ００１ ００ ２ １０ １１ ０ １ １１ ０ １１ ０１２ １２０ １２０ １２１ ０ ２１ ０ ２２ ２０１ ２０ ２ １０ ２ １１ ２１２ 3-ary 1-cube １１２ ２２１ ２ ２２ 3-ary 2-cube １２２ 3-ａｒｙ 3-ｃｕｂｅ 4 dimensional mesh 0*** 1*** 2*** Why K could get top 1 • The delay of BlueGeneQ/Sequoia – Financial crisis in USA • Withdrawal of NEC/Hitachi – As starting, the complex system of a vector machine and a scalar machine was planned. – All budget can be used only for scalar machine. • Budget reviewing made the project famous. – Enough fund was thrown in short period. • Engineers in Fujitsu did really good job. SACSIS2012 invited talk Peak performance 40TFLOPS The earth simulator Interconnection Network (16GB/s x 2) Node 1 7 0 1 … Vector Processor 1 …. Vector Processor 0 … Shared Memory 16GB Vector Processor Vector Processor 1 7 Node 0 … Vector Processor Vector Processor Shared Memory 16GB Vector Processor 0 Vector Processor Vector Processor Shared Memory 16GB 7 Node 639 The Earth simulator (2002) Simple NUMA TIT’s Tsubame Well balanced supercomputer with GPUs Nagasaki Univ’s DEGIMA GRAPE-DR Kei Hiraki “GRAPE-DR” http://www.fpl.org (FPL2007) Exa-scale computer • Japanese national project for exa-scale computer started. • Feasibility Study started. – U. Tokyo, Tsukuba Univ. Tohoku Univ. and Riken. • It is difficult to produce supercomputers with Japanese original chips. • In Japan, a vendor suffers loss for developing supercomputers. • The vendor may retrieve development fee later by selling smaller systems. • However, Japanese semiconductor companies will not be able to support a big money for development. • If Intel’s CPUs or NVIDIA’s GPUs are used, a huge national money will flow to US companies. • For exa-scale: 70,000,000 cores are needed. – The limitation of budget is severer than technical limit. Amdahl’s law Serial part １％ Parallel part 99％ Accelerated by parallel processing 0.01 + 0.99/p 50 times with 100 cores、91 times with 1000 cores If there is a small part of serial execution part, the performance improvement is limited. Why Exa-scale supercomputers? • The ratio of serial part becomes small for the large scale problem. – Linpack is scale free benchmark. – Serial execution part 1 day＋Parallel execution part 10 years → 1day+1day: A big impact. • Are there any big programs which cannot be solved by K but can be solved by Exa-scale supercomputers? – The number of programs will be decreased. – Can we find new area of application？ • It is important such a big computing power is open for researches. Should we develop a floating computation centric supercomputers? • What people wants big supercomputer to do? – Finding new medicines: Pattern matching. – Simulation of earthquake, Meteorology for analyzing global warming. – Big data – Artificial Intelligence • Most of them are not suitable for floating computation centric supercomputers. • “Supercomputers for big data” or “Super-cloud computers” might be required. Motivation and limitation • Integrated computer technologies including architecture, hardware, software, dependable techniques, semiconductors and application. • Flagship and symbols. • No-computer is remained in Japan other than supercomputers • A super computing power is open for peaceful researches. • It is a tool which makes impossible analysis possible. • What needs infinite computing power？ • Is it a Japanese supercomputer if all cores and accelerators are made in USA？ • Does floating centric supercomputer to solve LInpack as fast as possible really fit the demand? Look at Exa-scale computer project! Excise • A target program: serial computation part :1 parallel computation part: N3 • K: 700,000 cores • Exa: 70,000,000 cores • What N makes Exa 10 times faster than K?