ppt
advertisement
Prepared 6/4/2011 by T. O’Neil for 3460:677, Fall 2011, The University of Akron. A quiet revolution and potential buildup Computation: TFLOPs vs. 100 GFLOPs T12 GT200 G80 G70 NV30 NV40 3GHz Core2 Duo 3GHz Xeon Quad Westmere CPU in every PC – massive volume and potential impact Introduction to Massively Parallel Processing – Slide 2 Introduction to Massively Parallel Processing – Slide 2 • Solve computation-intensive problems faster • Make infeasible problems feasible • Reduce design time • Solve larger problems in the same amount of time • Improve an answer’s precision • Reduce design time • Gain competitive advantage Introduction to Massively Parallel Processing – Slide 4 • Another factor: modern scientific method Nature Observation Numerical Simulation Physical Experimentation Theory Introduction to Massively Parallel Processing – Slide 5 A growing number of applications in science, engineering, business and medicine are requiring computing speeds that cannot be achieved by the current conventional computers because • Calculations too computationally intensive • Too complicated to model • Too hard, expensive, slow or dangerous for the laboratory. Parallel processing is an approach which helps to make these computations feasible. Introduction to Massively Parallel Processing – Slide 6 One example: grand challenge problems that cannot be solved in a reasonable amount of time with today’s computers. • Modeling large DNA structures • Global weather forecasting • Modeling motion of astronomical bodies Another example: real-time applications which have time constrains. If data are not processed during some time the result become meaningless. Introduction to Massively Parallel Processing – Slide 7 Atmosphere modeled by dividing it into 3- dimensional cells. Calculations of each cell repeated many times to model passage of time. For example Suppose whole global atmosphere divided into cells of size 1 mile 1 mile 1 mile to a height of 10 miles (10 cells high) – about 5108 cells. If each calculation requires 200 floating point operations, 1011 floating point operations necessary in one time step. Introduction to Massively Parallel Processing – Slide 8 Example continued To forecast the weather over 10 days using 10-minute intervals, a computer operating at 100 megaflops (108 floating point operations per second) would take 107 seconds or over 100 days. To perform the calculation in 10 minutes would require a computer operating at 1.7 teraflops (1.71012 floating point operations per second). Introduction to Massively Parallel Processing – Slide 9 Two main types Parallel computing: Using a computer with more than one processor to solve a problem. Distributed computing: Using more than one computer to solve a problem. Motives • Usually faster computation • Very simple idea – that n computers operating simultaneously can achieve the result n times faster • It will not be n times faster for various reasons. • Other motives include: fault tolerance, larger amount of memory available, … Introduction to Massively Parallel Processing – Slide 10 “... There is therefore nothing new in the idea of parallel programming, but its application to computers. The author cannot believe that there will be any insuperable difficulty in extending it to computers. It is not to be expected that the necessary programming techniques will be worked out overnight. Much experimenting remains to be done. After all, the techniques that are commonly used in programming today were only won at considerable toil several years ago. In fact the advent of parallel programming may do something to revive the pioneering spirit in programming which seems at the present to be degenerating into a rather dull and routine occupation …” - S. Gill, “Parallel Programming”, The Computer Journal, April 1958 Introduction to Massively Parallel Processing – Slide 11 Applications are typically written from scratch (or manually adapted from a sequential program) assuming a simple model, in a high-level language (i.e. C) with explicit parallel computing primitives. Which means Components difficult to reuse so similar components coded again and again Code difficult to develop, maintain or debug Not really developing a long-term software solution Introduction to Massively Parallel Processing – Slide 12 Processors Highest level Registers Memory level 1 Small, fast, expensive Memory level 2 Memory level 3 Lowest level Large, slow, cheap Main Memory Introduction to Massively Parallel Processing – Slide 13 Execution speed relies on exploiting data locality temporal locality: a data item just accessed is likely to be used again in the near future, so keep it in the cache spatial locality: neighboring data is also likely to be used soon, so load them into the cache at the same time using a ‘wide’ bus (like a multi-lane motorway) from a programmer point of view, all handled automatically – good for simplicity but maybe not for best performance Introduction to Massively Parallel Processing – Slide 14 Useful work (i.e. floating point ops) can only be done at the top of the hierarchy. So data stored lower must be transferred to the registers before we can work on it. Possibly displacing other data already there. This transfer among levels is slow. This then becomes the bottleneck in most computations. Spend more time moving data than doing useful work. The result: speed depends to a large extent on problem size. Introduction to Massively Parallel Processing – Slide 15 Good algorithm design then requires Keeping active data at the top of the hierarchy as long as possible. Minimizing movement between levels. Need enough work to do at the top of the hierarchy to mask the transfer time at the lower levels. The more processors you have to work with, the larger the problem has to be to accomplish this. Introduction to Massively Parallel Processing – Slide 16 Introduction to Massively Parallel Processing – Slide 17 Each cycle, CPU takes data from registers, does an operation, and puts the result back Load/store operations (memory registers) also take one cycle CPU can do different operations each cycle Output of one operation can be input to next CPUs haven’t been this simple for a long time! Introduction to Massively Parallel Processing – Slide 18 “The complexity for minimum component costs has increased at a rate of roughly a factor of two per year... Certainly over the short term this rate can be expected to continue, if not to increase. Over the longer term, the rate of increase is a bit more uncertain, although there is no reason to believe it will not remain nearly constant for at least 10 years. That means by 1975, the number of components per integrated circuit for minimum cost will be 65,000. I believe that such a large circuit can be built on a single wafer.” - Gordon E. Moore, co-founder, Intel Electronics Magazine, 4/19/1965 Photo credit: Wikipedia Introduction to Massively Parallel Processing – Slide 19 Translation: The number of transistors on an integrated circuit doubles every 1½ - 2 years. Data credit: Wikipedia Introduction to Massively Parallel Processing – Slide 20 As the chip size (amount of circuitry) continues to double every 18-24 months, the speed of a basic microprocessor also doubles over that time frame. This happens because, as we develop tricks, they are built into the newest generation of CPUs by the manufacturers. Introduction to Massively Parallel Processing – Slide 21 Instruction-level parallelism (ILP) Out-of-order execution, speculation, … Vanishing opportunities in power-constrained world Data-level parallelism Vector units, SIMD execution, … Increasing opportunities; SSE, AVX, Cell SPE, Clearspeed, GPU, … Thread-level parallelism Increasing opportunities; multithreading, multicore, … Intel Core2, AMD Phenom, Sun Niagara, STI Cell, NVIDIA Fermi, … Introduction to Massively Parallel Processing – Slide 22 Most processors have multiple pipelines for different tasks and can start a number of different operations each cycle. For example consider the Intel core architecture. Introduction to Massively Parallel Processing – Slide 23 Each core in an Intel Core 2 Duo chip has 14 stage pipeline 3 integer units (ALU) 1 floating-point addition unit (FPU) 1 floating-point multiplication unit (FPU) FP division is very slow 2 load/store units In principle capable of producing 3 integer and 2 FP results per cycle Introduction to Massively Parallel Processing – Slide 24 Technical challenges Compiler to extract best performance, reordering instructions if necessary Out-of-order CPU execution to avoid delays waiting for read/write or earlier operations Branch prediction to minimize delays due to conditional branching (loops, if-then-else) Memory hierarchy to deliver data to registers fast enough to feed the processor These all limit the number of pipelines that can be used and increase the chip complexity 90% of Intel chip devoted to control and data Introduction to Massively Parallel Processing – Slide 25 Power Performance (courtesy Mark Horowitz and Kevin Skadron) Introduction to Massively Parallel Processing – Slide 26 CPU clock stuck at about 3 GHz since 2006 due to problems with power consumption (up to 130W per chip) Chip circuitry still doubling every 18-24 months More on-chip memory and memory management units (MMUs) Specialized hardware (e.g. multimedia, encryption) Multi-core (multiple CPUs on one chip) Thus peak performance of chip still doubling every 18- 24 months Introduction to Massively Parallel Processing – Slide 27 Processor Memory Processor Memory Global Memory Handful of processors each supporting ~1 hardware thread On-chip memory near processors (cache, RAM or both) Shared global memory space (external DRAM) Introduction to Massively Parallel Processing – Slide 28 Processor Memory ••• Processor Memory Global Memory Many processors each supporting many hardware threads On-chip memory near processors (cache, RAM or both) Shared global memory space (external DRAM) Introduction to Massively Parallel Processing – Slide 29 Four 3.33 GHz cores, each of which can run 2 threads Integrated MMU Introduction to Massively Parallel Processing – Slide 30 Cannot continue to scale processor frequencies No 10 GHz chips Cannot continue to increase power consumption Can’t melt chips Can continue to increase transistor density As per Moore’s Law Introduction to Massively Parallel Processing – Slide 31 Exciting applications in future mass computing market have been traditionally considered “supercomputing applications” Molecular dynamic simulation, video and audio coding and manipulation, 3D imaging and visualization, consumer game physics, virtual reality products, … These “super apps” represent and model physical, concurrent world Various granularities of parallelism exist, but… Programming model must not hinder parallel implementation Data delivery needs careful management Introduction to Massively Parallel Processing – Slide 32 Computers no longer get faster, just wider You must rethink your algorithms to be parallel! Data-parallel computing is most scalable solution Otherwise: refactor code for 2 cores, 4 cores, 8 cores, 16 cores, … You will always have more data than cores – build the computation around the data Introduction to Massively Parallel Processing – Slide 33 Traditional parallel architectures cover some super apps DSP, GPU, network and scientific applications, … The game is to grow mainstream architectures “out” or domain-specific architectures “in” CUDA is the latter Traditional applications Current architecture coverage New applications Domain-specific architecture coverage Obstacles Introduction to Massively Parallel Processing – Slide 34 Introduction to Massively Parallel Processing – Slide 35 Massive economies of scale Massively parallel Introduction to Massively Parallel Processing – Slide 36 Produced in vast numbers for computer graphics Increasingly being used for Computer game “physics” Video (e.g. HD video decoding) Audio (e.g. MP3 encoding) Multimedia (e.g. Adobe software) Computational finance Oil and gas Medical imaging Computational science Introduction to Massively Parallel Processing – Slide 37 The GPU sits on a PCIe graphics card inside a standard PC/server with one or two multicore CPUs Introduction to Massively Parallel Processing – Slide 38 Up to 448 cores on a single chip Simplified logic (no out-of-order execution, no branch prediction) means much more of the chip is devoted to computation Arranged as multiple units with each unit being effectively a vector unit, all cores doing the same thing at the same time Very high bandwidth (up to 140GB/s) to graphics memory (up to 4GB) Not general purpose – for parallel applications like graphics and Monte Carlo simulations Can also build big clusters out of GPUs Introduction to Massively Parallel Processing – Slide 39 Four major vendors: NVIDIA AMD: bought ATI several years ago IBM: co-developed Cell processor with Sony and Toshiba for Sony Playstation, but now dropped it for high-performance computing Intel: was developing “Larrabee” chip as GPU, but now aimed for high-performance computing Introduction to Massively Parallel Processing – Slide 40 A quiet revolution and potential buildup Computation: TFLOPs vs. 100 GFLOPs T12 GT200 G80 G70 NV30 NV40 3GHz Core2 Duo 3GHz Xeon Quad Westmere CPU in every PC – massive volume and potential impact Introduction to Massively Parallel Processing – Slide 41 A quiet revolution and potential buildup Bandwidth: ~10x T12 GT200 G80 NV40 NV30 G70 3GHz Core2 Duo 3GHz Xeon Quad Westmere CPU in every PC – massive volume and potential impact Introduction to Massively Parallel Processing – Slide 42 High throughput computation GeForce GTX 280: 933 GFLOPs/sec High bandwidth memory GeForce GTX 280: 140 GB/sec High availability to all 180M+ CUDA-capable GPUs in the wild Introduction to Massively Parallel Processing – Slide 43 “Fermi” 3B xtors RIVA 128 3M xtors 1995 GeForce® 256 23M xtors 2000 GeForce 3 60M xtors GeForce FX 125M xtors 2005 GeForce 8800 681M xtors 2010 Introduction to Massively Parallel Processing – Slide 44 Throughput is paramount Must paint every pixel within frame time Scalability Create, run and retire lots of threads very rapidly Measured 14.8 Gthreads/sec on increment() kernel Use multithreading to hide latency One stalled thread is o.k. if 100 are ready to run Introduction to Massively Parallel Processing – Slide 45 Different goals produce different designs GPU assumes work load is highly parallel CPU must be good at everything, parallel or not CPU: minimize latency experienced by one thread Big on-chip caches, sophisticated control logic GPU: maximize throughput of all thread Number of threads in flight limited by resources lots of resources (registers, bandwidth, etc.) Multithreading can hide latency skip the big caches Share control logic across many threads Introduction to Massively Parallel Processing – Slide 46 GeForce 8800 (2007) Host Input Assembler Thread Execution Manager Parallel Data Cache Parallel Data Cache Parallel Data Cache Parallel Data Cache Parallel Data Cache Parallel Data Cache Parallel Data Cache Parallel Data Cache Texture Texture Texture Texture Texture Texture Texture Texture Texture Load/store Load/store Load/store Load/store Load/store Load/store Global Memory Introduction to Massively Parallel Processing – Slide 47 16 highly threaded streaming multiprocessors (SMs) > 128 floating point units (FPUs) 367 GFLOPs peak performance (25-50 times that of current high-end microprocessors) 265 GFLOPs sustained for applications such as visual molecular dynamics (VMD) 768 MB DRAM 86.4 GB/sec memory bandwidth 4 GB/sec bandwidth to CPU Introduction to Massively Parallel Processing – Slide 48 Massively parallel, 128 cores, 90 watts Massively threaded, sustains 1000s of threads per application 30-100 times speedup over high-end microprocessors on scientific and media applications Medical imaging, molecular dynamics, … Introduction to Massively Parallel Processing – Slide 49 Fermi GF100 (2010) ~ 1.5 TFLOPs (single precision), ~800 GFLOPs (double precision) 230 GB/sec DRAM bandwidth Introduction to Massively Parallel Processing – Slide 50 Introduction to Massively Parallel Processing – Slide 51 32 CUDA cores per SM (512 total) 8x peak FP64 performance 50% of peak FP32 performance Direct load/store to memory Usual linear sequence of bytes High bandwidth (hundreds of GB/sec) 64K of fast, on-chip RAM Software or hardware managed Shared amongst CUDA cores Enables thread communication Introduction to Massively Parallel Processing – Slide 52 SIMT (Single Instruction Multiple Thread) execution Threads run in groups of 32 called warps Minimum of 32 threads all doing the same thing at (almost) the same time Threads in a warp share instruction unit (IU) All cores execute the same instructions simultaneously, but with different data Similar to vector computing on CRAY supercomputers Hardware automatically handles divergence Natural for graphics processing and much scientific computing; also a natural choice for many-core chips to simplify each core Introduction to Massively Parallel Processing – Slide 53 Hardware multithreading Hardware resource allocation and thread scheduling Hardware relies on threads to hide latency Threads have all resources needed to run Any warp not waiting for something can run Context switching is (basically) free Introduction to Massively Parallel Processing – Slide 54 Lots of active threads is the key to high performance: no “context switching”; each thread has its own registers, which limits the number of active threads threads on each SM execute in warps – execution alternates between “active” warps, with warps becoming temporarily “inactive” when waiting for data for each thread, one operation completes long before the next starts – avoids the complexity of pipeline overlaps which can limit the performance of modern processors memory access from device memory has a delay of 400600 cycles; with 40 threads this is equivalent to 10-15 operations, so hopefully there’s enough computation to hide the latency Introduction to Massively Parallel Processing – Slide 55 Intel Core 2 / Xeon / i7 4-6 MIMD (Multiple Instruction / Multiple Data) cores each core operates independently each can be working with a different code, performing different operations with entirely different data few registers, multilevel caches 10-30 GB/s bandwidth to main memory Introduction to Massively Parallel Processing – Slide 56 NVIDIA GTX280: 240 cores, arranged as 30 units each with 8 SIMD (Single Instruction / Multiple Data) cores all cores executing the same instruction at the same time, but working on different data only one instruction de-coder needed to control all cores functions like a vector unit lots of registers, almost no cache 5 GB/s bandwidth to host processor (PCIe x16 gen 2) 140 GB/s bandwidth to graphics memory Introduction to Massively Parallel Processing – Slide 57 One issue with SIMD / vector execution: what happens with if-then-else branches? Standard vector treatment: execute both branches but only store the results for the correct branch. NVIDIA treatment: works with warps of length 32. If a particular warp only goes one way, then that is all that is executed. Introduction to Massively Parallel Processing – Slide 58 How do NVIDIA GPUs deal with same architectural challenges as CPUs? Very heavily multi-threaded, with at least 8 threads per core: Solves pipeline problem, because output of one calculation can be used an input for next calculation for same thread Switch from one set of threads to another when waiting for data from graphics memory Introduction to Massively Parallel Processing – Slide 59 Introduction to Massively Parallel Processing – Slide 60 Scalable programming model Minimal extensions to familiar C/C++ environment Heterogeneous serial-parallel computing Introduction to Massively Parallel Processing – Slide 61 45X 17X 100X 110-240X 13–457x 35X Introduction to Massively Parallel Processing – Slide 62 Big breakthrough in GPU computing has been NVIDIA’s development of CUDA programming environment Initially driven by needs of computer games developers Now being driven by new markets (e.g. HD video decoding) C plus some extensions and some C++ features (e.g. templates) Introduction to Massively Parallel Processing – Slide 63 Host code runs on CPU, CUDA code runs on GPU Explicit movement of data across the PCIe connection Very straightforward for Monte Carlo applications, once you have a random number generator Harder for finite difference applications Introduction to Massively Parallel Processing – Slide 64 At the top level, we have a master process which runs on the CPU and performs the following steps: initialize card allocate memory in host and on device repeat as needed 1. 2. 3. 1. 2. 3. 4. copy data from host to device memory launch multiple copies of execution “kernel” on device copy data from device memory to host de-allocate all memory and terminate Introduction to Massively Parallel Processing – Slide 65 At a lower level, within the GPU: each copy of the execution kernel executes on an SM if the number of copies exceeds the number of SMs, then more than one will run at a time on each SM if there are enough registers and shared memory, and the others will wait in a queue and execute later all threads within one copy can access local shared memory but can’t see what the other copies are doing (even if they are on the same SM) there are no guarantees on the order in which the copies will execute Introduction to Massively Parallel Processing – Slide 66 Augment C/C++ with minimalist abstractions Let programmers focus on parallel algorithms, not mechanics of parallel programming language Provide straightforward mapping onto hardware Good fit to GPU architecture Maps well to multi-core CPUs too Scale to 100s of cores and 10000s of parallel threads GPU threads are lightweight – create/switch is free GPU needs 1000s of threads for full utilization Introduction to Massively Parallel Processing – Slide 67 Hierarchy of concurrent threads Lightweight synchronization primitives Shared memory model for cooperating threads Introduction to Massively Parallel Processing – Slide 68 Parallel kernels composed of many threads Thread t All threads execute the same sequential program Threads are grouped into thread blocks Threads in the same block can cooperate Block b t0 t1 … tB Threads/blocks have unique IDs Introduction to Massively Parallel Processing – Slide 69 CUDA virtualizes the physical hardware Thread is a virtualized scalar processor: registers, PC, state Block is a virtualized multiprocessor: threads, shared memory Scheduled onto physical hardware without pre- emption Threads/blocks launch and run to completion Blocks should be independent Introduction to Massively Parallel Processing – Slide 70 Block Memory Block ••• Memory Global Memory Introduction to Massively Parallel Processing – Slide 71 Cf. PRAM (Parallel Random Access Machine) model Processors Global Memory Global synchronization isn’t cheap Global memory access times are expensive Introduction to Massively Parallel Processing – Slide 72 Cf. BSP (Bulk Synchronous Parallel) model, MPI Processor Memory ••• Processor Memory Interconnection Network Distributed computing is a different setting Introduction to Massively Parallel Processing – Slide 73 Multicore CPU Manycore GPU Introduction to Massively Parallel Processing – Slide 74 Philosophy: provide minimal set of extensions necessary to expose power Function and variable qualifiers Execution configuration Built-in variables and functions valid in device code Introduction to Massively Parallel Processing – Slide 75 Introduction to Massively Parallel Processing – Slide 76 Application Description Source Kernel % time H.264 SPEC ‘06 version, change in guess vector 34,811 194 35% LBM SPEC ‘06 version, change to single precision and print fewer reports 1,481 285 >99% RC5-72 Distributed.net RC5-72 challenge client code 1,979 218 >99% FEM Finite element modeling, simulation of 3D graded materials 1,874 146 99% RPES Rye Polynomial Equation Solver, quantum chem, 2electron repulsion 1,104 281 99% PNS Petri Net simulation of a distributed system 322 160 >99% SAXPY Single-precision implementation of saxpy, used in Linpack’s Gaussian elim. routine 952 31 >99% TPACF Two Point Angular Correlation Function 536 98 96% FDTD Finite-Difference Time Domain analysis of 2D electromagnetic wave propagation 1,365 93 16% MRI-Q Computing a matrix Q, a scanner’s configuration in MRI reconstruction 490 33 >99% Introduction to Massively Parallel Processing – Slide 77 GeForce 8800 GTX vs. 2.2 GHz Opteron 248 Introduction to Massively Parallel Processing – Slide 78 10x speedup in a kernel is typical, as long as the kernel can occupy enough parallel threads 25x to 400x speedup if the function’s data requirements and control flow suit the GPU and the application is optimized Introduction to Massively Parallel Processing – Slide 79 Parallel hardware is here to stay GPUs are massively parallel manycore processors Easily available and fully programmable Parallelism and scalability are crucial for success This presents many important research challenges Not to mention the educational challenges Introduction to Massively Parallel Processing – Slide 80 Reading: Chapter 1, “Programming Massively Parallel Processors” by Kirk and Hwu. Based on original material from The University of Akron: Tim O’Neil, Kathy Liszka Hiram College: Irena Lomonosov The University of Illinois at Urbana-Champaign David Kirk, Wen-mei W. Hwu The University of North Carolina at Charlotte Barry Wilkinson, Michael Allen Oregon State University: Michael Quinn Oxford University: Mike Giles Stanford University: Jared Hoberock, David Tarjan Revision history: last updated 6/4/2011. Introduction to Massively Parallel Processing – Slide 81
