advertisement

Principles of Parallel Algorithm Design Two major steps in parallel computing 1. The decomposition of a computation into subtasks 2. map them to different processors It is crucial for programmers to understand the relationship between the underlying machine model and the parallel program to develop efficient programs. Types of concurrency ．Data parallelism: Identical operations are applied concurrently on different data Ex: C=A+B ci , j ai , j bi , j Image processing , dense linear algebra ．Task parallelism: ID# Model Year Color 4523 Accord 1997 Blue 3476 Taurus 1997 Green 7623 Camry 1996 Black 9834 Civic 1994 Black 6734 Accord 1996 Green 5342 Contour 1996 Black 3845 Maximum 1996 Blue 8354 Malibu 1997 Black 4395 Accord 1996 Red 7352 Accord 1997 Red We want to find (Model=Accord and Year=1996) and (Color=Green or Color=Blue) 1. creating task: 2. join : the merger of two (tasks) flows. 3. Synchronize (barrier) Wait at some point for all the other tasks to catch up. ．Functional(stream)Parallelism : refers to the simultaneous executing of different programs on a data stream. Remarks: Many problems exhibit a combination of data, task, stream parallelism. ．Two important notes 1. load balance : equal computational load with each task 2. communication costs : O( n 2 / p ) less communication & less data are needed. Runtime: Ts : serial runtime (one processor) The time elapsed between the beginning and the end of its execution on a sequential computer. T p : parallel runtime (execution time on P processors) The time elapsed from the moment that a parallel computation starts to the moment that the last processor finishes execution, which includes essential computation, overheads of parallelism communication load unbalances idliey serial components in program Serial components in program -something can NOT be broken down any farther -critical path : The smallest chain of instruction that have a serial ordering among them Ex: adding n numbers t c :time of adding two numbers T p 3t c Ts 7t c ( log2 n tc is the fastest possible runtime) Speed up s Ts Tp is a measure that captures the relative benefit of solving a problem in parallel. Note: We would like to choose the best algorithm run time to be Ts . Theoretically, speedup can never exceed the number of processors. However, sometimes super-linear speedup does occur due to non-optimal sequential algorithm or to machine characteristic (cache memory). Ex: n2 n 1.T p p n2 n 100 2. T p p n2 n 0 .6 p 2 3.T p p n-problem size p-number of processors Remark: we need to analysis to speed up Efficiency : is a measure of the fraction of time for which a processor is usefully employed E Ts p Tp Cost of an algorithm: Cost p p T p Cost p Ts p T p ideally Ts or E 1 p Tp real world : Cost optimal ~E~O(1) Cost p ~ O(T s ) Ex: Summing up n(=p) numbers Ts (n 1)t c ~ O (n) T p (t c t s t w ) log n Cost p n log n(t c t s t w ) ~ O (n log n) Not cos t optimal Now n>p n numbers per processor p n T p t c ( 1) (t c t s t w ) log p p t c (n 1) s n t c ( 1) (t c t s t w ) log p p t c (n 1) E t c (n p ) (t c t s t w ) p log p n 1 tc t s t w (n p) p log p tc 1. keep n fixed , n>>p tc E 1 ts tw E p E 2.keep p fixed n E 1 Remark: How to scale problem & processor for optimal performance? memory generally increase linearly with number of processors let n= kp 1 E= (k 1) p p c log p kp 1 (k p 1) if p E 0 Conclusion : Increasing processors Increasing efficiency. Problem size: the total number of basic operations required to solve the problem. A data parallelism of a matrix: Model 1: n2 n T p t c (t s t w )4 p p n 2tc E n 2 t c 4 p (t s t w n ) p 1 1 k1 p p k 2 n n2 To keep E constant n 2 O( p) So, the problem size p. Model 2: (2-dim problem Memory size O(n 2 ) n2 T p t c (t s t w n) p n 2tc E 2 n t c p (t s t w n) 1 1 t t n p p p 1 s w 2 1 k1 2 k 2 n n tc n To keep E constant we need n O( p) Not efficient. Scalability of Parallel Systems Very often, programs are designed and tested for smaller programs on fewer processors. However, the real problems the programs are intended to solved are much larger, and the machines contain larger number of processors. Parallel runtime: The time elapsed from the moment that a parallel computation starts to the moment that the last processor finishes execution, which includes essential computation: Interprocessor communication: Load Imbalance: sometimes (for example, search and optimization), it is impossible (or at least difficult) to predict the size of the subtasks assigned to various processors. Quick sort pick a pivot half of the elements are less than ideally pivot & half are grather pivot=5 Ts n log n tc each level contains n elements need ntc operations n T p (n )t c ~ 2n t c 2 Cost n(2n t c ) O (n 2 ) O(n log n) step1: sort local lists n n ( log )t c p p step2: merge need 2n tc p 2n 4n pn n ( ) ~ O (log n ) p p p p