Parallel Programming Models

advertisement

Lecture 7:

POSIX Threads - Pthreads

Parallel Programming Models

Parallel Programming Models:

Data parallelism / Task parallelism

Explicit parallelism / Implicit parallelism

Shared memory / Distributed memory

Other programming paradigms

Object-oriented

Functional and logic

Parallel Programming Models

 Shared Memory

The programmer’s task is to specify the activities of a set of processes that communicate by reading and writing shared memory.

Advantage: the programmer need not be concerned with data-distribution issues.

Disadvantage: performance implementations may be difficult on computers that lack hardware support for shared memory, and race conditions tend to arise more easily

 Distributed Memory

Processes have only local memory and must use some other mechanism (e.g., message passing or remote procedure call) to exchange information.

Advantage: programmers have explicit control over data distribution and communication.

Shared vs Distributed Memory

P P P

Bus

Memory

P

Shared memory

M

P

M

P

Network

M

P

M

P

Distributed memory

Parallel Programming Models

Parallel Programming Tools:

Parallel Virtual Machine (PVM)

Distributed memory, explicit parallelism

Message-Passing Interface (MPI)

Distributed memory, explicit parallelism

PThreads

Shared memory, explicit parallelism

OpenMP

Shared memory, explicit parallelism

High-Performance Fortran (HPF)

Implicit parallelism

Parallelizing Compilers

Implicit parallelism

Parallel Programming Models

Shared Memory Model

 Used on Shared memory MIMD architectures

 Program consists of many independent threads

 Concurrently executing threads all share a single, common address space.

 Threads can exchange information by reading and writing to memory using normal variable assignment operations

Parallel Programming Models

Memory Coherence Problem

 To ensure that the latest value of a variable updated in one thread is used when that same variable is accessed in another thread.

Thread 1 Thread 2

X

Hardware support and compiler support are required

 Cache-coherency protocol

Parallel Programming Models

Distributed Shared Memory (DSM) Systems

 Implement Shared memory model on Distributed memory MIMD architectures

 Concurrently executing threads all share a single, common address space.

 Threads can exchange information by reading and writing to memory using normal variable assignment operations

 Use a message-passing layer as the means for communicating updated values throughout the system.

Parallel Programming Models

Synchronization operations in Shared Memory Model

Monitors

Locks

Critical sections

Condition variables

Semaphores

Barriers

PThreads

POSIX Threads – Pthreads

www.pthreads.org/

PThreads

In the UNIX environment a thread :

Exists within a process and uses the process resources

Has its own independent flow of control

Duplicates only the essential resources it needs to be independently schedulable

May share the process resources with other threads

Dies if the parent process dies

Is "lightweight" because most of the overhead has already been accomplished through the creation of its process.

PThreads

Because threads within the same process share resources:

 Changes made by one thread to shared system resources will be seen by all other threads.

 Two pointers having the same value point to the same data.

 Reading and writing to the same memory locations is possible, and therefore requires explicit synchronization by the programmer.

PThreads

 pthread_create(thread, attr, start_routine, arg): creates new threads of control

• thread: unique identifier of the thread

• attr: used to set thread attributes (default NULL)

• start_routine: the C routine that the thread will execute once it is created

• arg: a single argument that may be passed (passed by reference) to start_routine (NULL if no arguments)

 pthread_exit(): A thread terminates when the function being executed by the thread completes or when an explicit thread exit function is called.

PThread Code

#include <pthread.h>

#include <stdio.h>

#define NUM_THREADS 5 void *PrintHello(void *threadid) { long tid; tid = (long)threadid;

} printf("Hello World! It's me, thread #%ld!\n", tid); pthread_exit(NULL); int main (int argc, char *argv[]) { pthread_t threads[NUM_THREADS]; int rc; long t; for(t=0; t<NUM_THREADS; t++){ printf("In main: creating thread %ld\n", t); rc = pthread_create(&threads[t], NULL, PrintHello, (void *)t); if (rc){ printf("ERROR; return code from pthread_create() is %d\n", rc); exit(-1);

}

}

} pthread_exit(NULL);

PThreads

 The data-oriented synchronization routines are based on the use of a mutex (mutual exclusion).

 A mutex is a dynamically allocated data structure that can be passed as an argument to the synchronization routines

 pthread_mutex_lock() and pthread_mutex_unlock(): Once a pthread_mutex_lock call is made on a specific mutex, subsequent pthread_mutex_lock calls will block until a call is made to pthread_mutex_unlock with that mutex.

PThreads

 Condition variables allow a thread to wait until a Boolean predicate that depends on the contents of one or more shared-memory locations becomes true.

 A condition variable associates a mutex with the desired predicate.

Before the program makes its test, it obtains a lock on the associated mutex. Then it evaluates the predicate.

 If the predicate evaluates to false , the thread can execute a pthread_cond_wait() operation, which atomically suspends the calling thread, puts the thread record on a waiting list that is part of the condition variable, and releases the mutex. The thread scheduler is now free to use the processor to execute another thread.

PThreads

 If the predicate evaluates to true , the thread simply releases its lock and continues on its way.

 If a thread changes the value of any shared variables associated with a condition variable predicate, it needs to cause any threads that may be waiting on this condition variable to be rescheduled. The pthread_cond_signal() causes one of the threads waiting on the condition variable to become unblocked, returning from the pthread_cond_wait that caused it to block in the first place. The mutex is automatically reobtained as part of the return from the wait, so the thread is in the position to reevaluate the predicate immediately.

Parallel Programming Models

Example: Pi calculation

P = f

0

1 f(x) dx = f

0

1 4/(1+x 2 ) dx = w f(x) = 4/(1+x 2 )

f(x i

) n = 10 w = 1/n x i

= w(i-0.5) f(x) x

0 0.1 0.2 x i

1

Parallel Programming Models

Sequential Code f(x)

#define f(x) 4.0/(1.0+x*x); main(){ int n,i; float w,x,sum,pi; printf(“n?\n”); scanf(“%d”, &n); w=1.0/n; sum=0.0; for (i=1; i<=n; i++){ x=w*(i-0.5); sum += f(x);

} pi=w*sum; printf(“%f\n”, pi);

}

0 0.1 0.2 x i

P = w ∑ f(x i

) f(x) = 4/(1+x 2 ) n = 10 w = 1/n x i

= w(i-0.5)

1 x

Parallel Virtual Machine (PVM)

Data Distribution f(x)

0 0.1 0.2 x i

1 x

#include <pthread.h>

#include <stdio.h>

#define f(x) 4.0/(1.0+x*x)

#define NUM_THREADS 4 float pi; pthread_mutex_t m1; void *worker(void args) { int i, p, n, id; float sum, w, x; p=args[0]; n=args[1]; id=args[2]; sum=0.0; w=1.0/n; for (i=id; i<n; i+=p) { x=(i+0.5)*w; sum+=f(x);

} sum=sum*w; pthread_mutex_lock(&m1); pi += sum; pthread_mutex_unlock(&m1);

}

PThread Code int main (int argc, char *argv[]) { pthread_t threads[NUM_THREADS]; int i, n, nproc, args[3]; scanf(“%d:, &nproc); scanf(“%d:, &n); args[0]=nproc; args[1]=n; pthread_mutex_init(&m1, NULL); for(i=0; i<NUM_THREADS; i++){ args[2]=i; pthread_create(&threads[i], NULL, worker, (void *)args[0]);

} for(i=0; i<NUM_THREADS; i++){ pthread_join(&threads[i], NULL); printf(“Pi=%f\n”, pi);

}

Download