Designing Parallel Programs

Automatic vs Manuel Parallelization

- Very often, manually developing parallel codes is a:
Time Consuming
Complex
Error-prone and
Iterative Process

- The most commonn type of tool used to automatically parallelize a serial program is a parallelizing compiler or pre-processor.

- A parallelizing compiler generally works in two different ways:

A. Fully Automatic
- The compiler analyzes the source code and identifies opportunities for parallelism.
- The analysis includes identifying inhibitors to parallelism and possibly a cost weighting on whether or not the parallelism would actually improve performance.
- Loops (do,for) are the most frequent target for automatic parallelization.

B. Programmer Directed
- Using "compiler directives" or possibly compiler flags, the programmer explicitly tells the compiler how to parallelize the code.
- May be able to be used in conjunction with some degree of automatic parallelization also.

- The most common compiler generated parallelization is done using on-node shared memory and threads (such as OpenMP).

Caveats (warning/caution) that apply to automatic parallelization:
1. Wrong results may be produced.
2. Performance may actually degrade.
3. Much less flexible than manual parallelization.
4. Limited to a subset (mostly loops) of code.
5. May actually not parallelize code if the compiler analysis suggests there are inhibitors or the code is too complex.

Understand the Problem and the Program

- The first step in developing parallel software: understand the problem that you wish to solve in parallel.
- Determine whether ot not the problem is one that can actually be parallelized
- Identify the program's hotspots
- Identify the program's bottlenecks
- Identify inhibitors to parallelism.
- Investigate other algorithms if possible.
- Take advantage of optimized third parallel party software and highly optimized math libraries available from leading vendors

Partitioning
- to break the problem into discrete "chunks" of work that can be distributed to multiple tasks (aka decomposition sad)

- 2 basic ways to partition computation work among parallel tasks:
1. Domain Decomposition
- the data associated with the problem is decomposed. Each parallel task then works on a portion of the data.

-1d; 2d (block, cyclic)

2. Functional Decomposition
- The focus is on the computation to be performed rather than the data manipulated by the computation.
- The problem is decomposed according to the work that must be done.
- Each task then performs a portion of the overall work.

Example:
- Ecosystem modeling
- Signal Processing
- Climate modeling

Communications
The need for communication between tasks depends upon your problem:

You DON'T need communications:
- problems that don't need for tasks to share data.
- composed of multiple tasks that can act independently of each other to do their portion of the work.
- these types of problems are often called embarrasingly parallel because they are so straight-forward. Very little intertask communication is required.

You DO need communications:
- most parallel application are not quite so simple, and do require tasks to share data with each other.

Factors to Consider:

C
L
V
S
S
E
O

1. Cost of communications:
- Inter-task communication virtually always implies overhead.
- Machine cycles and resources that could be used for computation are instead used to package and transmit data.
- Communications frequently requires some type of synchronization between tasks, which can result in spending time "waiting" rather than doing work
- Competiting communication traffic can saturate the network bandwidth, further aggravating performance problems

2. Latency vs Bandwidth:
- Latency: is the time it takes to send a minimal (0 bytes) message from point A to point B. it is usually expressed in microseconds.
- Bandwidth: is the amount of data that can be communicated per unit of time. usually expressed in gigabytes/sec or megabytes/sec
- sending many small messages can cause latency to dominate communication overhead.
- it is more efficient to send many small messages over large messages, thus increasing the effective communication bandwidth.

3. Visibility of communications:
- with the Message Passing Model, communications are explicit and generally quite visible and under control of the programmer
- with the Data Parallel Model, communications often occur tranparently to the programmer, particularly on the distributed memory architecture

4. Synchronous vs Asynchronous communications
Synchrounous
- requires some type of "handshaking" between tasks that are sharing data. This can be explicitly structured in code by the programmer, or it may happen at a lower level unknown to the programmer.
- are often referred to as blocking communications since other work must wait until the communications have completed.

Asynchronous
- allow tasks to tranfer data independently from one another.
- often referred to as non-blocking communication since other work can be done while the communications are taking place.
- greatest benefit: interleaving computation with communication

5. Scope of communications
A. Point-to-point
- involves two tasks; one being the producer/sender of data, and the other one is the receiver/consumer

B. Collective
- involves data sharing between more than two tasks, which are often specified as being members in a common group, or collective.
- broadcast, scatter, gather, reduction

6. Efficiency of communications
- communications performance
- Which implementation for a given model should be used?
- What type of communication operations should be used?
- Network media - some platforms may offer some more than one network for communications. Which one is the best?

7. Overhead and complexity

Synchronization

3 types of Synchronization
1. Barrier

2. Lock/Semaphore
3. Synchronous communication operations

Designing Parallel Programs

Data Dependencies

- a dependence exists between program statement when the order of program execution affects the result of the program

- a data dependence results from multiple use of same storage location/s in different tasks

- dependencies are important to parallel programming because they are one of the primary inhibitors to parallelism.

How to handle data dependencies

1. Distributed Memory Architecture
- communicate required data at synchrpnization point

2. Shared Memory Architecture
- synchronize read/write operations between tasks

Load Balancing
- refers to the practice of distributing approximate equal amounts of works among tasks so that tasks are kept busy all the time. Can also be considered as minimization of task idle time.

- it is important to parallel programming for the performance reasons. For example, if all tasks are subject to barrier sync point, then the slowest task will be the overall performance.

How to Achieve Load Balance
1. Equally partition the work each task receives
- array/matrix, evenly distribute the data set among the tasks.
- loops iterations, evenly distibute the iterations across the tasks.
- If a heterogenous mix of machines with varying performance characteristic are being used, be sure to use some type of performance analysis tool to detect any load imbalances.

2. Use dynamic work assignment
- certain classes results in load imbalances even if the data is evenly distributed among the tasks. * sparse arrays - some tasks will have actual data work while others have mostly zeros * adaptive grid methods - some tasks may need to refine their mesh while others don't. * n-body simulations - where some particles may migrate to/from their original task domain to another task; where paticles owned by some tasks requires more work than those owned by other tasks.

- scheduler-task approach
- algorithm which detects and handles load imbalances as they occur dynamically within the code.

Granularity

Computation/Communication Ratio:
-

Fine-grain Parallelism
- R
- L
- F
- I
- I

Coarse-grain parallelism:
- Relatively large amounts
- H
- I
- H

Which is best?
- depends on the algorithm and the hardware environment in which it runs.
-

I/O

The Bad News:
- inhibitors
- requires order of magniude more time than memory operations
- may be immature or not available for all platforms
- write operations can result in overwriting
- read operations can be affected by the file server's ability to handle multiple read requests at the same time.
- can cause sever bottlenecks and even crash file servers

The Good News:
- parallel file systems are available: example: G
L
O
P

- MPI- 1996

Debugging

debugging tools:
- T
- D
- I
- S

tools for parallel program performance analysis and tuning: (10)
- L
- T
- H
- O
- V
- V
- P
- m
- m
- m