A parallel system is a computer with more than one processor for parallel processing. Although there are many kinds of parallel computers, they are basically distinguished by the kind of interconnection between the processor, known as processing elements (PES), and the memory. One major way to classify parallel computers is based on their memory architectures. Shared memory based parallel computing systems have multiple processors that access all available memory as a global address space. The distributed system is a network of a set of asynchronously connected computing devices. Communication in DS is either through shared memory or through messages. In wide-spread distributed systems, work and information are physically distributed, implying that computing needs should be distributed. Along with the improvement in response time, this system contributes to offering local control over data. With this solid background of multiprocessor systems, parallel computing, distributed systems, and shared memory. speed-up performance law such as the Amdahl’s law was introduced to throw light on algorithm design for speed-up and operational efficiency of the parallel system. Concurrency is another important factor which enables distributed systems to share memory. and to better understand its execution, Dining philosophers’ algorithm Bully’s algorithm and Logical clocks were introduced. The memory organization of the parallel system also has a profound impact on algorithm design. In systems in which memory is distributed among the processors, the interconnection topology is the major factor in the design of an algorithm to solve a given problem. As the number of processors becomes larger, it is increasingly difficult to design algorithms to achieve high speed-up. On the other hand, systems with a global shared memory provide more flexibility for algorithm design, although memory module contention can adversely affect performance and the algorithm should be designed to minimize it (Yuan 1996).
Distributed and Parallel Systems