Parallel Computing

Parallel Programming is programming in a language that allows you to explicitly indicate how different portions of the computation may be executed concurrently by different processors.

PARALLEL PROGRAMMING in C with MPI and OpenMP
Michael J. Quinn

Parallel Processing, the concurrent or simultaneous execution of two or more parts of a single computer program, at speeds far exceeding those of a conventional computer. Parallel processing requires two or more interconnected processors, each of which executes a portion of the task; some supercomputer parallel-processing systems have hundreds of thousands of microprocessors. The processors access data through shared memory. The efficiency of parallel processing is dependent upon the development of programming languages that optimize the division of the tasks among the processors.

The Columbia Electronic Encyclopedia, Sixth Edition
Copyright © 2003, Columbia University Press.

Some History

'Those responsible for the designs of the very first electronic computers with internally stored (and hence machine modifiable) programs, namely von Neumann and Turing, were well aware of the significance of parallelism both in performing elementary operations, and in overlapping them with other operations such as input and output. '. . . the extent of concurrency in actual early machines was limited more by engineering considerations (of the day), than by the aspirations of some of the designers.'

Parallel processing in large-scale simulation: motivation and methods
by Anthony W. Burton
Lucas Automotive Ltd.

STRETCH (the IBM 7030) is the largest, fastest, operating general purpose computer. It combines fixed word length arithmetic for performing floating point operations with the flexibility of variable word length arithmetic in which the words can be composed of "bytes" with from one to eight bits in a byte. The logic includes "look ahead" of instructions, in which as many as four instructions can be processed simultaneously. The arithmetic unit is extremely fast and the main memory has a two micro-second cycle time with multiple memory modules operating simultaneously. It is extremely difficult, however, to estimate accurate timing for individual instructions because of the complexity of "look ahead" plus the variable length of time required to execute VFL (variable field length) instructions. A reasonable estimate, however, is that approximately 1,000,000 instructions per second can be executed.

G.R. Trimble, "STRETCH," Computer Usage Communique, 1963

'Figure ... shows the architecture for a satellite processor configuration. The purpose of such a configuration is to improve system throughput by offloading processes from the central processor and executing them on the satellite processors. Each satellite processor has no local peripherals except for those it needs to communicate with the central processor...'