An Introduction to
Parallel and Distributed Systems:

"As a cell design becomes more complex and interconnected a critical point is reached where a more integrated cellular organization emerges, and vertically generated novelty can and does assume greater importance."     Carl Woese Professor of Microbology, University of Illinois "At the highest level, we're looking at 'scaling out' (vs. 'scaling up,' as in frequency), with multicore architecture. Basically, instead of having one big x86 processor, you could have 16, 32, 64, and so on, up to maybe 256 small x86 processors on one die. We'll have the transistor count (thanks to Moore's Law) to do incredible things we could only dream about a few years ago."     "In future architectures, we'll have multiple devices working in parallel. That means you can break a problem up to be solved in pieces. In these new architectures, the biggest problem won't be building the hardware. The biggest challenge will be helping people configure algorithms to solve problems in parallel, and providing the software tools to extract that inherent parallelism out of the programs." Stephen S. Pawlowski Chief Technology Officer of Intel's Digital Enterprise Group

1. Background
   1. Processes and Threads,sockets and shared memory.
   2. Parallel Computing
   3. Hardware Parallelism
   4. Functional Parallelism

 

2. Super Computers and General Purpose Parallel Processing Computers
   1. A General Purpose Parallel Processing Computer - The Beowulf Cluster
   2. Parallel Architectures - The Course Textbook, Chapter 2

 

3. MPI The Message Passing Interface
   1. hello_COMM_WORLD.c - Ordered Message Passing
      1. The Terminal View
      2. A Communications Sketch
      3. The Code
      4. WORLD_hello_COMM.c - First Come First Serve
             The Terminal View - The Code
   2. Computing pi
      1. Numerical Integration
      2. pirules.c - Broadcast-Collect
      • The Terminal View
      • A Communications Sketch
      • The Code
        
      • pirulesfaster.c
            The Terminal View - The Code
   3. Notes from Chapter 4 of the Textbook
   4. Finding Primes
      1. Moore's Law and Computational Complexity
      2. Guest Lecture - Professor Bhatia
      3. The Gnu Multiple Precision arithmetic library
         • The Makefile
         • Some Basic Arithmetic     (The TerminalView)
      4. prime-gmpmpi.c
         • The Terminal View
         • A Communications Sketch
         • The Code
   5. Debugging MPI Programs
      1. Debugging The Old Fashoned Way - with printf()
         • The Terminal View + Debug Files
         • prime-debug.c
           
      2. Using TotalView®

    

4. Parallel Algorithm Design
   1. Foster's Design Methodology -The Course Text, Chapter 3
      • Hypercubes
      • A sketch of logpirules
   2. Limiting Factors in Massively Parallel Processing - Ahmdal's Law-The Course Text, Chapter 7
      • What did Eugene Ahmdal say?
   3. ? Defeating RSA ?
      • The Mathematics of RSA
      • The Sieve of Eratosthenes-The Course Text, Chapter 5
      • RSA revisited
      A survey of RSA Attacks
      • The Design of RSAcracker
        • Design Documents
          • A Communications Template
        • A Serial Code Solution
      • rsa-gmpmpi.c
        
      • A Critique of RSAcracker

 

5. Monte Carlo Methods
   1. Mathematical Background
      • Randomness and Entropy
   2. Monte Carlo Methods-The Course Text, Chapter 10
   3. pimonte_simple.c
   4. pimonte_mpi.c
   5. seed_mpi.c
   6. pisprng.c
   7. PRNGs
   8. The Scalable Parallel Random Number Generators library(2.0)
   9. The Scalable Parallel Random Number Generators library(4.0)

 

6. Linear Systems and Some Analysis of Parallel Algorithms
   1. A Review of Linear Algebra
   2. Three Applications of Matrix Algebra
   3. Matrix - Vector Multiplication The Course Text, Chapter 8
   4. The MPI Alls
   5. Communications Patterns
   6. Under the Hood
      • algath.c
      • allreduce.c
      • Logarithmic Expansion

 

7. Parallel Programming on Hybrid Systems - OpenMP and MPI
   1. 
        Tutorials and Guides
      
      • Tutorial
      • Exercises
      • Sun Microsystem - OpenMPI API User's Guide
   2. 
        Some Examples
      
      • ompinfo.c
      • omp_hello.c
      • omp_reduction.c
      • pimonti_omp.c
      • terminal outputs
      • simplemult.c
      • matrixm_omp.c
      • terminal outputs
   3. Shared-Memory Programming- the Course Text Chapter 17
   4. Some other pragmas
   5. The API Specification
   6. OpenMP and MPI- the Course Text Chapter 18

1. Voyager Notes

2. logpirules notes

3. pirules code revisited

4. Lecture Notes, Tutorials, and Reference Materials:
   • Michael T. Heath, Professor and Fulton Watson Copp Chair, Department of Computer Science University of Illinois Champagne-Urbana, has kindly allowed us, this semester, to use material from his course on Parallel Numerical Algorithms. A number of my talks will directly follow his Lecture Notes.
   • An Introduction to Computational Complexity
5. Development Tools and Software:
   • The MPI Home Page. This site contains a useful programmers reference.
   • A variety of documents and tutorials on MPI
   • The OpenMP Home Page
   • The Etnus Home Page - TotalView
   • Intel Cluster OpenMP Users Guide

6. Software Libraries:
   • The Gnu Multiple Precision arithmetic library (GMP)
   • The Scalable Parallel Random Number Generators library (SPRNG) Version 2.0
   • The Basic Linear Algebra Subprograms(BLAS) and the C interface to the BLAS (CBLAS)
   • The Gnu Scientific Library(GSL)
   • Making libgmp.a and libsprng.a
7. C to HTML translations were done using CToHTML
8. The WhiteBoard

---

The Textbook for this course is:
  
Parallel Programming
   in C with MPI and OpenMP
       Michael J. Quinn
       McGraw-Hill 2004