MPI

MPI, short for Message Passing Interface, is a standardized and portable library specification for parallel programming in distributed-memory systems. It was developed in the early 1990s by the MPI Forum, a consortium of researchers and vendors, to provide a unified standard for high-performance computing (HPC) applications. MPI is widely used in scientific simulations, data analysis, and large-scale computational tasks. Implementations such as Open MPI and MPICH provide the libraries, headers, and runtime environments for Linux, macOS, and Windows, and programs are compiled with standard compilers using commands like mpicc for C or mpic++ for C++.

MPI exists to enable communication between processes running on different nodes in a distributed system, allowing scalable parallel computation. Its design philosophy emphasizes explicit message passing, portability, and high performance, giving developers control over data distribution, synchronization, and parallel execution while maintaining consistency across heterogeneous HPC environments.

MPI: Point-to-Point Communication

Point-to-point communication uses functions like MPI_Send and MPI_Recv to transfer messages between processes.

# include the MPI header
#include <mpi.h>
#include <stdio.h>

int main(int argc, char** argv) {
int rank, size;
MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);

int data = rank * 10;
if(rank == 0) {
    MPI_Send(&data, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);
} else if(rank == 1) {
    MPI_Recv(&data, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE);
    printf("Received: %d\n", data);
}

MPI_Finalize();
return 0;

}

These functions enable explicit communication between pairs of processes, providing fine-grained control over data exchange. This approach is similar to collective operations in MPI Collective contexts or the abstractions in OpenMP for shared-memory systems.

MPI: Collective Communication

Collective communication involves operations like broadcast, scatter, gather, and reduction across all processes in a communicator.

# example in C using MPI_Bcast
int value;
if(rank == 0) {
    value = 42;  # root process sets value
}
MPI_Bcast(&value, 1, MPI_INT, 0, MPI_COMM_WORLD);
printf("Rank %d received value %d\n", rank, value);

Collective operations simplify synchronizing and distributing data efficiently to multiple processes. These are foundational concepts in parallel computing and align with abstractions in MPI-IO and Parallel Algorithms.

MPI: Derived Data Types and Communication

MPI allows creation of derived data types to send complex structures in a single message.

typedef struct {
    double x, y, z;
} Point;

MPI_Datatype MPI_POINT;
MPI_Type_contiguous(3, MPI_DOUBLE, &MPI_POINT);
MPI_Type_commit(&MPI_POINT);

Point p;
MPI_Send(&p, 1, MPI_POINT, dest, tag, MPI_COMM_WORLD);

Derived types reduce the complexity of sending structured data, enabling efficient and type-safe communication. This feature complements file I/O and process management in large-scale HPC workflows.

MPI is used extensively in scientific simulations, numerical computing, and distributed data analysis. It integrates with high-level libraries such as MPI for Python (mpi4py) and Cray MPICH to support modern HPC applications. When combined with OpenMP and Parallel Algorithms, MPI provides a robust environment for scalable, parallel computation across distributed systems.