How to run MPI jobs with PBS/Pro
Here is a quick step-by-step guide to getting started running MPI jobs using the Intel compiler suite on Eureka and Yucca.
We want to run an MPI job that uses a total of 64 processes (cores). We also want to limit to 8 the number of processes running on each node (this allows us the flexibility of controlling how the system allocates the compute cores so we can have OPENMP threads or other special needs taken into account.
To compile a simple "hello world" mpi program (after logging into Eureka):
module add intel/intel-12-impi # activate the Intel compiler suite cp /share/apps/intel/impi/4.0.3.008/test/test.c test.c # make a copy of the sample hello world program mpicc test.c -o testc # compile the sample program
Create a file called testc.pbs with the following (starting in column 1):
#!/bin/bash #PBS -l select=8:ncpus=8:mpiprocs=8 cd $PBS_O_WORKDIR module add intel/intel-12-impi echo echo The following nodes will be used to run this program: echo cat $PBS_NODEFILE echo mpirun ./testc exit 0
The line #PBS -l select=8:ncpus=8:mpiprocs=8, controls how the system allocates processor cores for your MPI jobs.
- select=# -- allocate # separate nodes
- ncpus=# -- on each node allocate # cpus (cores)
- mpiprocs=# -- on each node allocate # cpus (of the ncpus allocated) to MPI
By varying the above, you can control how cpu resources are allocated, The above example allocates 64 cores all of which are for use by MPI (8 nodes with 8 cpus on each node).
If, for example, your program is hybrid MPI/OPENMP program that runs 8 MP threads on 4 mpi control processes, you would use something like: #PBS -l select=4:ncpus=12:mpiprocs=4.
To submit the test job:
qsub -q compute testc.pbs