There are several groups of machines you may use for assignments/project in this course. Note that some of these machines are heavily used for research purposes, so it is important to follow the policies when using them.
node17, node19, node20, ..., node28
Each machine has two CPU chips, each containing two hardware hyperthreads.
So each machine includes four processors in total.
These machines are relatively slow and they are not as heavily used as other machines.
There is no specific policy about using them and they are good places to develop and
debug your programs.
node33, node34, ..., node72
Each machine also has two CPU chips, each containing two hardware hyperthreads.
But these machines are more powerful and they are more heavily used.
You need to use qsub on node64 to run jobs on these nodes.
As a quick reference, after logging on node64,
% qsub -I -l nodes=nodexx
This will reserve and give you a shell on a specific node.
For further information, please read and follow the
Node Usage Policy.
node4x2a
This machine has four CPU chips, each containing two cores, each further containing
two hardware hyperthreads. So each machine includes 16 processors in total.
Given its uniqueness, it can be heavily used at times. You should be courteous to
others who also use this machine. Specifically, you should always check whether the
machine is being used before running your program. Since this machine was purchased
primarily for research purposes, you should give priority to those who use it for
research.
In order to build and test your MPI programs in the experiment machines, there are a few steps you need to follow.
Set up environment variables
Several MPI versions have been installed at different directories.
You need to choose the right version depending on the test machine.
If you use C-shell or one of its variants, the following shell script should select
the right MPI version. For automatic setup, you can add the script to
.cshrc in your home directory.
# if you are using nodes
foreach n (`seq -s " " 17 72`)
if (`hostname` == node$n.cs.rochester.edu || `hostname` == node$n) then
setenv MPI_HOME /u/kshen/install/mpich-1.2.6-install
endif
end
# if you are using node4x2a
if (`hostname` == node4x2a.cs.rochester.edu || `hostname` == node4x2a) then
setenv MPI_HOME /u/cs458/installed/mpi/1.2.7/x86_64
endif
setenv PATH ${MPI_HOME}/bin:${PATH}
if ( "$?LD_LIBRARY_PATH" ) then
setenv LD_LIBRARY_PATH ${MPI_HOME}/lib:${LD_LIBRARY_PATH}
else
setenv LD_LIBRARY_PATH ${MPI_HOME}/lib
endif
If you do not use C-shell or you want to have customized control of your shell environment,
please feel free to do it in your own way. In that case, reading the above script
should at least help you figure out what you need to do.
Specify RSH connection
When running MPI on multiple nodes, you need to specify a host file for the RSH connections which
MPI needs. You need to create a .rhosts file in your home folder, specifying the host
machines you will start mpirun. (Say you will start mpirun in node17 to
run on node19, node20 ..., you need to add node17 to this file.) After you create
the .rhosts file, remember to change the permission of the file to 600 by running
chmod 600 .rhosts.
Compile/link your program
Instead of gcc, you should use mpicc to compile and link your
MPI programs. A sample Makefile can be found in the sample MPI version of SOR
at /u/cs458/apps/sor/mpi/.
Run make clean; make to compile your code on any of the experiment platforms we are using.
You can follow this makefile to compile and link your own programs.
NOTICE: if you are testing on higher nodes (node33-node72), you should compile your program on lower
nodes (node17-node28).
Run your program
You need to create a file to indicate the available processors for running MPI programs.
The file contains the names of the machines and the number of available processors on
each machine. A sample file (hosts) can be found in /u/cs458/apps/sor/mpi/.
For a specific example, if your system contains two machines (node17/node19)
with two processors on each machine, then you file may look like this:
node17.cs.rochester.edu:2
node19.cs.rochester.edu:2
You can run your MPI program in the following way:
mpirun -np 2 -machinefile hosts linux/sor
Flag -np is followed by the number of MPI processes in the run.
Flag -machinefile is followed by the hosts file described above.