CSC 2/458: Build Notes

Makefiles

When writing Makefiles, model it based on the the machine specific makefiles (e.g., Makefile.niagara1) provided as part of the 1st assignment. To use a specific makefile type make -f Makefile. < machine >. These makefiles include machine specific compiler options. For example, the niagara machine runs Solaris SunOS 5.10, which recommends that programs pass the -mcpu=v9 flag to the compiler.


Turn In

USING THE TURN-IN SCRIPT

The turn-in script will recursively copy all files and subdirectories of your current directory to a location in the course's account. Please avoid running it at your home directory or any other directory that contains files irrelevant to the current project. For any problems while using it please contact the TA with an email (sending the script's output will help).You may turn-in your project as many times as you need. However, only the MOST RECENT submission will be graded. DO NOT FORGET to submit a README file (text or pdf format) with your code, and include your name and e-mail address in the README file. Check the appropriate assignment page for information related to the contents of the README file.


INSTRUCTIONS:

* From the grad network:

1) Go to the directory that contains ALL the files that are related
to your current project.

2) Run the script:

/u/cs458/bin/turnin

Upon successful completion you will get an output of the form:

File Submission completed successfully at: [CURRENT DATE] Where CURRENT DATE represents the current date and time. You will also receive through email a message with subject ``Submission Notification'', which will contain a listing of all the files you submitted along with their last modification date and size.



Assignment #3 Build Notes

In this assignment you'll be working with three different programming models (1) MPI, (2) OpenMP, (3) Cilk. Besides the difference between message passing and shared memory, MPI is a library interface whereas OpenMP and Cilk consist of a set of directives that need special compiler support. Although both of them are language independent, we'll be focusing on C/C++. In this assignment, your working environment will be on any of the following department machines.

Environment Variable Setup

Please add the following to your .cshrc in order to automatically find the executables, libraries, and include files.
# Set the default home directories for MPI and Cilk
setenv MPI_HOME /u/{cs258 or cs458}/installed/mpi/3.0.2/x86
setenv CILK_HOME /u/{cs258 or cs458}/installed/cilk/5.4.6/cilk-5.4.6

if (`uname -i` == "x86_64") then                                             
     setenv MPI_HOME /u/cs458/installed/mpi/3.0.2/x86_64
     setenv CILK_HOME /u/cs458/installed/cilk/5.4.6/cilk_x86_64-5.4.6
endif


# if your target machine is niagara1, fix the home paths on GCC, MPI, and
Cilk.
if ( `hostname` == "niagara1.cs.rochester.edu" ) then
   setenv GCC_HOME /u/{cs258 or cs458}/installed/gcc/4.2.0/niagara
   setenv MPI_HOME /u/{cs258 or cs458}/installed/mpi/3.0.2/niagara
   setenv CILK_HOME /u/{cs258 or cs458}/installed/cilk/5.4.6/cilk_niagara-5.4.6
endif


# set MPI binary and library paths
setenv PATH ${MPI_HOME}/bin:${PATH}

if ( "$?LD_LIBRARY_PATH" ) then
 setenv LD_LIBRARY_PATH ${MPI_HOME}/lib:${LD_LIBRARY_PATH}
else
 setenv LD_LIBRARY_PATH ${MPI_HOME}/lib
endif


# set cilk binary and library paths
setenv PATH ${CILK_HOME}/support:${PATH}
setenv LD_LIBRARY_PATH ${CILK_HOME}/lib:${LD_LIBRARY_PATH}

Tool-hardware compatibility is an important issue. We have set up a variety of compiler-hardware setups at /u/cs2(or4)58/installed. Each specific compiler needs you to set up the PATH and LD_LIBRARY_PATH variables.

OpenMP

Setup

gcc supports openmp from 4.1.2 on the x86-Linux but only from 4.2 on Sparc-Solaris. 4.1.2 is the default compiler on cycle1. To use 4.2 on the sun boxes; on sync and swym it is installed under /usr/grads/share/gcc-4.2.0; on niagara1 it is installed under /u/cs4(or 2)58/installed/gcc/4.2.0/niagara. For node4x2a it is installed under /u/cs4(or 2)58/installed/gcc/4.2.0/x86_64

How to Compile

In addition to using gcc 4.2, you need to compile with the -fopenmp command-line option:
 gcc -fopenmp -o hello hello.c 

How to program

For a programming guide you may refer to this tutorial. We have also included a simple hello world program at /u/cs4(or 2)58/apps/test_omp.

MPI

MPI is a message-passing library in which the programmer is expected to explicitly specify the communication and transfer of data between execution threads. The data communication can be implemented using a number of underlying protocols, shmem (shared memory on niagara,sync,node4x2a,discovery) or TCP/IP (cluster machines node33-64).

Setup

The MPI versions have been set up at, /u/cs(2 or 4)58/installed/mpi/3.0.2/ < machine > where < machine > is niagara or x86.

How to compile

Basically, like gcc, mpicc performs both compilation and linking. A sample Makefile and program can be found at /u/cs458/apps/test_mpi. You may use the directory as a template and plug in your own programs. You also need to specify the include path to help mpicc find the header "-I/u/cs458/$MPI_HOME/include/".

How to program

Take a look at MPI tutorial from Lawrence Livermore National Lab.

How to run

mpirun is the script for launching your mpi tasks. The syntax is
 mpirun -np  -machinefile   
np specifying the number of processors to use and the machine file listing the processors to be selected from.

NOTE: On the graduate network, any jobs submitted to the cluster nodes ranging from node33 to node64 are subject to centralized management. For detail instructions, please refer to ClusterJobs. In short, you need to wrap your jobs in a shell script and then pass it as an argument to qsub. Here's a script that could be used in submitting your MPI jobs.

#!/bin/csh
source ~/.cshrc.mpi

cd $PBS_O_WORKDIR
#set your machinefile
cat $PBS_NODEFILE > ./cluster_nodes 
#count how many nodes you reserve

#run mpi on these nodes
mpirun -np `cat ./cluster_nodes | wc -l` -machinefile ./cluster_nodes
./linux/ < your-mpi-program >  >  ./mpi.out
Now you ssh to node64.cs.rochester.edu, enter your mpi working directory and then copy and fix run.sh. To run your mpi program, just type qsub -l nodes=4 script.sh

CILK

Cilk is a set of language extensions that allow users to specify parallelism points without worrying about threads or scheduling issues. The cilkc program that you can use to compile .cilk programs into executables.

How to set up

We have installed three different versions of the cilk system: i386 /u/cs2(or 4)58/installed/cilk/5.4.6/x86 (note: on i386 machines, not x86_64 or node4x2a), niagara /u/cs2(or 4)58/installed/cilk/5.4.6/niagara, and /u/cs2(or 4)58/installed/cilk/5.4.6/x86_64/.

Here's an example to set the path. 

if ( `hostname` == cycle1.cs.rochester.edu )  then
    setenv CILK_HOME /u/cs458/installed/cilk/5.4.6/x86
endif

if ( `hostname` == cycle1.csug.rochester.edu )  then
    setenv CILK_HOME /u/cs258/installed/cilk/5.4.6/x86
endif

if ( `hostname` == niagara1.cs.rochester.edu )  then
   setenv CILK_HOME /u/cs458/installed/cilk/5.4.6/niagara/
endif

if ( `hostname` == niagara2.cs.rochester.edu )  then
   setenv CILK_HOME /u/cs258/installed/cilk/5.4.6/niagara/
endif

if ( `hostname` == sync.cs.rochester.edu )  then
   setenv CILK_HOME /u/cs458/installed/cilk/5.4.6/sync/
endif


setenv PATH ${CILK_HOME}/bin:${PATH} 
setenv LD_LIBRARY_PATH ${CILK_HOME}/lib:${LD_LIBRARY_PATH}

How to compile

Similar to gcc, cilkc takes in a number of parameters substitutes the cilkc annotations with C routines and subsequently uses the gcc to generate the executable. Refer section 2.2 (page 7) in the cilk manual Cilk Manual"

 Example: cilkc -O2 hello.cilk -o hello 
We have also included a cilk2c program that performs a source-source conversion of cilk programs to .c programs.

How to run

Cilk programs take an extra "--nproc' argument along with your standard program specific variables. Cilk works only on shared memory machines.

How to program

The path /u/cs4(or 2)58/installed/cilk/5.4.6/examples includes a number of .cilk programs for your reference. Take a look at the Cilk Manual" as well. We have also included a simple Fibonacci program at /u/cs4(or 2)58/apps/test_cilk.