CSC 2/458: Build Notes

Grads

The installed base compilers on the Sun/Sparc (i.e., swym,sync,niagara1) machines are typically a few generations old. To use our tested versions please include the following blurb in your .cshrc file in your home directory. You may ask what does it do ? Essentially, it gets the compiled program to use gcc 4.1.1 libraries and binary when compiling.

if ($?prompt) then                                                               
   if (`uname` == "SunOS") then                                                  
        setenv PATH /usr/grads/share/gcc/bin:${PATH}                             
        setenv LD_LIBRARY_PATH /usr/grads/share/gcc/lib:${LD_LIBRARY_PATH}       
    endif                                                                        
endif                 

Undergrads and Non-CSers

You might have found out by now that the niagara2 boxes lack the appropriate make, compiler and other build files. This is because the sysadmins haven't set up the default .cshrc file yet, like on the grad niagara1. Typically you will find at the top of your .cshrc the following source /usr/staff/lib/defaults/system.cshrc The system.cshrc is machine specific so Sun boxes get something different from intel.

To work around, I have installed a custom gcc-4.1.1 at ~/cs258/installed/gcc-4.1.1 I have shared it with you folks. To set up the appropriate build files, include the following in your .cshrc


if ($?prompt) then
   if (`uname` == "SunOS") then
        setenv PATH /u/cs258/installed/gcc-4.1.1/bin/:/usr/ccs/bin:/usr/bin:/usr/sbin/:"$PATH"
        setenv LD_LIBRARY_PATH /u/cs258/installed/gcc-4.1.1/lib:/usr/staff/lib

    endif
endif

(Important: Also comment out the "source /usr/staff" at the top of the .cshrc. Commenting out means prefix the statement with a "#". Why, because otherwise the rest of the .cshrc wont load)

I have also created a new makefile for niagara2. Makefile.niagara2 This file exists only for the sor app. gauss doesn't include any machine specific specifications. Although, you can include this flag if you like. This includes -mcpu=v9 option which applies the SPARC v9 ISA specific optimizations.

Makefiles

When writing Makefiles, model it based on the the machine specific makefiles (e.g., Makefile.niagara1/2) provided as part of the 1st assignment. To use a specific makefile type make -f Makefile. < machine >. These makefiles include machine specific compiler options. For example, our nigara's run Solaris SunOS 5.10 which recommend programs to pass the -mcpu=v9 flag to the compiler.


Turn In

USING THE TURN-IN SCRIPT

The turn-in script will recursively copy all files and subdirectories of your current directory to a location in the course's account. Please avoid running it at your home directory or any other directory that contains files irrelevant to the current project. For any problems while using it please contact the TA with an email (sending the script's output will help).You may turn-in your project as many times as you need. However, only the MOST RECENT submission will be graded. DO NOT FORGET to submit a README file (text or pdf format) with your code, and include your name and e-mail address in the README file. Check the appropriate assignment page for information related to the contents of the README file.


INSTRUCTIONS:

* From the grad network:

1) Go to the directory that contains ALL the files that are related
to your current project.

2) Run the script:

/u/cs458/bin/turnin

Upon successful completion you will get an output of the form:

File Submission completed successfully at: [CURRENT DATE] Where CURRENT DATE represents the current date and time. You will also receive through email a message with subject ``Submission Notification'', which will contain a listing of all the files you submitted along with their last modification date and size.



Assignment #3 Build Notes

In this assignment you'll be working with four different programming models (1) MPI, (2) OpenMP, (3) Cilk. Besides the difference of message passing and shared memory, MPI is a library interface whereas OpenMP and Cilk consists of a set of directives that need special compiler support. Although both of them are language independent we'll be mostly targeting them on C/C++. In this assignment your working environment will be on any of the following department machines.

Obviously, tool-hardware compatibility is an important issue. We have set up a variety of compiler-hardware setups at /u/cs2(or4)58/installed. Each specific compiler needs you to set up the PATH and LD_LIBRARY_PATH variables.

OpenMP

Setup

gcc supports openmp from 4.1.2 on the x86-Linux but only from 4.2 on Sparc-Solaris. 4.1.2 is the default compiler on cycle1. To use 4.2 on the sun boxes; sync and swym its installed under /usr/grads/share/gcc-4.2.0. . For niagara1 its installed under /u/cs4(or 2)58/installed/gcc/4.2.0/niagara. For node4x2a its installed under /u/cs4(or 2)58/installed/gcc/4.2.0/x86_64 To set-up use some script like the following. Add a specific if for each machine.

if ( `hostname` == sync.cs.rochester.edu )  then
   setenv GCC_HOME /usr/grads/share/gcc-4.2.0
   setenv PATH ${GCC_HOME}/bin:${PATH}
   setenv LD_LIBRARY_PATH ${GCC_HOME}/lib:${LD_LIBRARY_PATH}
endif

How to Compile

In addition to using gcc 4.2, you need to compile with the -fopenmp command-line option:
 gcc -fopenmp -o hello hello.c 

How to program

For programming guide you may refer to this tutorial. We have also included a simple hello world program at /u/cs4(or 2)58/test_omp.

MPI

MPI is a message-passing library in which the programmer is expected to explicitly specify the communication and transfer of data between execution threads. The data communication can be implemented using a number of underlying protocols, shmem (shared memory on niagara,sync,node4x2a,discovery) or TCP/IP (cluster machines node33-64).

Setup

The MPI versions have been set up at, /u/cs4(or 2)58/installed/MPI/1.2.7/ < machine > where < machine > is discovery,niagara,x86_64(node4x2a). On sync/swym filesystem you will find it at /u/cs458/installed/mpi/1.2.7/sync/ . For the node machines you will find it at /u/cs458/installed/mpi/1.2.7/nodes. UGrads and Non-CSers for your i386 machines (cycle and similar) use /u/cs258/installed/mpi/1.2.7/nodes, for niagara2 use /u/cs258/installed/mpi/1.2.7/niagara. Similar to gcc use some script like the following in your .cshrc.

foreach n (`seq -s " " 33 64`)
    if (`hostname` == node$n.cs.rochester.edu) then
        setenv MPI_HOME /u/cs458/installed/mpi/1.2.7/nodes
    endif
end

if ( `hostname` == sync.cs.rochester.edu )  then
   setenv MPI_HOME /u/cs458/installed/mpi/1.2.7/sync/
endif

setenv PATH ${MPI_HOME}/bin:${PATH}
setenv LD_LIBRARY_PATH ${MPI_HOME}/lib:${LD_LIBRARY_PATH}

How to compile

Basically, like gcc, mpicc could run the errands of both compilation and linking. A sample Makefile and program can be found at /u/cs458/test_mpi. You may use the directory as a template and plug in your own programs. You also need to specify the include path to help mpicc find the header "-I/u/cs458/$MPI_HOME/include/".

How to program

Take a look at MPI tutorial from Lawrence Livermore National Lab.

How to run

mpirun is the script for launching your mpi tasks. The syntax is
 mpirun -np  -machinefile   
np specifying the number of processors to use and the machine file listing the processors to be selected from.

NOTE: On the graduate network, any jobs submitted to the cluster nodes ranging from node33 to node64 are subject to centralized management. For detail instructions, please refer to ClusterJobs. In short, you need to wrap your jobs in a shell script and then pass it as an argument to qsub. Here's a script that could be used in submitting your MPI jobs.

#!/bin/csh
source ~/.cshrc.mpi

cd $PBS_O_WORKDIR
#set your machinefile
cat $PBS_NODEFILE > ./cluster_nodes 
#count how many nodes you reserve

#run mpi on these nodes
mpirun -np `cat ./cluster_nodes | wc -l` -machinefile ./cluster_nodes
./linux/ < your-mpi-program >  >  ./mpi.out
Now you ssh to node64.cs.rochester.edu, enter your mpi working directory and then copy and fix run.sh. To run your mpi program, just type qsub -l nodes=4 script.sh

CILK

Cilk is a set of language extensions that allow users to specify parallelism points without worrying about threads or scheduling issues. The cilkc program that you can use to compile .cilk programs into executables.

How to set up

We have installed three different versions of cilk system: i386 /u/cs4(or 2)58/installed/cilk/5.4.6/x86 (note i386 machines not x86_64 or node4x2a), Niagara /u/cs4(or 2)58/installed/cilk/5.4.6/niagara and sync (only on Grad side) /u/cs458/installed/cilk/5.4.6/sync/.

Here's an example to set the path. 

if ( `hostname` == cycle1.cs.rochester.edu )  then
    setenv CILK_HOME /u/cs458/installed/cilk/5.4.6/x86
endif

if ( `hostname` == cycle1.csug.rochester.edu )  then
    setenv CILK_HOME /u/cs258/installed/cilk/5.4.6/x86
endif

if ( `hostname` == niagara1.cs.rochester.edu )  then
   setenv CILK_HOME /u/cs458/installed/cilk/5.4.6/niagara/
endif

if ( `hostname` == niagara2.cs.rochester.edu )  then
   setenv CILK_HOME /u/cs258/installed/cilk/5.4.6/niagara/
endif

if ( `hostname` == sync.cs.rochester.edu )  then
   setenv CILK_HOME /u/cs458/installed/cilk/5.4.6/sync/
endif


setenv PATH ${CILK_HOME}/bin:${PATH} 
setenv LD_LIBRARY_PATH ${CILK_HOME}/lib:${LD_LIBRARY_PATH}

How to compile

Similar to gcc, cilkc takes in a number of parameters substitutes the cilkc annotations with C routines and subsequently uses the gcc to generate the executable. Refer section 2.2 (page 7) in the cilk manual Cilk Manual"

 Example: cilkc -O2 hello.cilk -o hello 
We have also included a cilk2c program that performs a source-source conversion of cilk programs to .c programs.

How to run

Cilk programs take an extra "--nproc' argument along with your standard program specific variables. Cilk works only on shared memory machines.

How to program

The path /u/cs4(or 2)58/installed/cilk/5.4.6/examples includes a number of .cilk programs for your reference. Take a look at the Cilk Manual" as well. We have also included a simple Fibonacci program at /u/cs4(or 2)58/test_cilk.