Assignment 4: Comparing Systems

How do we tell if one computer is faster than another?  Will a 2.4 GHz Intel processor run twice as fast as a 1.2 GHz PowerPC?  If computer A runs program P twice as fast as computer B, can we expect computer A to run program Q twice as fast as well?  To give you some appreciation of the subtleties of these questions, this assignment will have you actually benchmark several different machines with several different applications. 

You are to work in teams of one, two, or three (your choice).  Each team must send an e-mail message listing its members to cs252 by noon on Tuesday, March 6.  Please use the subject [cs252] Assignment 4 Group- uname1, uname2, uname3 where uname is your user name.  You will lose points if you fail to do this. 

There are lots of extra credit opportunities on this assignment.  If you’re looking for some points this would be an easy way to get some. 

We are giving you three benchmark programs, all written in C:

  • pi.c computes the first N digits of pi, for your choice of N.  It does no floating-point arithmetic, and very little in the way of memory access, but it does a lot of integer math. 
  • mandel.c plots the Mandelbrot function (the most famous fractal) for your choice of origin, radius, and image size.  It does a lot of floating-point math, but relatively little integer math or memory manipulation.  By default the program computes the image and then just throws it away.  If you give it a -p command-line argument it will print the image to the standard output in pgm greyscale format, readable by (among other things) the Linux gimp utility.  You can use “> file.pgm” on the command line to save the output to a file.  To see the result with the default parameters, click here
    Mandel requires the standard math library, which needs to be linked correctly during the compilation process.  Make sure to use the -lm flag (as listed in the source header comment) when compiling. 
  • transpose.c transposes a large matrix (your choice of size), the way one might to, say, print on a landscape mode printer.  It does no floating-point math, very little integer math, but a lot of memory access. 

You are to compile and run all three programs on machines with three different ISAs.  The easiest three to pick are probably a PC (x86), a Sun (SPARC), and a (not-too-recent) Macintosh (PowerPC).  You are not to choose any two machines with the same ISA (e.g., a Pentium III and a Pentium 4).  You may want to choose team members who are familiar with machines you don’t use much, and vice versa. 

C compilers are readily available on PCs and Suns in the CSUG labs, and on various public machines around campus.  (Most of the csug machines are Pentium 4s, but harley is a Sun, and you should be able to connect to it via ssh.  It’s a bit old and slow, so if you have access to machines on the CS research network or elsewhere on campus you might want to use them instead.)  You can use gcc to compile programs on x86/Linux, Sun (Sparc/Solaris), and Mac OS X machines.  C compilers are also available for Windows machines, via Visual Studio or Cygwin, but you probably don’t want to use those unless someone on your team is already familiar with them. 

For each program on each machine, you should measure program run time in seconds.  Read the header of each source file for information on run-time parameters.  Choose parameters that give you run times on the order of a minute, so you have 2–3 significant digits of accuracy if measuring with a stopwatch.  Be sure to use the same parameters on all machines, so you can make head-to-head comparisons.  Note that for the transpose program the n and t parameters change run time in different ways; your comparative performance across machines may depend on the mix of n and t

You should try to make sure that the machine you are using is otherwise unloaded at the time you collect your results, and you should probably take the minimum over several runs in order to weed out the effect of other random activity stealing time away from your benchmarks. 

On Unix-based systems (including Linux, Solaris, and Mac OS X) you can measure run time using the time command.  The reported “real” time is what you want, assuming the system is lightly loaded.  It should be approximately equal to the sum of the ‘system’ and ‘user’ times.  If it’s not, the machine isn’t lightly loaded. 

You should also make note of the advertised clock speed in MHz for each machine.  On most machines there is a software tool or configuration file that will give you this information.  On a Sun, run psrinfo -v.  On a Mac, run the Apple System Profiler (the “About This Mac” item in the Apple Menu).  On a Linux box, read the file /proc/cpuinfo

What to hand in

After collecting and discussing results within your group, you are to describe those results in a file named README.pdf, in Adobe Portable Document Format.  I assume that you will discuss the content extensively with your partner (if any), but you must compose and hand in your own writeup— no sharing of text allowed.  This will help convince me that everyone understands the results. 

You may create your writeup in whatever word processor you like, but you must convert it to PDF.  Uniformity of format makes life much easier for the graders, and almost anything can be converted to PDF.  One possible route is to generate postscript by “printing” to a file, and then run the postscript through ps2pdf on the csug machines.  Be sure to preview your file with acroread on the CSUG machines prior to turning it in.

If you don’t have a favorite word processor, I strongly recommend that you learn to use LaTeX.  It has a bit of a "learning curve", but it’’ the document system of choice for all of mathematics and most of computer science.  Every CS major should know how to use it.  You can easily locate Latex tutorials on the net.  If you want to become an expert, I recommend the Guide to LaTeX by Helmut Kopka and Patrick W. Daly.  Note that if you use LaTeX you can generate PDF directly using pdflatex

You may create your bar graphs using any tool you like, so long as you incorporate the output into your PDF file.  The Linux gimp tool can convert among many different graphics formats. 

Your README.pdf file must contain: 

  1. Another list of the members of your group (even though the writeup is your own). 
  2. A description of the machines you used, and of how you collected your numbers (including how you measured time).  Your machine description should include model and make, processor speed, amount of memory, and anything else you think might be relevant (speed of system bus, amount of cache, OS release, etc.). 
  3. A description of the compiler that you used (per machine).  In particular, the name and version of the compiler, and any flags that you compiled with. 
  4. Four bar graphs, one for each program, and one for clock speed.  In each graph there should be three individual bars, one for each machine.  To make the results as intuitive as possible (“bigger bars are better”), plot digits per second for the pi program, pixels per second for the mandel program (remember that the number of pixels is the square of n), and megabytes per second for the transpose program (where the number of bytes transposed is sizeof(int) == 4 times the square of N, times the number of iterations). 
  5. A narrative discussion of your results.  What did you learn about the nature of performance?  Which machine(s) were fastest for which programs/metrics?  What might explain these results? 

To turn in your writeup, put it in a directory by itself and run

        /u/cs252/bin/TURNIN . comparing_systems
Watch the WebCT forum for updates. 

Extra Credit suggestions

  • Try additional programs.  Based on what you have seen with the provided programs, try to predict how other programs will compare across machines.  How accurate were your predictions? 
  • Try different combinations of n and t in the transpose program.  Plot run time against n or t or both on a single machine, and then on multiple machines. 
  • Compare machines within a family: Pentium III v. Pentium IV v. Athalon, or G3 v. G4 v. G5 (or iBook v. PowerMac). 
  • Compare multiple compilers for the same machine, if you have access to them.  Or try different optimization levels with the same compiler.  Do some research into what the compiler actually does with different levels of optimization, and correlate that to the differences you see between the different programs on different optimizations. 
  • The provided benchmarks were designed to stress integer, floating point, and memory performance.  Craft a benchmark (or set of benchmarks) designed to stress I/O performance.  How do the results compare to those obtained with the provided benchmarks? 
  • Visit spec.org and learn about industry standard benchmarks.  How do your results compare with the pulished SPEC results? 

DUE DATES:

For the list of group members: noon, Tuesday, March 6. 

For the final write-up: noon, Friday, March 9. 


Last Change: 13 March 2007 / Michael Scott's email address