Due by 11:59pm, Friday, October 11.
You earn 5% bonus if you make your last (therefore final) turn-in by 11:59pm, Wednesday, October 9. You earn 10% bonus if you make your last turn-in by 11:59pm, Monday, October 7.
Please direct your questions to the TA about this assignment.
In this assignment, you will program three applications on the Hadoop parallel data processing platform. This will allow you gain practical experience on MapReduce programming and learn the performance implication of parallel data processing.
HDFS is made of a single NameNode and multiple DataNodes. The NameNode tracks the location of the files in the DataNodes and is responsible for coordinating them to ensure consistency of data. When a user wishes to read data, it asks the NameNode for which DataNodes hold the data and then contacts the DataNodes to get the data. Similarly, when writing data the user asks the NameNode where to store the data and then uploads the data directly to the DataNodes. To handle large files, HDFS splits files into blocks of uniform size. DataNodes hold blocks instead of files, and blocks from a single file may be spread across multiple DataNodes.
MapReduce is made of a single JobTracker and multiple TaskTrackers. Users write MapReduce jobs, which consist of two types of tasks: maps and reduces. The JobTracker takes a description of these tasks and schedules them across the TaskTrackers. One TaskTracker runs on each DataNode, and the JobTracker ensures that tasks run close to the nodes that have the data for them.
In our cluster, the NameNode and JobTracker both run on node17.cs.rochester.edu. On larger clusters, these servers run on separate computers. A DataNode and TaskTracker run on node18, node20, node21, node23, node24, node25, node26, node27 and node28. From the university network, you can check the status of the HDFS cluster by visiting http://node17.cs.rochester.edu:50070/ which will display that status of the DataNodes and allow you to view the files in the cluster. The status of the MapReduce cluster is at http://node17.cs.rochester.edu:50030/ which will show the status of the TaskTrackers and the jobs that will run, are running, and have run. There will be a link for every job which will display the current status, number of tasks, and various statistics for the run.
You do not need to log into any of these machines to run your programs. You instead run your program on another machine on the research network such as cycle1, cycle2, or cycle3 and the program will connect to the NameNode and JobTracker as needed. More details on how to do this and how to read the status of the MapReduce cluster are below. If you are interested in more detail about the architecture of HDFS, you can read the HDFS Architecture Guide.
The Hadoop API is the place to check for detailed information on what function to call. Some of the more useful classes are JobConf and the InputFormat subclasses. You will have to implement subclasses of Tool, Mapper, and Reducer. Extending MapReduceBase will give you empty versions of functions not all map or reduce tasks need.
All of the official documentation can be found at the main documentation index.
/input/gutenberg
directory in the cluster. Some words
used in all of
the files are: for, one, thousand, and years. Some words used in only a
single file: academics, faints, tag, and yams.
We provide another dataset in /input/webster.1
for performance
testing.
FileSplit fileSplit = (FileSplit) reporter.getInputSplit(); String filename = fileSplit.getPath().getName();
matrix.dat 1 1 1.1 1 2 2.2 2 1 3.3 2 2 4.4 vector.dat 1 2The vector and matrix files can be found in HDFS at
/input/vector.dat
and /input/matrix.1
. The first entry in the result matrix should
be 10240.0.
/input/points.1
.
It was generated at random in 5 clusters. The TA's code stopped after 7 iterations and
was able to find one of the clusters near 665539, 159651. Due to the way k-means
works, your code may have slightly different coordinates than that.
JobConf
for iterations. If you don't
call JobConf.setJarByClass
with the job's class, it can't be reused for
multiple iterations.
src/
. After running ant
, it will create a jar file
build/Hadoop.jar
.
You can run a class from that jar file with the following command:
~cs576/hadoop-1.2.1/bin/hadoop jar build/Hadoop.jar <class> <arguments>Where
class
is the name of your main class and the
arguments
are passed to your program. The hadoop
program
will automatically read the configuration to run your program on the cluster.
It will output a log file containing any errors and a simple progress meter.
At the end it will output a small pile of counters. You may find the following
useful when attempting to debug your programs:
* Map input records * Map output records * Combine input records * Combine output records * Reduce input records * Reduce output recordsMapReduce tasks generally expect their input and output to be in the HDFS cluster. You can upload a file with the command
~cs576/hadoop-1.2.1/bin/hadoop fs -put <file> .
,
which will upload file
into your home directory on the HDFS cluster.
If this fails, you may need to create your home directory with the command
~cs576/hadoop-1.2.1/bin/hadoop fs -mkdirs /user/<username>
,
where username
is your username on the department computers.
Similarly, files can be downloaded via
~cs576/hadoop-1.2.1/bin/hadoop fs -get <file> .
, which will download
the file from HDFS to your current directory.
Each mapreduce task must have an output path, which must not exist before running
the program, that will be created as a directory in HDFS. That directory will
contain logs of your program. The output from the reduce tasks will be saved as
part-00000
(large outputs will create part-00001
and so
on, but that should not be required for this assignment). If your output is a
text file, you can read it without downloading it with the command
~cs576/hadoop-1.2.1/bin/hadoop fs -cat <output_directory>/part-00000
(where output_directory
is the path set by your program). If you
used a SequenceFileOutputFormat, you can get a text version with
~cs576/hadoop-1.2.1/bin/hadoop fs -text <output_directory>/part-00000
.
You can check the log of your program for a "Launched map tasks" counter, which will tell you how many tasks were run at once to handle your data.
~cs576/hadoop-1.2.1/bin/hadoop job -list
. If someone
is running a job already, please wait until it is complete to start your own.
Conversely, please give some space between your jobs so others have a chance to run.
If your code has a bug in it that causes it to run indefinitely, then you can
stop it with the command ~cs576/hadoop-1.2.1/bin/hadoop job -kill <job_ID>>
where job_ID
is the ID given in the log at the start of your job,
on the cluster status page, or in the output of the job list command above.
Please note that simply killing the program running on your own computer is
not enough to stop the job on the cluster.
If you suspect that someone else's program is left hanging in the system,
please let the TA know.
You should electronically turn in the required files. Instructions for electronic turn-in can be found on the course Web site.