

1. Choose models
==================

All models need the data from ibm1 model1, so you have to run ibm1 first: set run_ibm1_classic="true". You will see the model output from ibm1, so you could skip running ibm1 thereafter: set run_ibm1_classic to false.
You may then run the HMM (run_hmm_empty_2="true") and/or the fertility HMM: (run_hmm_fertility_empty="true").

2. Choose iterations
==================

If you run run-model.sh without a parameter, you will see
 "<app> [prefix] [reverse (true|false)]  [data_size] [model_init(default|ibm1-n|hmm-2)] [model_from] [model_to] [model(ibm1|hmm)] [hadoop(true|false)] p0 sample"

which means you can specify the iterations by changing the model_from
and model_to to numbers you like in run.sh. Numbers start from 1. If
it is bigger than 1, you have to have the model output in the current
directory already, and the system will resume from that iteration. If
it is 1, it uses the model_init, usually a particular iteration of
ibm1 (say ibm1-5)

3. specify data
==================

 run.sh <prefix>

Trains IBM model 1 and then HHHM-fert model
on data in files <prefix>.train and <prefix>.test
Then evaluates alignment produced on <prefix>.test against <prefix>.afile
  gold standard alignment.

Train and test files are one line per sentence pair, with tab
character between English and Chinese sentences, and spaces between
words.  Files contain regular (ASCII) text, no conversion to numeric
word IDs is used.  Format for gold standard alignment file is one
line per alignment link:

<sent_no> <english_pos> <chinese_pos> P|S

where P is possible and S is sure.


4. specify data size (using partial data)
==========================================

You may specify to use the first n sentences of the training data. In
this way, the system will create the partial file for you
automatically.

If you run run-model.sh without a parameter, you will see
 "<app> [prefix] [reverse (true|false)]  [data_size] [model_init(default|ibm1-n|hmm-2)] [model_from] [model_to] [model(ibm1|hmm)] [hadoop(true|false)] p0 sample"
set data_size to the number you like in run.sh

5. change samples for the fertility HMM
========================================

change $sample in run.sh. For other models, it makes no difference

6. Change p0 for the HMM and the fertility HMM
========================================

change $p0 in run.sh. For ibm1, it makes no difference.

7. Creating GIZA++ style alignment results for MOSES training
========================================

search for MOSES in run-model.sh, you will see
    if [ $i -ne 15 ]
    then
	cp $test_file bitext.test
    else
	cp bitext.train bitext.test
    fi
You may change 15 to 5 if you want to evaluate the results at the 5th iteration(s).


# note on performance (speed, AER and BLEU)
=============================================
You may find the fertility model does a poor job than the HMM if the data size is small (1,000 sentence pairs) , but it is better and faster than the HMM if the data size is big (larger than 100,000 sentence pairs). 

Sep 12, 2010
Sam Shaojun Zhao
