A number of tools have been developed for transcribing and annotating dialogs from recorded conversations.
Descriptions of most of these tools can be found in Dialogue
Transcription Tools [Heeman & Allen 95].
Dialog transcription guidelines
- dat2raw: used to download dialogue from a DAT drive.
- showraw: used to mark downloaded dialogue from DAT.
- cutupraw: used to cut marked dialogues obtained from showraw.
- showbreakup: confirm cuts to raw dialogue file.
- cleanupraw: does cleanup of files and discards cut pieces of dialogue. Creates the final .fea file.
- transcriber: transcription tool for creating annoation windows for words, break indices, tones, etc.
- scatter: tool for assisting in doing word transcriptions
- dtranscriber/ptranscriber: similar to transcriber but also displays dialogue context of the utterance.
- transcriptor: for printing transcription
- startupdate: first program used for updating utterance breakups. Creates subdirectory new for creating new breakup.
- showupdate: display current breakup.
- update: creates all dialogue and utterance level files according to data in new.
- installupdate: installs updates in new into current directory and deletes new directory.
- cancelupdate: cancles an update that has not yet been installed.
Speech Repair Tools
- dd-classes.pl file1 file2: given class files output merged version
classes of form: class < X.Y = Z
- add-freq.pl: adds up total number of elements in classes.
can used in prob files (format: class < X.Y = Z where Z is a probability)
to sum probabilities
- add.pl: adds a column of numbers
- bt-reparandum.pl: print distance of boundary tones
- et.pl: takes .misc file as argument
outputs list of editing terms for utterance
- fluent.pl: counts number of transitions in set of utts
transitions being transitions between words.
- not-fluent.pl: count number of speech repairs
- parse-srepair.pl: converts verbose output of speech
repair into list of numbers.
- prob.pl file1 file2: output file1/file2
i.e. if line i of file1 = 0.1 and line i of file2 = 10 output is .001
- setup-cv: (csh script) sort silences
break into classes. parse repairs, break into classes
- sil-classes.pl: given a file w/ a sorted list of
numbers output amount of these numbers fitting into
classes < 0.1, < 0.2 ..... < X.Y where X.Y fits max
number in file.
- silence.pl: ouput all silence lengths
- speech-repair.pl: output silence lengths
before interruption points
- training-data.pl/training-data2.pl: create intermediate form
- update.pl: update dat file. translate influence-on-listener=info-request
- bigram.pl - does simple bigrams. P(val1_i | val2_i-1)
probability of val1 (of tag) in current utterance given val2 (of tag) in
previous utterance. tag should be in all lower case.
if TRUE arg given then gives probability of val1 (from context-tag) and val2
(from tag) occuring in same utterance.
if FALSE arg given then gives probability of val1 of tag in current utterance
given val2 (of context-tag) in previous utterance. tags should be in all
- judge.pl - spit out
lines announcing a probability greater than
- judgebigrams - given a tag
search all context files for probabilities greater
- disagree.pl : prints each line of the
dialog preceded by the votes for each tag if there was a disagreement.
kappa score (and PE and PA) is given as well. Note, tag should not
be response-to which is handled below
- response-to.pl :
if the annotators had different values for the response-to tag
of an utterance, the utterance is printed out (to stdout) along
with the various answers. The kappa and Percent Expected agreement
and Percent actual agreement are calculated and printed as well
- 4tags <4 sgml-annotation-files>: runs disagree on the 4 files with
all the tags creating files for the results. runs response-to as well.
- tags <2 sgml-annotation-files>: same as 4tags but with 2
- printdat.pl : print utterances followed by their annotations
- printdat-old.pl : same as printdat.pl except works with the
old set of DAMSL tags (illoc-act, response, etc.)
- printsegments.pl : print segment names and their text followed
by their annotations
- format.pl : given a www version of a trains dialog
such as /u/www/research/speech/93dialogs/d92a-2.1 it will strip off
the unwanted formating and print the dialog to stdout. The result
can then be segmented. (briefly tested under perl 5.003)
- showdialog: right now simply runs xwaves on dialog.fea. until we
get a new version of xwaves we can't do anything else with the
new version of Solaris we are running.
- raw2sgml : dlgfile should be of the format output
by format.pl. The format is raw text. New turns are begun with
either s: or u: and no <> brackets are allowed. when turns should be
split into one than one utterance. The utterances should be separated by
"$"'s. addsegments.pl, convert.pl, turnspeech.pl, and fix.pl are called
to convert dlgfile into an sgml file ready for dat. dlgname is encoded
in the sgml and raw2sgml will create a file called dlgname.sgml
(briefly tested under perl 5.003)
- addsegments.pl : dlgfile should be of the format output
by format.pl. The format is raw text. New turns are begun with
either s: or u: and no <> brackets are allowed. $ are then added
to show where turns should be split into more than one utterance.
$ are meant to show utterance endings so
this program puts $ at the end of turns so the annotator is not
required to. It is smart enough not to add a $ if it
is already there. does not modify dlgfile, outputs a modified version
- turnspeech.pl : the turns of the input sgml file are assumed not to
have speech info. this program outputs (to stdout) an sgml file where the
turns have speech info (based on the snd info of their component utterances)
- fix.pl : removes spurious spaces from input file, outputs result
- startwords: copies the dialog.log and any exiting word transcriptions to the local directoyr, which should be named the same as the dialogue directory.
- words: used for transcribing words in the local directory. Similar to transcriber, but without the misc, tones, and breaks.
- strip_header: awk script to remove the header.
- uttsdiff: compares two dialog.data to check which utterances need to be fixed up.
- uncompressdialog: takes speaker0.fea and speaker1.fea and
uncompresses them into speaker00.fea and speaker01.fea
and converts them into speaker0.au and speaker1.au
- mux.m: MAT lab script. takes two au files
called speaker0.au and speaker1.au and outputs a file, dialog.au, a mux'ed
version of the two. this file will have no au header. Use ~mcore/bin/convert
to translate it to the dialog.fea file used by dat. ~mcore/bin/cleanup
will then erase the intermediate files. ~mcore/uncompressdialog will
take two NIST compressed files from the TRAINS CD ROM, speaker0.fea and
speaker1.fea and create speaker0.au and speaker1.au files for mux.m
to operate on. This can take a while. You might want to type
control Z in matlab and then background "bg" the job.
- convert: takes a headerless dialog.au file and converts it
into a dialog.fea file
- cleanup: For dialog-dat, erases dialog.au and speaker audio files
Should be used after one channel dialog.fea created
CISD Home Page
URCS Home Page
Last change: 21 November 2000 by R. Paul McCarty