Building Speech Recognition Models for TRIPS
Contents:
-
1. Obtaining the training data
-
1.1. Collecting initial data
-
1.2. Building an 'artificial' grammar
-
1.3. Generating an 'artificial' corpus
-
1.4. Using class definitions to generate 'artificial' corpus
-
2. Building language models
-
2.1. Configuration
-
2.1.1. Data files and directories
-
2.1.2. The configuration file
-
2.2. The makefile
-
2.3. lm-ez*
-
2.3.1 Options
-
2.3.2 Examples
-
2.4 Creating native Sphinx class based models
-
2.4.1 Class definition file
-
2.4.2 Generating 'artificial' corpus and inclass probability model
-
3. Configuring Sphinx
-
3.1 lm.ctl
-
4. Related information
-
5. Notes
1. Obtaining the training data
1.1 Collecting initial data
-
Script data provides good initial data.
-
For the various objects involved, one could use maps, tables and other
sources of knowledge provided in the description of the domain.
-
Explanations of how things should work could be used as well.
-
To be readily usable, the text data should be saved using the following
format:
-
one sentence per line;
-
words/phrases need to be written the way they should appear in the language
model; words that are usually written differently may be kept in text,
but should be enclosed in square brackets (E.g., "30 days" should
be replaced by "thirty days" or "thirty [30] days" if
"30" is not going to be in the vocabulary); everything included
in square brackets will be disregarded for the purpose of language modeling,
therefore other events or comments may be bracketed as well (see the rules
for speech transcription);
-
comments should be marked by '#' in the first column; and
-
sentence markers should not be used.
Note that, although currently capitalization and punctuation are not accounted
for by the models, in order to keep text data files as natural as possible,
it is advisable that:
-
words should be capitalized according to the current usage of English;
and
-
punctuation signs are used.
1.2 Building an 'artificial' grammar
-
A good start is one of the already existing grammars. One needs to make
the necessary additions to cover the initial data; already existing constructs
may be deemed unnecessary. Grammars exist for the following domains: Pacifica,
CPoF,
Monroe
County, and CAMPS.
Notes:
-
the grammars are RCS-ed, so if they aren't where they are supposed to be,
they probably need to be checked out.
-
copies of the grammars should be found in the domains' data folders (see
section
2.1)
-
The CAMPS grammar follows a more principled design than its predecessors.
The current
layout is in no way definitive, but it should be at least reasonable.
It is very succint, but it should be able to provide useful hints when
looking for particular constructs, or when adding new constructs.
-
Syntax
There are two types of rules: lexical and utterance definitions. Lexical
definitions have the form:
where each disjunct Di is a sequence of 1 or more items:
and each item is either a word, or an already defined non-terminal, or
a disjunctive group, or an optional item. Disjunctive groups have the form:
where each token item Ti is a word or an already defined non-terminal,
and optional items have the form:
where C is an item as defined above (except optional items), or
it can be a sequence of items written:
Utterance definitions are sequences of items enclosed between sentence
markers:
Certain limitations are due to the way the tools that operate on
these grammars have been designed. Not allowed are:
-
Recursion.
-
Nesting levels higher than 2 (note that this doesn't limit the expressiveness
of the grammar since it can be overcome by introducing new non-terminals).
-
White space in disjunctive groups and optional items.
The grammar file should be organized this way:
[header]
# __START__
... lexical definitions ...
# __UTTERANCES__
... sentence definitions ...
# __END__
[trailer]
Comments are allowed in between definitions; they are marked by a '#'
as the first character of the line.
-
Testing
For testing the grammar two tools may come handy: gramdump*(1)
and checkcfg*. gramdump*is
used mainly for generating sentences by randomly traversing the grammar.
checkcfg* is used for testing whether a sentence is in the language
generated by the grammar. Here are some typical invocations of these tools:
example% gramdump -g my_grammar
generates 20 word sequences from the grammar my_grammar.
example% gramdump -g my_grammar -d > my_grammar.dmp
dumps the grammar my_grammar in a less peculiar format that can
be used by checkcfg*.
example% checkcfg -g my_grammar.dmp
allows interactive testing of the coverage of the grammar. The tool checks
whether given utterances are in the language generated by the grammar,
and warns if words are used that are not in the grammar's vocabulary.
example% checkcfg -g my_grammar.dmp -v my_grammar.vocab -n
saves the grammar's vocabulary in the file my_grammar.vocab.
1.3 Generating an 'artificial' corpus
-
The tool gramdump* is used again. Typical usage(2):
example% gramdump -g my_grammar -n 50000 \
> /u/trains/lm/domains/$DOMAIN/corpora/corpus-50k.txt
-
The number of sentences generated (option -n) depends on the grammar; for
a small to medium-sized vocabulary in the Pacifica domain a 70,000-sentence
corpus proved the best performance in a WER test.
1.4 Using class definitions to generate 'artificial' corpus
gramdump takes an argument to specify a generic class definition file as follows. This is necessary to generate native Sphinx class based models.
gramdump -classes -g my_grammar
The format of the class definition file is as follows:
<lm_classes>
<class name="CLASSNAME">
<member>MEMBER1</member>
<member>MEMBER2</member>
...
<subclass>CLASSNAME1</subclass>
<subclass>CLASSNAME2</subclass>
...
</class>
<class name="CLASSNAME1">
...
</class>
</lm_classes>
where the above is equivalent to the artificial corpus definition:
<CLASSNAME> ::= MEMBER1 | MEMBER2 | <CLASSNAME1> | <CLASSNAME2>
2. Building models
2.1 Configuration
2.1.1 Data files and directories
Most directories are created automatically by the tools used for building
the models. For the destination folders it is possible to specify in the
command line what they should be. See the Tools Manual for details. The
data files are generally found in /u/trains/lm/domains.
This folder contains a subfolder for every domain, plus a subfolder called
general,
that contains files that are TRIPS-dependent rather than domain-dependent.
The data used for phonetic and language modelling comprise corpora,
phonetic dictionaries, and, optionally, tag dictionaries, lists of compound
words, lists of words and/or tags used for class-based adaptation (see
below), grammars for building artificial corpora, etc. It will become clear
what kind of data is needed and where it should be stored in the following
section. Typical sub-directories found in the domain folders are:
-
config - holds configuration files;
-
corpora - holds, obviously, text corpora;
-
grammars - holds 'artificial' grammars (see above);
-
varia - holds odd things that usually are not needed
for building models (E.g., notes, statistics, etc.)
2.1.2 The configuration file
The configuration file provides a convenient way of defining the parameters
and the environment for building new SR models. It includes definitions
for several variables that specify the type of models being built and the
data being used. Practically, to build SR models it is generally enough
to create a good configuration file -- all the rest can be done automatically.
Here is a list of commonly used variables:
-
parameters that control the type of language model being built
-
N - the order of
the model (2 for bigram, 3 for trigram, etc.)
-
SLM-DISCOUNT - the
discounting strategy (the method of redistributing some probability mass
from the frequent events to the rare and/or unseen events). Can be: linear,
absolute,
good_turing,
or witten_bell.
-
CUTOFFS - cutoffs
for n-grams (n>1). A cutoff of K means that all n-grams occurring
K or fewer times are discarded. If the parameter is omitted, then all the
cutoffs are set to zero.
-
VAR - a list of
options that specify special kinds of LMs to be built (note that several
models can be obtained with one command). Several options have been implemented;
however, in the current version not all may be available. Here are some
possible values (the first two are the ones commonly used in TRIPS):
-
-t, for class-based adapted models (also for word-based models
(standard) at some point);
-
-f, for models built by adaptation from out-of-domain (OOD) corpora;
-
-l, for phrase models (i.e., the vocabulary may contain word compounds
as lexical idems);
-
-b, for "bleached" models in one incarnation, and for models with
feature-augmented words in another;
-
-x, for models with POS-augmented words;
-
-xb, for models with words augmented with both features and POS
tags.
-
parameters specifying the location of various files used during the process
of building the models
-
CORPORA - a list
of paths to the locations of the corpora being used for training and/or
testing. Note: the files may be gzipped. (E.g.)
If part of the corpus has to be set aside for the evaluation, one may specify
here the amount of testing material by setting either the variable TSTN,
or TSTP. TSTN
represents the number of sentences to be included in the test data, and
TSTP
represents the percentage of the corpus to be taken off for testing purposes.
-
COMPOUNDS - a file
containing the word sequences that are to be treated as lexical items (optional).
(E.g.)
-
DS_PHONDIC - the
location of the domain-specific phonetic dictionary (optional). This file
should contain pronunciation variants that are not in the base dictionary
(CMU's cmudict.0.4). The
phonetic lexicon (phoneset.0.4)
should be consulted before adding pronunciations. (E.g.)
-
L-TAGDICS, E-TAGDICS,
T-TAGDICS
- various classes of tag dictionaries. The first two are more general,
and are used for class-based adaptation using OOD corpora. The last one
is domain-specific, and usually points to a file called /u/trains/lm/domains/$DOMAIN/$DOMAIN-extra.tagdic
(E.g.).
-
FILTER, F-TAGS,
F-WORDS
- files used for filtering tag dictionaries in building class-based adapted
models.
F-TAGS and F-WORDS
are, respectively, lists of tags (E.g.)
and words (E.g.)
that should be filtered out. FILTER
is a script that does the filtering. There is only one filter currently
in use, called cleandic*;
however, it appears as a configuration variable because there may be other
ways of deciding which words/tags to keep and which to throw.
In principle other variables that appear in the makefile (see next section)
can be set in the configuration file; however, doing so is not recommended.
2.2 The makefile
The process of building language and pronunciation models for TRIPS is
driven by one, fairly large, makefile.
It also includes support for perplexity evaluation. For full control over
the process, one should use the gmake utility directly. Alternatively,
the lm-ez* script could be used, which is more transparent and
provides some extra functionality.
2.3 lm-ez*
lm-ez* is a Perl script
that is in fact a wrapper for the makefile, but with the advantage
of a simpler command line syntax. In addition, it checks the results of
individual steps in building the models, and prompts the user with detailed
information when certain decisions need to be made, for example,
how to deal with incomplete tag dictionaries, or with incomplete pronunciation
dictionaries. The user has the choice to correct things manually, or can
let the script take care of the problems it encounters. The script can
also build several models at once, and prompts the user to decide whether
to build interpolated models as well.
One particularly useful feature is that it may build models from a
new text and combine the models obtained with some given base models; this
allows the tool to be used for static adaptation. The models are combined
by linear interpolation, and therefore the user will be prompted to provide
interpolation weights. Based on past experience, we found the following
sets of weights to produce good results:
-
0.65/0.35 for the interpolation of class-based and, respectively, word-based
models trained on artificial corpora;
-
0.15/0.85 for the interpolation of class-based and, respectively, word-based
models trained on small sample texts;
-
0.45/0.55 for the interpolation of sample-based interpolated models with
artificial corpus-based interpolated models. In general, when interpolating
a sample-based model with a larger, background model, we recommend using
a weight between 0.4 and 0.5 for the former.
2.3.1 Options
-m model |
Model name (required). |
-d lmdir |
Destination directory. If not set, it will be model
by default. |
-t text |
Use text as training corpus. |
-c config_file |
Configuration file. |
-b basemodel basedic |
Base language and pronunciation models. |
-l logdir |
Log directory. |
-w |
Build word-based models. |
-k |
Build class-based models. |
-v |
Perform tag dictionary check (look for missing words). |
-n |
Clean temporar files after finishing building the models. |
2.3.2 Examples
The following two examples:
example% lm-ez -d lm -m monroe -l logs -w \
-c /u/trains/lm/domains/monroe/config/monroe.config
example% lm-ez -d lm -m monroe -w
both do the same thing: build a word-based model for the monroe
domain in the folder lm. In the second example, the log directory
and the configuration file will receive default values, equal to the corresponding
values set explicitly in the first example.
-
Check the missing-words.txt file and add pronunciations for each word in
that file, then save the file in the directory /u/trains/lm/domains/$DOMAIN/.
Give the pronunciation file an appropriate name (E.g., $DOMAIN-extra.dic)
and set the variable DS_PHONDIC in
the configuration file. Next run of
lm-ez will consider these
pronunciations as well.
2.4 Creating native Sphinx class based models
The technique presented above uses tagdics to build word based models which smooth probabilities based on classes. An alternative technique is to generate class based models and a second file which defines word probabilities within each class. This has a number of advantages, specifically it reduces an O(n2) search space of bigram probabilities to O(2n). But it requires some additional steps during LM building.
2.4.1 Class definition file
You will need to create a class definition file. The format is defined in 1.4 above. This can be generated from an artificial grammar using the program grammar2class as follows.
grammar2class -g <grammarfile> > my_classes
You may need to edit this file to add or remove classes you want to use.
Of course you can always create your own class file by hand.
2.4.2 Generating 'artificial' corpus and inclass probability model
Now when running gramdump you will need to specify the class file you created above.
gramdump -classes my_classes -g my_grammar -n 10000 > my_dump
Since this is a class based model (not a word based model) you will need to generate the in class probability model for sphinx. This can be accomplished using classdump and dump2corpus in two steps.
classdump -c my_classes -sphinx > my_inclass
classdump will create a uniform probability model for words in classes for Sphinx. You will next need to run dump2corpus with the -c option to update the probabilities to reflect the distribution in the dump file, or your tagged corpus.
dump2corpus -d my_dump -c my_classes > my_corpus
Not to make things too complicated, but you have a number of options when using classdump and dump2corpus above to control class assignment. Sphinx and lm-ez expect that a word can only belong to one class. Because the class definition and artificial grammars allow words to belong to multiple classes (some nested).
By default, every word is given it's own lexical entry where the class it was found in is attached. In this way every word gets a unique entry for every class it appears in. For example, in the medadvisor domain, the drug Aspirin is a non-prescription drug, which belongs to the superclass of drugs, and drugs belong to the superclass of substances. So all of the following would appear in the medadvisor model if no options are used when running classdump and gramdump.
ASPIRIN/[DRUG_NON_PRESCRIPTION]
ASPIRIN/[DRUG_NON_PRESCRIPTION]/[DRUG]
ASPIRIN/[DRUG_NON_PRESCRIPTION]/[SUBSTANCE]
lm-ez removes the trailing class tags and obtains the correct pronunciation regardless of the redundant entries.
Two other possibilities exist when tagging these words. We can only use the highest class (e.g. Aspirin is in the class [SUBSTANCE]) or we can use the lowest classes (e.g. Aspirin is in the class [DRUG_NON_PRESCRIPTION]). To specifiy the former we add the -top option when running dump2corpus and classdump.
classdump -c my_classes -sphinx -top > my_inclass
dump2corpus -d my_dump -sphinxclass my_inclass -top > my_corpus
For the aspirin example, this generates the following entries.
ASPIRIN/[DRUG]
ASPIRIN/[SUBSTANCE]
Alternatively, we may want every class to be expanded into all it's constituents.
classdump -c my_classes -sphinx -bottom > my_inclass
dump2corpus -d my_dump -sphinxclass my_inclass -bottom > my_corpus
For the aspirin example, this generates the following entries.
ASPIRIN/[DRUG_NON_PRESCRIPTION]
3. Configuring Sphinx
Once you have your ngram model and dictionary after running lm-ez on the class based corpus you generated above, you need to configure sphinx to read these files.
3.1 lm.ctl
Sphinx can be instructed using commandline arguments what ngram model and dictionary to use at runtime.
sphinx2-continuous -dictfn <dictionaryfile> -lmfn <ngramfile>
Alternatively, you can use a control file, which we usually name lm.ctl
The control file is organized as follows.
{ <inclass_lm> }
<ngramfile> <lmname>
{ <CLASSNAME1> <CLASSNAME2> ... }
The lines in curly braces are option lines specifying an inclass model and the classes from the model which will be used by the named LM. If you are not using Sphinx native class based models (see 2.4 above) then you don't need to specify those lines in lm.ctl
You can also specify multiple models with different names. For example,
pacifica/lm.bigram pacifica
monroe/lm.bigram monroe
medadvisor/lm.bigram medadvisor
medadvisor-demo/lm.bigram medadvisor-demo
test/lm.bigram test
4. Related information
The CMU Pronouncing
Dictionary (current version: 0.6)
The CMU-Cambridge
Statistical Language Modeling toolkit (current version: 2.05)
5. Notes
1. Here we use the convention to add a star (*) to
the name of all executables, to make them more readily distinguishable
among the many data files that appear throughout the text.
2. In all examples $DOMAIN represents
the name of the domain for which models are built, and should be replaced
with this name in order for the commands to work. Alternatively, before
trying the sample commands the user may want to setenv $DOMAIN to the name
of the current domain.