Traditional speech to speech agents have been developed under a pipeline architecture where one module feeds the next in a serial manner. These have several shortcomings as earlier processing may only arrive at certain interpretations with information acquired in later stages of processing. As we move towards a continuous understanding architecture, there is a need to create corpora which can be used to develop this newer approach.
In this talk we present an annotation scheme that addresses the issue of incremental language processing in a multimodal dialogue. The annotation is done at the word level, a finer level of detail that breaks away from typical sentence level labeling. By delaying labels within the utterance until speaker's intentions are fully disambiguated, one can see how early understanding is achieved by examing the listener reactions.