Rough Statistical Analysis of Workshop Codings

At the 1997 AAAI Fall Symposium on Communicative Action in Humans and Machines, many of the participants annotated the first half of a dialog from the Coconut domain (see below for more details on this domain). Click here to see the entire dialog. The participants only had 15 minutes to label the dialog excerpt based on an eight page reference guide to the annotation scheme and a brief introduction talk. Despite the limitations of this experiment, this exercise is still interesting as a test of the different interpretations of tags such as "ReAssert" by untrained annotators.

In most cases, there were 17 annotators except for the Agreement tag and the Response To label. In the case of the Agreement tag, one annotator did not have time to finish and skipped this tag. Most annotators forgot to mark what their backward functions responded to and only six annotators recorded this information. Other Backward Function and Other Forward Function were each used only once over the 17 annotators and 35 utterances so they were not included in these statistics. Siegel and Castellan in their book, "Nonparametric Statistics for the Behavioral Sciences" show how to test the significance of kappa scores, to see whether a kappa score was a result of chance or reflects the agreement among the annotators. Siegel and Castellan assume kappas are normally distributed. One tailed significance levels are used here. The significance is calculated through calculating z and looking up the significance in a table. The equations below show how z is calculated. Understanding had a significance level above .1 showing that the kappa was not significant. Looking at the results of the understanding labeling shows a high degree of disagreement making the estimate of kappa variance high giving the non significant rating. Response-to had a negative variance (-0.008362); it is unclear what this means. The kappa for Response-to is included below under the assumption that it is significant. The rest of the results have positive variances and significant kappas; their significance levels are included in the results section.

Results

Forward Looking Functions

  • Statement (Kappa 0.63)
  • Influence On Addressee Future Action (Kappa 0.65)
  • Committing Addressee Future Action (Kappa 0.39)

    Backward Looking Functions

  • Answer (Kappa 0.57)
  • Understanding
  • Agreement (Kappa 0.36)
  • Response To (Kappa 0.52)

    Discussion

    Inter-annotator agreement was best on Statement, Influence On Addressee Future Action, and Answer. This result is not surprising as they have strong ties to syntax. Commitment, Understanding, and Agreement are not as straightforward. Commitment is not well linked to syntax. Some people marked various statements such as "I have $500 to spend" or "The cheapest sofa I have costs $300" as offers while others did not. Imperatives such as "so buy your $300 yellow sofa" were also problematic. Since the participants are supposed to develop a joint plan, the speaker in a sense is committing to action by saying "so buy your $300 yellow sofa". Statements about the future such as "I'll buy for 100 dollars" were problematic as they were marked as commits, offers, and no commitment.

    A problem faced by Understanding and Agreement is that these phenomena can be signaled implicitly ("so buy your $300 yellow sofa" implicitly accepts "The cheapest sofa I have costs $300").

    The example below shows a common pattern of disagreement. Here, four people considered both 11 and 12 as acceptances while eight people thought the acceptance only consisted of 11. I checked the dialogs by hand to make sure the four people marking utterance 12 as accept also marked utterance 11 as accept.

            None  Accept
      11       4 |  12  | :STEVE] [11]:  You are right,
      12      12 |   4  | :[12]: we have $1000 to spend.
      
    Implicit actions are even more of a problem for Understanding. At most of the change of turns, a few annotators would mark signal understanding/acknowledgment because by not saying "I didn't understand" you implicitly signal understanding. Utterance 10, "Also...Your $550 + my $450 is $1000", is a correct misspeaking of utterance 4, "So we have a total of $900 to spend." It is unclear whether people did not think correct misspeaking was appropriate or like myself did not notice the correction. Utterances 17 and 18 are not correct misspeaking because the speaker corrects themself. Both of these sets of utterances caused disagreements.

    The inter-annotator agreement on Response To reflects disagreements about not only what was responded to but whether an utterance has a backward function (understanding, agreement, answer) at all. The latter of these disagreements will also be reflected in agreement on understanding, agreement, and answer. There were 19 utterances where annotators disagreed on Response To; five of these utterances involved disagreements about the extent of a response. This is understandable as there was no discussion of this labeling in the reference sheet.

    Coconut Domain

    The goal of dialogs in this domain is for the two participants to collectively decide how to furnish two rooms with a limited budget. Each participant is given some money and a list of furniture for sale. The participants must collaborate to find out what furniture their partner has available. While the main goal is to negotiate the design of a two room house, the participants also have specific secondary goals which further complicate the problem solving task. They are instructed to try to meet as many of these goals as possible. The secondary goals are: 1) Match colors within a room, 2) Buy as much furniture as you can, 3) Spend all your money. The items of highest priority are a sofa for the living room and a table and 4 chairs for the dining room. In the dialog coded, the first participant (Steve) has $550 and the second participant (Garrett) has $450. Click here for Steve and Garrett's furniture lists. Click here for the Coconut dialog web page. This paper gives more details about the experimental setup.

    Last change: 24 Nov 97 by mcore

    Click here to send in comments or questions.


    Back to the DAMSL annotation page | Back to the symposium web page | To the DRI web page