Ph.D. Thesis Defense
This thesis focuses on improving human-human interaction during group discussions through feedback from effective human-machine interaction. Identifying verbal and non-verbal attributes and allowing people to be aware of them are crucial for maintaining a safe and effective exchange of ideas. This work explores capturing the related attributes (e.g., talk-time, turn-taking, interruption, shouting, valence, attention, engagement, consensus, etc.) from audio, video, and language information from remote group discussions, and developing automated feedback systems (e.g., chatbot, conversational agents, visualization) for mediation purposes.
First, we present a multi-modal dataset for interpersonal disrespect or toxicity annotated from 59 YouTube News Show dyadic remote discussion videos. Our models perform with an accuracy of over 60% using the visual features and close to 80% on the audial features to recognize the attributes of disrespect.
Next, we develop automated systems that capture group dynamics features and provide privacy preserving feedback. For keeping heated discussions respectful, we build a videochat platform integrated with real-time feedback processed mainly on the client side. Validation from 40 participants reveals that real-time feedback can reduce expressiveness during an ongoing discussion, however bears a carry-over effect improving the following discussion without real-time mediation.
For post-meeting reflection, we develop a fully automated collaboration platform ‘CoCo’ that can hold videoconferencing meetings, process data post-session in bulk on the server side, and present feedback through an interactive chatbot. Evaluation from 39 participants shows the improvement in group dynamics after receiving feedback. We also explore post-session feedback in an in-situ workplace setting. We survey the challenges faced in remote meetings by employees (N = 150), and as per their needs design and evaluate a wireframe prototype (N = 16) and an interactive feedback dashboard named ‘MeetingCoach’ (N = 23). The study proposes actionable suggestions, personalized modeling, and privacy preserving feedback that can potentially improve meeting effectiveness and inclusivity.
For pre-meeting training, we present a suggestive chatbot incorporated with motivational interviewing (MI) technique for improving conversational skills. Evaluation from a consensus based résumés study with 21 participants showcases the effectiveness of suggestive MI chatbot in encouraging users to apply the information delivered by agents. We highlight strategies to reach a consensus fulfilling individual and team goals.
Finally, we explore improving agent capabilities in terms of empathy and affect. We design the dialogue of an empathetic conversational agent and evaluate it in a Wizard-of-Oz based study with 34 participants. Our results show improved human-machine interaction (HMI) and its impact on mitigating user’s negative affect. We also explore sentiment detection on an HMI dataset. We build and compare multimodal LSTM fusion (accuracy_nothreshold = 67.5%, accuracy_threshold = 71.8%) and hierarchical (accuracy_nothreshold = 60.9%, accuracy_threshold = 71.8%) models. The results show the importance of increasing agent capabilities in becoming more affective and interactive to effectively communicate with users.
Overall, the findings of this thesis work provide useful information to the research community regarding understanding group dynamics and designing effective feedback agents.
Advisor: Prof. Ehsan Hoque (Computer Science)
Committee: Prof. Zhen Bai (Computer Science), Prof. Chenliang Xu (Computer Science), Dr. Amanda Stent (Bloomberg), and Dr. Mary Czerwinski (Microsoft Research)
Chair: Prof. Melissa Sturge-Apple (Psychology)