Login
Computer Science @ Rochester
Friday, April 05, 2013
12:45 PM
CSB 703
Iftekar Naim
University of Rochester
Text Alignment for Real-Time Crowd Captioning
Real-time captioning provides deaf and hard of hearing people access to speech in classrooms, meetings, and on live televisions. Primary approaches for real time captioning relies either on expensive professional stenographers or on error prone automated speech recognition (ASR) systems. Recent work has shown that a feasible alternative is to combine the partial captions by ordinary typists, each of whom types part of what they hear. We propose an improved method for combining partial captions into a final output based on weighted A* search and multiple sequence alignment (MSA). Our method outperforms the current state-of-the-art on Word Error Rate (WER) (29.6%), BLEU Score (41.4%), and F-measure (36.9%). The end goal is for these captions to be used by people, and so we also compare how these metrics correlate with the judgments of 50 study participants, which may assist others looking to make further progress on this problem.

Joint work with: Daniel Gildea, Walter Lasecki, and Jeffrey Bigham