Abstract:
In this paper, we introduce the idea of automatically illustrating complex sentences as multimodal summaries combining pictures, structure and simplified text. By including text and structure in addition to pictures, multimodal summaries provide additional clues of what happened, who did it, to whom and how - to people who may have difficulty reading or looking to skim quickly. We present ROC-MMS, a system for automatically creating multimodal summaries (MMS) of complex sentences by generating pictures, textual summaries and structure. We show that pictures alone are insufficient to help people understand most sentences, especially for readers who are unfamiliar with the domain. An evaluation of ROC-MMS in the Wikipedia domain illustrates both the promise and challenge of automatically creating multimodal summaries.