A fundamental capability of human intelligence is being able to learn to act by watching instructional videos. Such capability is reflected in abstraction and summarization of the instructional procedures as well as in answering questions such as "why" and "how" something happened in the video. This project aims to build computational models that are able to perform well in above tasks, which require, beyond the conventional recognition of objects, actions and attributes in the scene, the higher-order inference of any relations therein. Here, the higher-order inference refers to inference that cannot be answered immediately by direct observations and thus requires stronger semantics. The developed technology will enable many applications in other fields, e.g., multimedia (video indexing and retrieval), robotics (reasoning capability of why and how questions), and healthcare (assistive devices for visually impaired people). In addition, the project will contribute to education and diversity by involving underrepresented groups in research activities, integrating research results into teaching curriculum, and conducting outreach activities to local K-12 communities.
The research will develop a framework to perform higher-order inference in understanding web instructional videos, such that models devised in this framework are capable of not only discovering and captioning procedures that constitute the instructional event but also answering questions such as why and how something happened. The framework is built on a video story graph that models the dynamics (the composition of actions at different scales) and evolution (the change in object states and attributes), and it supports higher-order inference upon deep learning units and incorporation of external knowledge graph in a unified framework. Methodologies to extract such video story graphs and use them to discover, caption procedures and perform question-answering will be explored. Expected outcomes of this project include: a software package for constructing and performing inference on video story graphs and incorporating external knowledge; a web-deployed system to process user-uploaded instructional videos; and a large video dataset with procedure and question-answering annotations.
Data, demos, and software are downloadable by following the individual tasks below.
The proposed project will make a breakthrough in understanding web instructional videos through computational modeling of the higher-order inference capability in human intelligence. In particular, it aims to create a sound framework that combines the structured reasoning capability in graph theory with the superior feature learning capability in deep neural networks, e.g., the preliminary dynamic graph module we developed. Furthermore, the project is expected to not only achieve a better performance on video captioning and QA tasks but also open up the black box of the decision-making process in deep learning. For example, our work on interpretable and controllable audio-visual captioning unfolds many interesting scientific inquires. Last but not least, the optimization problems associated with the video story graph learning and inference are non-convex and high-dimensional that require novel advances in the associated mathematics.
The proposed project is also of keen interest to other research communities. Mission-critical applications such as autonomous driving and security surveillance need robust reasoning capability to run in real-world scenarios. Researchers in multimedia are interested in indexing and retrieving information from large-scale Internet videos. Researchers in data mining and social media analysis are interested in analyzing user behaviors from images and videos uploaded by users to social media sites. In all of these cases, the proposed work is a fundamental facilitating technique. Methodologies and techniques developed in this project can also be applied to human-computer interaction and collaborative robots.