With the massive amount of images and videos, the ability to effectively search and localize the visual content of interest is of great importance. In this thesis, we will study visual content search systems that adopt languages or other cross-modality signals as input queries, because such queries are easier to generate and often provide more accurate target specification than image- or tag-based queries. Despite many sub-tasks have already been explored, previous studies approach each task independently. In this proposal, we unify related tasks into a general language-based visual content search framework by abstracting away the difference in input output formats and searching space. Furthermore, we extend the language-based visual content search to further include methods that have cross-modality queries as an intermediate representation. We expect our future study in this direction will benefit content-based search tasks.
In this thesis, we will study language-based visual content search and localization following two major directions. On the one hand, the unified task motivates the development of general methods. To be specific, certain problems such as feature embedding and multi-modality fusion are shared among different sub-tasks and universal solutions can be proposed. On the other hand, certain sub-tasks pose their unique challenges. For example, tracking by language requires the model to be robust to the frame degradation caused by motion and lighting changes. Specifically in this thesis, we will study human-centered visual content search tasks. With more information conveyed by the temporal dynamics (actions) instead of subjects or their attributes, human-centered search tasks pose the unique requirement of human understanding. We present our preliminary work to show our previous explorations and possible future solutions to both the general methods and task specific challenges. The timelines for future work are introduced at the end of the proposal.
Advisor: Prof. Jiebo Luo (Computer Science)
Committee: Prof. Daniel Gildea (Computer Science), Prof. Chenliang Xu (Computer Science), Prof. Ehsan Hoque (Computer Science) and Prof. Zhiyao Duan (Electrical and Computer Engineering)