Chenliang Xu


My research aims to solve core Computer Vision problems as well as to push the limits of their understanding in the broader AI contexts, e.g., multisensory perception, cognitive robotics, and data science.

I'm grateful to the following entities for their support of my research:

This page only maintains master-level grant project. For the latest updates and individual projects, please refer to the publications page.

Current Projects:

Scalable Deep Bayesian Tensor Decomposition
Many applications in the real world, such as online shopping, recommendation, social media and information security, involve interactions among different entities. For example, online shopping behaviors can be simply described by the interactions between customers, commodities and shopping web sites. These interactions are naturally represented by tensors, which are arrays of multiple dimensions. Each dimension represents a type of entities (e.g., customers or commodities), and each element... more

Learning Dynamics and Evolution towards Cognitive Understanding of Videos
A fundamental capability of human intelligence is being able to learn to act by watching instructional videos. Such capability is reflected in abstraction and summarization of the instructional procedures as well as in answering questions such as "why" and "how" something happened in the video. This project aims to build computational models that are able to perform well in above tasks, which require, beyond the conventional recognition of objects, actions... more

Audio-Visual Scene Understanding
Understanding scenes around us, i.e., recognizing objects, human actions and events, and inferring their spatial, temporal, correlative and causal relations, is a fundamental capability in human intelligence. Similarly, designing computer algorithms that can understand scenes is a fundamental problem in artificial intelligence. Humans consciously or unconsciously use all five senses (vision, audition, taste, smell, and touch) to understand a scene... more

Applying Video Segmentation to Coarse-grain Mapping Operators in Molecular Simulations
We apply advances in computer vision to improve models of multiscale systems in chemistry. Multiscale systems describe chemical and physical processes that occur on many different time and spatial scales, for example, both very fast and very slow motions may contribute to the overall process. In both the computer processing of videos and the modeling of multiscale... more

Past Projects:

Cross-Modal Audio-Visual Generation
Cross-modal perception, or intersensory phenomenon, has been a long-lasting research topic in psychology and neurology; various studies have discovered strong correlations in human perception to auditory and visual stimuli. Despite many existing works in computational multimodal modeling, a large portion of the effort has been focused on indexing and retrieval the multimedia content. Although joint representations of multiple modalities and... more

Video-To-Text Summarization
The problem of generating natural language descriptions of images and videos has been steadily gaining prominence in the computer vision community and beyond. It is important for at least three reasons: i) transducing visual data into textual data would permit well-understood text-based indexing and retrieval mechanisms essentially for free; ii) fine-grained object models and region... more

Actor-Action Video Understanding
Action is the heart of video understanding. As such, it has received a significant amount of attention in the last decade. The emphasis has moved from small datasets of a handful of actions to large datasets with many dozens of actions; from constrained domains like sporting to videos in-the-wild. However, existing works emphasize a small subset of the broader action understanding problem. First, they all assume the agent of the action, which we call the actor, is a human adult, ignoring the diversity of actions performed by other... more

Perceptual Analysis of Semantic Retention
in Video Supervoxel Segmentation
Video supervoxel segmentation has recently been established and applied to large-scale data processing, which potentially serves as an intermediate representation to high level video semantic extraction. The supervoxels are rich decompositions of video content: they capture object shape and motion well. Yet, it is not known if the supervoxel segmentation retains the semantics of the... more

Supervoxels: Toward an Efficient
Hierarchical Representation for Video Analysis
Images have many pixels; videos have more. It has thus become standard practice to first preprocess images and videos into more tractable sets by either extraction of salient points or over-segmentation into superpixels. Preprocessing these outputs is more perceptually meaningful than raw pixels, which are merely... more