Chenliang Xu


My research thrusts include computer vision and its relations to natural language, robotics and data science. I make every effort to keep this page up to date, but please refer to the publications for the most recent listing.

Cross-Modal Audio-Visual Generation

(generated from sound)
Cross-modal perception, or intersensory phenomenon, has been a long-lasting research topic in psychology and neurology; various studies have discovered strong correlations in human perception to auditory and visual stimuli. Despite many existing works in computational multimodal modeling, a large portion of the effort has been focused on indexing and retrieval the multimedia content. Although joint representations of multiple modalities and their correlations are explored in these works, they do not need to model the details of the samples. Here, the focus of my research is on fine-grained cross-modal audio-visual generation that advances the frontier in multimodal modeling. We have developed algorithms in audio-visual source association that are able to segment corresponding audio-visual data pairs; we have created deep generative neural networks utilizing adversarial training that are able to generate one modality, i.e., audio/visual, from the other modality, i.e., visual/audio. The outputs of cross-modal generation are beneficial to many applications, such as aiding hearing- or visually-impaired and content creation in virtual reality. List below are relevant publications:
  1. L. Chen, S. Srivastava, Z. Duan and C. Xu. Deep Cross-Modal Audio-Visual Generation. arXiv:1704.08292, 2017.
  2. B. Li, C. Xu and Z. Duan. Audio-Visual Source Association for String Ensembles through Multi-Modal Vibrato Analysis. In Proceedings of Sound and Music Computing (SMC), 2017.

Sequence Discovery, Reasoning and Summarization in Video Storytelling

(example output of v2t)
The problem of generating natural language descriptions of images and videos has been steadily gaining prominence in the computer vision community and beyond. It is important for at least three reasons: i) transducing visual data into textual data would permit well-understood text-based indexing and retrieval mechanisms essentially for free; ii) fine-grained object models and region labeling would provide significant richness to multimedia retrieval techniques; and iii) grounding representations of visual data in natural language has great potential to overcome the inherent semantic ambiguity in closed-world recognition tasks. On the other hand, videos, compared to images, contain rich temporal structures and causalities, hence they introduce a new level of difficulties. Here, the focus of my research is on designing efficient, scalable methods to harness the temporal richness in video storytelling by automatically discovering sequences, reasoning their orderings and summarizing their content in natural languages. We have developed early prototypes using both bottom-up and top-down information to translate short clips to texts. More recently, we have developed methods utilizing deep neural networks to automatically discover and summarize procedures in long untrimmed instructional videos. The YouCook dataset we collected has been widely used in the high-level video understanding community. List below are relevant publications:
  1. L. Ding and C. Xu. TricorNet: A Hybrid Temporal Convolutional and Recurrent Network for Video Action Segmentation. arXiv:1705.07818, 2017.
  2. L. Zhou, C. Xu and J. J. Corso. ProcNets: Learning to Segment Procedures in Untrimmed and Unconstrained Videos. arXiv:1703.09788, 2017.
  3. L. Zhou, C. Xu, P. Koch and J. J. Corso. Watch What You Just Said: Image Captioning with Text-Conditional Attention. arXiv:1606.04621, 2017.
  4. P. Das, C. Xu, R. F. Doell and J. J. Corso. A Thousand Frames in Just a Few Words: Lingual Description of Videos through Latent Topics and Sparse Object Stitching. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013.

Fine-Grained Recognition and Segmentation of Actors and Actions in Videos

(example frames from A2D dataset)
Action is the heart of video understanding. As such, it has received a significant amount of attention in the last decade. The emphasis has moved from small datasets of a handful of actions to large datasets with many dozens of actions; from constrained domains like sporting to videos in-the-wild. However, existing works emphasize a small subset of the broader action understanding problem. First, they all assume the agent of the action, which we call the actor, is a human adult, ignoring the diversity of actions performed by other actors. Second, the prior literature largely focuses on action recognition, which is posed as the classification of a temporally pre-trimmed clip into one of k action classes from a closed-world setting. Here, my research overcomes both narrow viewpoints and introduces a new level of generality to the action understanding problem by considering multiple different classes of actors undergoing multiple different classes of actions. I am interested in simultaneously solving the set of 3W questions, namely, what action is happening, who is performing the action, and where is the action happening in space and time. We have developed various methods in advancing probabilistic graphical models to model actor-action interactions and have found that inference jointly over actors and actions outperforms inference independently over them. Our work has also enriched the scope of action understanding to consider temporally untrimmed long videos, adaptive video scales and weakly supervised learning setting. List below are relevant publications:
  1. Y. Yan, C. Xu, D. Cai and J. J. Corso. Weakly Supervised Actor-Action Segmentation via Robust Multi-Task Ranking. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  2. C. Xu, C. Xiong and J. J. Corso. Action Understanding with Multiple Classes of Actors. arXiv:1704.08723, 2017.
  3. T. Han, H. Yao, C. Xu, X. Sun, Y. Zhang and J. J. Corso. Dancelets Mining for Video Recommendation Based on Dance Styles. IEEE Transactions on Multimedia (TMM), 2017.
  4. C. Xu and J. J. Corso. Actor-Action Semantic Segmentation with Grouping Process Models. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  5. C. Xu, S.-H. Hsieh, C. Xiong and J. J. Corso. Can Humans Fly? Action Understanding with Multiple Classes of Actors. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.

Supervoxels--Toward an Efficient Hierarchical Representation in Early Video Analysis

(supervoxels of streamgbh)
Images have many pixels; videos have more. It has thus become standard practice to first preprocess images and videos into more tractable sets by either extraction of salient points or oversegmentation into superpixels. Preprocessing these outputs is more perceptually meaningful than raw pixels, which are merely a consequence of digital sampling. However, the same practice does not entirely exist in video analysis. Although many methods do indeed initially extract salient points or dense trajectories, few methods rely on a supervoxel segmentation, which is the video analog to a superpixel segmentation. Here, the focuses of my research are on exploring the potential of supervoxel segmentations in advancing various video analysis tasks and on designing efficient supervoxel algorithms to better model space-time of video. We have created a supervoxel library and benchmark, i.e., LIBSVX, that is widely used in the early video analysis community. We have promoted the idea of streaming processing in video segmentation and developed the first streaming hierarchical supervoxel algorithm. The supervoxels generated by our algorithm are being used in various video analysis tasks, such as object tracking, video object segmentation, and action localization. List below are relevant publications:
  1. C. Xu and J. J. Corso. LIBSVX: A Supervoxel Library and Benchmark for Early Video Processing. International Journal of Computer Vision (IJCV), 2016.
  2. C. Xu, S. Whitt and J. J. Corso. Flattening Supervoxel Hierarchies by the Uniform Entropy Slice. In Proceedings of IEEE International Conference on Computer Vision (ICCV), 2013.
  3. C. Xu, C. Xiong and J. J. Corso. Streaming Hierarchical Video Segmentation. In Proceedings of European Conference on Computer Vision (ECCV), 2012.
  4. C. Xu and J. J. Corso. Evaluation of Super-Voxel Methods for Early Video Processing. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.

Perception Analysis of Semantic Retention in Video Supervoxel Segmentation

(supervoxel video of baby crawling)
Video supervoxel segmentation has recently been established and applied to large-scale data processing, which potentially serves as an intermediate representation to high level video semantic extraction. The supervoxels are rich decompositions of video content: they capture object shape and motion well. Yet, it is not known if the supervoxel segmentation retains the semantics of the underlying video content. Here, my research studies the preliminary human perception of video supervoxel segmentation and its utilities in guiding the design of machine vision algorithms. We have conducted a systematic study of how well the actor and action semantics are retained in video supervoxel segmentation. Our study has human observers watching supervoxel segmentation videos and trying to discriminate both actor and action. We have gathered and analyzed a large set of 640 human perceptions over 96 videos at 3 different supervoxel scales. The ultimate findings have suggested that a significant amount of semantics have been well retained in the video supervoxel hierarchies. The results have further inspired our design of a new supervoxel feature. List below are relevant publications:
  1. C. Xu, R. F. Doell, S. J. Hanson, C. Hanson and J. J. Corso. A Study of Actor and Action Semantic Retention in Video Supervoxel Segmentation. International Journal of Semantic Computing (IJSC), 2013.
  2. C. Xu, R. F. Doell, S. J. Hanson, C. Hanson and J. J. Corso. Are Actor and Action Semantics Retained in Video Supervoxel Segmentation? In Proceedings of IEEE International Conference on Semantic Computing (ICSC), 2013.