Monday, November 21, 2016
12:00 PM
CSB 209
Chris Kanan
Rochester Institute of Technology
Visual Question Answering: Algorithms, Datasets, & Challenges


Algorithms for extracting semantic information from images and video have dramatically improved over the past four years, with today’s best deep convolutional neural networks (CNNs) now rivaling humans at image recognition. These successes have prompted researchers to pursue building new systems that are capable of a multitude of tasks. In Visual Question Answering (VQA), an algorithm is given a text-based question about an image, and it must produce an answer. Although the first VQA datasets were released less than two years ago, algorithms are already approaching human performance. However, these results may be misleading due to biases in existing benchmarks. In this talk, I review the current state of VQA algorithms, including algorithms from my lab. I then analyze existing datasets for VQA and demonstrate that they have severe flaws and limitations. Lastly, I discuss what a better dataset would look like, and examine which kinds of questions are easy and which are hard for today's best algorithms.


Christopher Kanan is an assistant professor in the Chester F. Carlson Center for Imaging Science at the Rochester Institute of Technology. His lab uses machine learning, especially deep learning, to solve problems in computer vision, with an emphasis on task-driven algorithms for understanding images and videos. He is also working on incorporating brain-inspired mechanisms into neural networks. Dr. Kanan received a Ph.D. in computer science from the University of California at San Diego, where he worked on brain-inspired algorithms for object recognition, neural networks, active vision, and cognitive modeling. He received an M.S. in computer science from the University of Southern California. Before coming to RIT, Dr. Kanan was a postdoctoral scholar at the California Institute of Technology, and later worked as a Research Technologist at NASA’s Jet Propulsion Laboratory, where he used deep learning to develop vision systems for autonomous ships.