Monday, October 07, 2019
12:00 PM
1400 Wegmans Hall
Jivko Sinapov
Tufts University
Learning About Objects Through Behavioral Exploration and Multisensory Perception
Infants use exploratory behaviors to learn about the objects around them. Psychologists have theorized that behaviors such as grasping touching, pressing, and lifting, coupled with the visual, tactile, haptic and auditory sensory modalities, enable infants to form grounded object representations. For example, scratching an object can provide information about its roughness, while lifting it can provide information about its weight. In a sense, the exploratory behavior acts as a ``question'' to the object, which is subsequently ``answered" by the sensory stimuli produced during the execution of the behavior. In contrast, most object representations used by robots today rely solely on computer vision or laser scan data, gathered through passive observation. Such disembodied approaches to robotic perception may be useful for recognizing an object using a 3D model database, but nevertheless, will fail to infer object properties that cannot be detected using vision alone.

To bridge this gap, our research has pursued a developmental framework for object perception and exploration in which the robot's representation of objects is grounded in its own sensorimotor experience with them. In this framework, an object is represented by sensorimotor contingencies that span a diverse set of exploratory behaviors and sensory modalities. In this talk, I will highlight results from several large-scale experimental studies which show that the behavior-grounded object representation enables a robot to solve a wide variety of perceptual and cognitive tasks relevant to object learning. I will discuss recent work on how robots can ground language in multisensory experience with objects and will conclude with a discussion on open problems in multisensory symbol grounding, which, if solved, could result in the large-scale deployment of robotic systems in real-world domains.

Jivko Sinapov received his Ph.D. in computer science and human-computer interaction from Iowa State University (ISU) in the Fall of 2013, after completing the BSc. in Computer Science at the University of Rochester. While working toward his Ph.D. at ISU's Developmental Robotics Lab, he developed novel methods that enable robots to learn the common-sense properties of objects and tools through behavioral object exploration and multi-modal perception. He went on to be a clinical assistant professor with the Texas Institute for Discovery, Education, and Science at UT Austin and a postdoctoral associate working with Peter Stone at the UTCS Artificial Intelligence lab. Jivko Sinapov joined Tufts University in the Fall of 2017 as the James Schmolze Assistant Professor in Computer Science. Jivko's research interests include cognitive and developmental robotics, computational perception, human-robot interaction, and reinforcement learning