Thesis Proposal

How to find Dana

Keywords: Active Vision, Video Surveillance, Unsupervised Learning, Ubiquitous Computing.

The main purpose of this research is to find a universally reliable mechanism to keep the location and status of Professor Dana H. Ballard available to grads of the Department of Computer Science at the University of Rochester. Given the fact that he doesn't have a cell phone and doesn't answer phone rings or doorbells, this is the greatest challenge to various technologies of today, e.g. Video Surveillance, Wireless Communication, Global Positioning System, Speech Recognition, Sensor Networks and Machine Learning. The method we propose to attack this problem is robot swarm with p2p communication. This thesis proposal will define the problem I am going to address, defend its significance, provide relative background knowledge, summarize previous attempts (aka failures), describe the methodology, and predict a final product.






Well, this was improvised after Dana, my adviser, forgot to attend a group meeting two years ago. Now I am in my 5th year. Maybe it is a good idea to put together a brief description of my research on my web page.

Generally, my research interests lie in the boundary of machinery computation and natural intelligence. I would like to see how the nature, including human and human society, deals with complex computational tasks, and on the other side, how artificial computation could possibly reproduce natural intelligence, either behaviorally or structurally, and surpass it.
All I said above is remotely related to my thesis work. In everyday research, I am doing one of the three projects.

Visual Routines This is a hypothesis by Ullman saying that the brain has some neural circuits which implement various popular visual behavior segments like finding your car in a parking lot. When you need one, you just load it and fire. A routine would subconsciously execute a sequence of primitive operations, and you don't have to know what they are and how they are programmed. These basic operations constitute an instruction set, which, analogous to that in a digital computer, defines all building blocks of arbitrarily complex visual behaviors.
We implemented this model in a situated manner. Most of the time people are not just sitting there, rolling their eyes, and reconstructing a 3D model of the world. Rather, vision is used to retrieve information which is necessary for solving everyday tasks like making a coffee. In this sense, vision always has a purpose: facilitating human-world interaction. Inevitably, we incorporated motor operations, e.g. grabbing an object.
So we have a robot whose vision and motion is fully controlled by routines. On the left is its passport photo. Too realistic to be true? Well, it is in virtual reality. Silicon Graphics Inc. has a virtual reality platform called Performer, which provides basic rendering and scene manipulating functionalities. On top of that, a graphic human model, bought from Boston Dynamics Inc. (which is still in business because they just updated their web page), is given binocular vision. What it sees is processed by a dedicated computer, the mind of that virtual human. This computer analyzes visual stimuli and sends commands to the body to move eye balls and other body parts, which, in turn, could produce new visual stimuli.
Behavior Modeling
Everyday life tasks could have multiple solutions. For example, in making a peanut butter and jelly sandwich, some people spread peanut butter first, while others spread jelly first. People break down a task in similar ways but the order of solving individual subtasks is highly variable. We are interested in how to automatically identify subtasks and how to model the variation of execution orders.
The approach we are taking, is to parse natural behaviors into routines, and look for similarities among routines used for the same task. A string segmentation program takes these routines as input and produces a segmentation such that every input routine can be reconstructed by concatenating these segments in a certain order.
With this segmentation, a task can be modeled by a Markov model, shown on the left. If you roll a dice, this model would produce a complete recipe of making a sandwich.
Look-aheads
In natural human-world interaction, where eyes are used to guide hand movement, fixation points normally appear around hands. However, an interesting observation is that people use fixation proactively, i.e. well before it is "needed". An example is, while you are pouring bottled coke into a cup, you don't alway stare at the cup, you might take a quick look at something else, say, the cap of the bottle, or some napkins, then come back to the cup. This indicates, (a) multiple objects in the view are crying out for your gaze, (b) your eyes and hands can work in parallel in some extent, (c) a cognitive plan partially determines which object to look at, (d) this process is stochastic, and (e) visual memory is involved. We are currently trying to build a computational model for this phenomenon.