| Visual Routines |
This is a hypothesis by Ullman saying that
the brain has some neural circuits which implement various popular visual
behavior segments like finding your car in a parking lot. When you need
one, you just load it and fire. A routine would subconsciously execute
a sequence of primitive operations, and you don't have to know what
they are and how they are programmed. These basic operations constitute
an instruction set, which, analogous to that in a digital computer,
defines all building blocks of arbitrarily complex visual behaviors. We implemented this model in a situated manner. Most of the time people are not just sitting there, rolling their eyes, and reconstructing a 3D model of the world. Rather, vision is used to retrieve information which is necessary for solving everyday tasks like making a coffee. In this sense, vision always has a purpose: facilitating human-world interaction. Inevitably, we incorporated motor operations, e.g. grabbing an object. So we have a robot whose vision and motion is fully controlled by routines. On the left is its passport photo. Too realistic to be true? Well, it is in virtual reality. Silicon Graphics Inc. has a virtual reality platform called Performer, which provides basic rendering and scene manipulating functionalities. On top of that, a graphic human model, bought from Boston Dynamics Inc. (which is still in business because they just updated their web page), is given binocular vision. What it sees is processed by a dedicated computer, the mind of that virtual human. This computer analyzes visual stimuli and sends commands to the body to move eye balls and other body parts, which, in turn, could produce new visual stimuli. |
Behavior Modeling![]() |
Everyday
life tasks could have multiple solutions. For example, in making a
peanut butter and jelly sandwich, some people spread peanut butter
first, while others spread jelly first. People break down a task in
similar ways but the order of solving individual subtasks is highly
variable. We are interested in how to automatically identify subtasks
and how to model the variation of execution orders. The approach we are taking, is to parse natural behaviors into routines, and look for similarities among routines used for the same task. A string segmentation program takes these routines as input and produces a segmentation such that every input routine can be reconstructed by concatenating these segments in a certain order. With this segmentation, a task can be modeled by a Markov model, shown on the left. If you roll a dice, this model would produce a complete recipe of making a sandwich. |
Look-aheads![]() |
In natural human-world interaction, where
eyes are used to guide hand movement, fixation points normally appear
around hands. However, an interesting observation is that people use fixation
proactively, i.e. well before it is "needed". An example is, while you
are pouring bottled coke into a cup, you don't alway stare at the cup,
you might take a quick look at something else, say, the cap of the bottle,
or some napkins, then come back to the cup. This indicates, (a) multiple
objects in the view are crying out for your gaze, (b) your eyes and hands
can work in parallel in some extent, (c) a cognitive plan partially determines
which object to look at, (d) this process is stochastic, and (e) visual
memory is involved. We are currently trying to build a computational model
for this phenomenon. |