The purpose of the visual measures is to transform the intensity images into a more compact and descriptive space, while still capturing the pose of the object.
We use real time visual feature trackers of three different kinds to obtain visual information. The Oxford snakes [4] are used to track surface discontinuities. A locally developed template matching tracker tracks multiple local features from surface markings or corners, and for reliability in repeated experiments, or to deal with smooth featureless surfaces, such as the lightbulb in fig. 6, we use special purpose trackers, tracking attached targets or small lights. To improve tracking, viewing geometry models are widely used. For instance the Oxford snake package uses an affine model to constrain the motion of the spline control points to rigid 3 D deformations, and an strain energy model for nonrigid image plane deformations. The convolution trackers uses point velocity for prediction.
The image Jacobian provides a new model for tracking.
As noted in section 2, the subspace of of possible
solutions
to
is of size
rather than
(and
). In our active framework the
agent also knows along which direction
the system
changes. This leaves only a one dimensional search space along
in feature space.
Note however that we cannot simply constrain
the tracker output to this space. That would take away the innovation
term in our model updating, and the system would no longer adapt
its model to a changing environment. Instead we use
to detect outliers
(e.g. stemming from occluded features or the tracker tracking the
wrong thing) and to constrain the tracking search window to a small
``cylinder'' around
. In a future development
we intend to use the predictor in a more general Kalman filter.
The idea in the subspace eigen image method is to project the raw
intensity values onto a basis of m eigen images. Representations
based on this idea have been used for recognition problem
``what'', and for indexing locations ``where''[20, 22, 21].
There are several ways to choose the
eigen images. In our case we will be looking at the same agent,
in different poses, and all the images we want to represent are
fairly similar. In this case it is advantageous to use a basis
specifically designed for the agent. In summary (see also
[20, 22])
this can be done by acquiring a (large) number p of size
images
of the
agent in different poses. Let the mean image
, and for each image
in the data set form the difference image
.
Form a measurement matrix
,
and calculate the covariance matrix
. The principal
components of this data are the eigenvectors to the matrix C.
The eigenvectors form an orthogonal basis for the original image
set, accounting for the variation in the data in decreasing order,
according to the corresponding eigenvalues. A dimensionality
reduction is achieved by using instead of all
eigenvectors
only a subspace of say the first
eigenvectors.
For practical reasons usually
, and the covariance
matrix C will be rank deficient. We can then save computational effort
by instead computing
and using the p eigenvectors
of L to form the m first eigenvectors
of A by
,
where
.
After a basis has been acquired, (which for a particular agent
typically only needs to be done once), any new image
can be
represented
in this basis as a perception vector
and a given
can be transformed
(with some quality loss)
into a corresponding image by the inverse formula
.