We improve the human-computer interaction experience in AR systems by simulating global lighting effects between virtual and real objects.
Bo Hu (Ph.D. Thesis Work)
Chris Brown (Thesis Advisor)
Randal Nelson (Thesis Committee)
Kyros Kutulakos (Thesis Committee)
A. Murat Tekalp (Thesis Committee)
Andrew Choi (Designer of UI to Radiance)
Bo Hu
Block diagram of the system. We capture images covering the whole environment
using a pan-tilt camera and generate an image mosaic from the images. We then
use a semi-automatic method to create a lighting model. We also approximate
the 3D geometry of scene object using a level-set based algorithm. With the
lighting model and the 3D geometry we can render both virtual and real objects
with correct lighting. Blending the rendering and input images, we can
created augmented images.
The goal of this research is to improve human-computer interaction. Computer-generated
graphics can be inserted into live video feeds for several purposes:
training, enhancement of content, memory aid, design. If the graphical
objects were in fact real, they would interact with objects in the scene
(obscure them, cast shadows on them). More veridical renderings of these
physical interactions would enhance the quality and possibly the usefulness
of the augmented display.
Augmented reality mixes computer-generated graphics with real-world images
(in our case, video streams). Ideally, such mixing is undetectable (as it
is in many movie post-production effects) and real-time (unlike movies).
From flight simulators to surgical training, heads-up displays to assembly
and maintenance assistants, design to advertising, augmented reality (AR)
is becoming a technically viable supplement to, or even alternative to, the
real world. But AR is currently limited by the difficulty of simulating
interactions between real and virtual objects. We are developing the basic
science, algorithms and implementations for improved ``interaction'' of
virtual and real entities in augmented reality applications. This project
develops theory and tools for AR with real-virtual interactions: real
objects obscuring virtual objects and vice versa, real lighting used for
virtual objects, real objects casting shadows on virtual objects and vice
versa.
http://www.cs.rochester.edu/u/bh/research.html
Augmented Reality, AR, camera calibration, pose estimation, image mosaics, lighting modeling, lighting simulation, 3D reconstruction, levelset method, shape from silhouette, curvature flow, heat flow, PDE, computer vision.