| ARIGS | Image Mosaicking | 3-D Object Modeling |
|
This workspace augmented reality (AR) system serves as the test-bed of new AR techniques, new applications and human perception research. It is a stereo video-see-through head mounted display (HMD) based system. The system runs at real time with no lag between video and graphics. The delay is less than 100ms.
This system is a joint project of Department of Computer Science, University of Rochester , University of Rochester Medical Center and Siemens Corporate Research. System ComponentsCalibration, Tracking and AugmentingA metal plate with retro-reflective markers is used for both calibration and tracking. Tsai's method is used to calibrate all the 3 cameras. Once every camera is calibrated individually, the relative poses between the tracking camera and scene cameras are computed. We again use Tsai's method to determine the pose of the tracking camera (which is the pose of the HMD). The software runs in a server/client manner. The server computes the pose of the tracking camera and broadcasts to 2 clients which take care the augmenting of left eye and right eye respectively.Each client deduces the pose of its scene camera from the pose of the tracking camera. It then translates and rotates the virtual OpenGL camera according to the scene camera. Graphical objects are rendered through the 'calibrated' OpenGL camera. Live video is texture mapped to the background according to the view angle of the scene camera. Timing and Performance
The timing of the system is shown above. The left is the server side and the right is the client side. Starting with the tracker camera gets its frame, 2ms later, the markers are found and it takes Tsai 8 ms to compute the pose. Meanwhile, there is a sync handshake between the server/client, making sure the right frame is rendered. In other words,
we make sure that the virtual objects that augment a real scene are rendered from the camera pose that was valid when the particular image of the real scene was taken. This synchronization together with the stable pose estimation results in a very natural, believable impression of the augmented world. The virtual objects do not swim or jump or jitter, they are firmly anchored in the scene. We measured the time from having a video frame stored in memory to swapping the augmented frame into the front framebuffer to be 24 to 30 ms. Now the camera takes about one frame's time (33ms) to shoot one frame, and it may take about another frame's time to read the data into memory. And once the rendered image is in the front framebuffer, it will be displayed at the next vertical refresh of the monitor. At 60Hz refresh rate, that may take up to 16ms. Overall, we estimate the latency of the system, i.e. the time difference between a live action and its display in the augmented view, to be about 0.1 sec. This latency is small enough to become apparent only for fast movements.
Augmented Scene3D PerceptionPeople perceive depth by physiological cues, such as binocular disparity and motion parallax (kinetic cue), and psychological cues, such as perspective illusion, interposition (occlusion), size constancy and etc. For a user, one cue may be stronger than another. Our system works in binocular and the user can get a very good 3D sense from the disparity. The 3D perception can be further enhanced by the kinetic cue if a user moves his head around. However, psychological cues, especially occlusion, interfere most of our testers' perception. In the above candle-lighting video, the user has a very good idea of where the candle by just looking through the HMD. But when the user reaches out her/his hand to light it, no matter the red bar before the behind the candle in space it always looks behind the candle because the red bar belongs the background image. And here the occlusion comes in to interfere the user's 3D perception. We can solve this specific problem by tracking the red bar, but ultimately a dense depth map of the real scene should be known to solve the occlusion problem. It of course very hard to get a dense depth map on the fly, lest in real time. The way we are trying to solve or alleviate this problem is to use psychological cues over psychological cues. See next section for details.Application: IGSOne of the applications of the system is image guide surgery (IGS). An AR system gives surgeons the ability of X-Ray vision. In other words, the surgeon can see an augmented human tissue, say a tumor, inside the patient's head. Click to see large pictures and videos.PublicationLinks |