Visual computing algorithms recognize, manipulate, and synthesize massive visual data such as images, videos, and 3D graphics. These algorithms enable new use-cases on emerging mobile/embedded devices such as AR/VR headsets, robotics, and energy harvesting systems. The computer systems challenge is that many emerging mobile visual computing applications encompass five distinct domains that have been traditionally considered separately: imaging, vision, graphics, video processing, and acoustics. While algorithms from these domains are exercised in today’s mobile applications, they are mostly isolated from one another throughout the system stack and rarely interact in an application. As a result, today’s mobile computing platform is a collection of segregated sub-systems, each specialized for a domain.
Future mobile visual applications require the five domains to operate collectively and synergistically. Consider an Augmented Reality application; it must perform imaging to convert camera sensor data to continuous frames, trigger vision algorithms to extract semantics information from the frames, and draw virtual graphical contents to augment the real scene, all the while producing an immersive acoustic experience and encoding the generated video content for future playback. The fusion of the five computing domains presents unprecedented opportunities to co-design and co-optimize the five domains simultaneously. Simply stitching together systems designed for the individual domains loses bigger opportunities and leads to an inefficient whole.
Our research, instead, designs Systems-on-a-chip (SoC) architectures that exploit the synergies across domain-specific accelerators. We investigate the right programming and OS abstractions to enable a horizontal, cross-domain design, and are interested in understanding how the horizontal, cross-domain design can be done while keeping modular development and compositional reasoning.