Schedule (Tentative)

Below is a list of topics that we will cover. The actual schedule will be posted/updated as the semester unfolds. Overall, we will start with fundamental building blocks and the move onto high-level application domains. Topics are organized as modules, each of which will begin with a sequence of lectures led by the instructor followed by a set of paper discussions led by students. Programming assignments will be interspersed throughout the semester. Expect 2~3 programming assignments.

Capturing Visual Data: Camera Systems
  • Optics basics
  • Human visual systems
  • Image sensors
  • Display technologies
  • Display controller
  • Modern digital camera pipeline (basic ISP pipeline, high dynamic range, portrait mode, etc.)
  • Case-study on Google Pixel Phones
Interpreting Visual Data: Computer Vision, Image/Video Processing
  • Hand-crafted features
  • Learnt-features, deep learning
  • Monocular/stereo vision
  • Video compression and streaming
  • 360-degree video
  • Efficient deep learning (pruning, quantization, hardware acceleration)
  • Optical deep learning, optical computing (PSF, phase masks, DOE, metasurface)
  • Systems and architecture for real-time (live) computer vision
  • Large-scale video analytics at data-center scale
  • Event-based cameras (neuromorphic computing)
Generating Visual Data: Computer Graphics
  • Rasterization
  • Ray-tracing
  • Graphics pipeline
  • Texture
  • Hardware support for graphics (GPU)
  • Real-time graphics, hardware for real-time ray-tracing
  • Rendering and vision co-design
  • Physics simulation and physically-based modeling
Beyond Pixels: Point Cloud
  • Point cloud data acquisition and representation
  • Voxels, spatial data structures
  • Point-based graphics/rendering
  • 3D printing
  • Point cloud compression
  • Point cloud analytics (deep learning models, reconstruction, registration, hardware support)
Augmented and Virtual Reality
  • Why the hype and why are they hard? (design constraints and objectives)
  • Use-cases (entertainment, military, business, cultural heritage preservation, etc.)
  • VR video capturing, streaming, rendering
  • Real-time graphics for VR (quality vs. speed trade-off)
  • Optics and display for AR/VR
  • Vergence-accommodation conflict, multi/vari-focal display, lightfield display
  • Optics/sensing/computing co-design
Autonomous Machines (Robots, Drones, Self-Driving Cars)
  • Why the hype and why are they hard? (design constraints and objectives)
  • Sensing, perception, and planning
  • Localization (Kalman filter, bundle adjustment, VIO/SLAM)
  • Case-study on autonomous micromobility vehicles