Spring 2022 - CSC 577: Advanced Machine Vision

Basic Information

Meet Time & Location: Tue/Thu 09:40-10:55 (Hylan Building 203, in-person only)
Instructor: Prof. Jiebo Luo
Office Hours: T/R 14:00-15:00

The complete/accurate information of the course should be found on Blackboard.


This course covers advanced research topics in computer vision. Approaches for learning from unimodal (e.g., images and videos) and multimodal data (e.g., vision and language) will be discussed, including topics from small data learning, video analytics, vision and language, 3D vision, image and video generation, trustworthy AI, vision for medicine, and multimedia. The course will be a mix of instructor-led lectures, student-led presentations, paper reading/critiquing, guest lectures, and a semester-long project (team of 2).


CSC 249/449 or CSC 246/446 or CSC 298/578 (Deep Learning and Graphical Models) or permission of the instructor.

Course Objectives

PhD Students: enriched research experience in or related to computer vision; MS Students: strengthened training for a software engineer career in computer vision and artificial intelligence.


This is a tentative schedule of the course and is subject to changes over the semester.
Course materials will be posted in Blackboard Learning Modules.

W Date Topic Comment
1 Thu 1/13 Course Introduction and Overview
Tue 1/18
T1 Vision and Language: Lecture by Prof. Jiebo Luo

Thu 1/20
T1  Video-aided Unsupervised Grammar Induction, Songyang Zhang, Linfeng Song, Lifeng Jin, Kun Xu, Dong Yu, Jiebo Luo, NAACL 2021 [Best Long Paper]
Songyang Zhang
Tue 1/25
T1 Improving Vision-and-Language Navigation with Image-Text Pairs from the Web. Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, Dhruv Batra. 2020. arXiv TBD

Thu 1/27
T2 Image and Video Generation
Wei Xiong
Tue 2/1 T2 Advanced generation guided by semantics and sentiments Haitian Zheng, Jie An

Thu 2/3 T2 Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering. Yuxuan Zhang, Wenzheng Chen, Huan Ling, Jun Gao, Yinan Zhang, Antonio Torralba, Sanja Fidler. 2020. arXiv TBD
Tue 2/8 T3 3D Pose and Motion: Lecture by Dr. Junsong Yuan (UB)

Thu 2/10
T3 HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation. arXiv
Jingyuan Chen
Tue 2/15
T4 Few-shot and Representation Learning: Lecture by Prof. Jiebo Luo

Thu 2/17
T4 Lecture by Prof. Xi Peng (UDel) Domain Generalization
Tue 2/22
T4 When Does Self-supervision Improve Few-shot Learning? Jong-Chyi Su, Subhransu Maji, Bharath Hariharan. 2020. arXiv

Thu 2/24
Project Proposal
Tue 3/1 T5 Computer Vision for Medicine: Lecture by Prof. Jiebo Luo

Thu 3/3 Automated Bone Mineral Density Prediction and Fracture Risk Assessment using Plain Radiographs via Deep Learning, Chen-I Hsieh*, Kang Zheng*, Chihung Lin, Ling Mei, Le Lu, Weijian Li, Fang-Ping Chen, Yirui Wang, Xiaoyun Zhou, Fakai Wang, Guotong Xie, Jing Xiao, Shun Miao#, Chang-Fu Kuo#, Nature Communications, 12, Article number: 5472 (2021)
Weijian Li
Tue 3/8 Spring Break

Thu 3/10 Spring Break
Tue 3/15 T6 Fairness/Biases in Computer Vision Wei Zhu

Thu 3/17
T6 Learning from Failure: Training Debiased Classifier from Biased Classifier. Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, Jinwoo Shin. 2020. arXiv TBD
Tue 3/22
Mid-term Project Discussion

Thu 3/24
T7 Multimodal Machine Learning
Multimodal Machine Learning: A Survey and Taxonomy,
Tadas Baltrusaitis, Chaitanya Ahuja, Louis-Philippe Morency. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 41, Issue 2, February 2019 pp 423–443. https://doi.org/10.1109/TPAMI.2018.2798607

Tue 3/29 T7 Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separation. Hang Zhou, Xudong Xu, Dahua Lin, Xiaogang Wang, Ziwei Liu. 2020. arXiv Ashwin Ramesh

Thu 3/31
T7 SoundSpaces: Audio-Visual Navigation in 3D Environments. Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicenc Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, Kristen Grauman. 2020. arXiv
Tue 4/5 T8 Video Analytics: Lecture by Prof. Haibin Ling or Prof. Michael Ryoo (Stony Brook)

Thu 4/7 T8 Procedure Planning in Instructional Videos. Chien-Yi Chang, De-An Huang, Danfei Xu, Ehsan Adeli, Li Fei-Fei, Juan Carlos Niebles. 2020. arXiv TBD
Tue 4/12 T9 Learning with Unpaired Data: Lecture by Prof. Jiebo Luo

Thu 4/14
Project Presentation 3
Tue 4/19
Project Presentation (cont.) 3

Thu 4/21
Project Presentation (cont.) 3
Tue 4/26
Project Presentation (cont.) 3

Course Work and Evaluation

All assignments should be submitted on Blackboard, and no late work without pre-approval will be accepted.

Other Policies

Code of Conduct: Check here for classroom "etiquette" expectations.

Academic Honesty: All assignments and activities associated with this course must be performed in accordance with the University of Rochester's Academic Honesty Policy. More information is available here. You are expected to read, understand, and follow the policy.

Disabilities: Please see me about your required accommodations as early as possible. The University of Rochester respects and welcomes students of all backgrounds and abilities. In the event you encounter any barrier(s) to full participation in this course due to the impact of a disability, please contact the Office of Disability Resources. The access coordinators in the Office of Disability Resources can meet with you to discuss the barriers you are experiencing and explain the eligibility process for establishing academic accommodations. You can reach the Office of Disability Resources at: disability@rochester.edu; (585) 276-5075; Taylor Hall; link.

Accommodations for COVID-related Illness: Please let me know as early as possible. At the same time, You should contact the College Center for Advising Services (CCAS) or the Graduate Education and Postdoctoral Affairs office. These offices will be able to assist you and refer you to the right resources.