CSC 576, Fall 2016, Advanced Machine Learning and Optimization

Lecturer: Ji Liu, Office hour: Tue & Thu 4:405:40pm
TA: Haichuan Yang (hyang1990.cs@gmail.com) TA office hour: Mon. 1012pm at CSB 725 hyang1990.cs@gmail.com
Course Description

Data analytics is the key component of machine learning and optimization. It refers to the process of collecting, organizing, and analyzing large sets of data ("big data") to discover patterns and other useful information. Not only will big data analytics help you to understand the information contained within the data, but it will also help make predictions and decisions. This course introduces advanced machine learning modeling skills, optimization algorithms, and theories to big data analytics.
 computational methods for large scale linear systems
 matrix / tensor decomposition / completion / recovery (recommendation system, image/video inpainting, video surveillance)
 1st order optimization methods: unconstrained optimization methods (gradient descent, optimal 1st order method, stochastic gradient method), constrained optimization methods (ADMM, smoothing scheme)
 feature selection and sparse learning (L1 norm minimization, nuclear norm, group sparsity, greedy algorithms)
 others, e.g., submodular optimization
Mathematics is heavily involved in this course. It especially emphasizes deep understanding of fundamental models and algorithms. This course would NOT cover any software or tools used in data analytics. Basically, this is NOT an application or software teaching course.
The following topics will be covered
Basic knowledge in linear algebra, matrix analysis, and statistics, and basic concepts in optimization (such as convex set / function) are useful for you to follow this course smoothly. Research experiences in machine learning, signal processing, and computer vision / image processing would be definitely helpful to understand this course better, but not required.
Course Schedule
 Section 0: Overview [slides] [syllabus]
 Section 1: Review for Linear Algebra and Matrix Analysis [notes] [referenceStanfordCS229]
 Section 2: (Large) Linear System [notes]
 Section 3: Matrix Decomposition / Completion [notes] [slides] [other materials]
 Section 4: Sparsity: Sparse Learning and Compressed Sensing [notes] [reference 1] [reference 2] [course notes]
 Section 5: Variants in Sparse Learning and Compressed Sensing [notes]
 Section 6: Optimization: Foundations [notes] [reference book]
 Section 7: Gradient Descent [notes] [reference book] [reference paper]
 Section 8: Coordinate Descent [notes]
 Section 9: Accelerated (Optimal) Gradient Descent [notes] [reference paper]
 Section 10: Stochastic Gradient "Descent" [notes]
 Section 11: ADMM [notes] [reference book]
 Section 12: Feature Selection [notes]
 Section 13: Submodular Optimization [notes]
Midterm

Nov. 3, In Class
Homework
 Homework 1 (Due on Sep. 22) [pdf]
 Homework 2 (Due Oct. 6) [pdf] [training] [testing]
 Homework 3 (Due Oct. 25) [pdf]
 Homework 4 (Due Nov. 10) [pdf] [dataset]
 Homework 5 & 6 (Due Nov. 24) [pdf] [dataset]
Paper Presentation
Project
 Project Proposal (Due Oct. 27 before the class): Submit a proposal with 12 pages to describe the project you want to work on. This proposal can be a part of your final project. You can organize your own team with up to 2 members. Each team only needs submit one proposal. Besides of the options provided in syllabus, you can choose to participate a data completion, for example, you google data completion and it may return some webpage like [kaggle]. If you decide to choose a data competition as your project, an active one or a recent (inactive) one is preferred. In addition, your team is allowed to contain up to 3 members (this is the only exception), considering that there is usually more work to do in data competition. The topic chosen in the proposal can only be changed in some very rare cases with additional penalty of losing 20% points. So please be serious on your proposal.
 Project Presentation: cross and peer validation on Dec. 6, 8, 13, and 15 in class. Each team will present its project to the rest students in the class, who will grade the presentation performance. Each presentation includes 15 mins for speakers + 3 mins for Q&A. The weight of presentation is 30% and the weight of the final project report is 70%.
 The weight of the final project report is roughly 70%. It is due on Dec. 22 midnight. Please submit only one report for each team through blackboard.