Optimization for Machine Learning, Fall 2024

This course primarily focuses on algorithms for large-scale optimization problems arising in machine learning and data science applications. The first part will cover various first-order methods including gradient and subgradient methods, mirror descent, proximal gradient method, accelerated gradient method, Frank-Wolfe method, and dual methods. The second part will survey topics in machine learning from an optimization perspective, e.g., stochastic optimization, distributionally robust optimization, online learning, and reinforcement learning.

Course Information

• YouTube videos from Fall 2023

• Syllabus

• Instructor: Jiaming Liang

• Teaching Assistant: Lin Zang

• Meeting Information: 9:40-10:55 am, Tuesday/Thursday, Hylan Building 203

• Office Hours

  • 4:00-5:00 pm, Wednesday, Wegmans Hall 2403 (Jiaming Liang)

  • 4:00-5:00 pm, Friday, Wegmans Hall 1219 (Lin Zang)

• Textbooks

  • Amir Beck. First-order methods in optimization. SIAM, 2017.

  • Yurii Nesterov. Lectures on convex optimization. Springer, 2018.

• Recommended Readings

  • Fatma Kilinc-Karzan and Arkadi Nemirovski. Mathematical Essentials for Convex Optimization. Cambridge University Press, 2024+.

  • Guanghui Lan. First-order and Stochastic Optimization Methods for Machine Learning. Springer, 2020.

  • Benjamin Recht and Stephen Wright. Optimization for Data Analysis. Cambridge University Press, 2022.

  • Suvrit Sra, Sebastian Nowozin, and Stephen Wright, eds. Optimization for Machine Learning. MIT Press, 2011.

Topics

• Introduction

  • Applications in ML [slides]
    

  • Convexity and complexity [notes]
    

• First-order methods I: basic concepts

  • Gradient method [notes]
    

  • Subgradient method [notes]
    

  • Mirror descent [notes]
    

  • Proximal gradient method [notes]
    

  • Accelerated gradient method [notes]
    

  • Frank-Wolfe method [notes]
    

• First-order methods II: advanced topics

  • Dual methods [notes]
    

  • Stochastic approximation [notes]
    

  • Smoothing techniques [notes]
    

• Selected topics in machine learning

  • Online Optimization [notes]