IE 510, Nonlinear Programming, TR 2-3:20 PM Spring 215, 203 Transportation
Building
Nonlinear optimization is about maximizing or minimizing not-necessarily-linear functions, possibly subject to constraints. It finds applicability in a variety of fields ranging from economics,
finance, statistics to engineering disciplines. This course will focus on the fundamental theory of
unconstrained and constrained optimization problems. We will discuss many different methods with
cross-cutting applicability and prove a number of results pertaining to the success or failure of these
techniques.
Besides students within IE, this course would be of interest to students from math, ECE, economics, computer science, mechanical engineering, and other engineering disciplines. No prior
background in optimization is necessary. However, please note that this will be a mathematically sophisticated class that will require you to be extremely comfortable with writing rigorous
proofs. The course will involve homework assignments with challenging problems which will require
creativity and inventiveness on your part to solve. I strongly recommend you to avoid taking this
course unless you have a strong mathematical background.
We will cover a broad range of topics.
Fundamentals of Nonlinear Optimization;
• Descent methods: line search, curve fitting, steepest descent, Newton’s method.
• The conjugate gradient method and its applications.
• Quasi-Newton methods, penalty, barrier, and augmented Lagrangian methods.
• KKT conditions and duality in nonconvex optimization.
• Dual, primal-dual, and cutting plane methods.
• Acceleration of gradient descent and optimal methods.
• Algorithms for semidefinite programming.
• When is an optimization problem a semidefinite program?
• Sequential quadratic programming.
Advanced Topics, as time permits:
• Algorithms for compressed sensing, matrix completion, and statistical estimation problems.
• Connections between online learning and optimization algorithms.
• The ADMM (alternating direction method of multipliers) and its applications in learning
theory and distributed optimization.
Grading: Homework (50%) and Project (50%).
I anticipate assigning a homework every two weeks. The final project will involve a report and
possibly a presentation on a paper of your choice from a list provided by the instructor.
Textbook: Linear and Nonlinear Programming, D. Luenberger, Y. Ye, Springer, 2008.
Course Instructor: Alex Olshevsky, 145 Coordinated Science Labs, aolshev2@illinois.edu.
Beyond the textbook (which we will follow pretty closely for about half of the course), we will
cover at least a little bit from each of the following sources:
• D. P. Bertsekas, Nonlinear Programming, Athena Scientific, 1995.
• A. Nemirovski, Numerical Methods for Nonlinear Continuous Optimization, lecture notes,
available at http://www2.isye.gatech.edu/~nemirovs/Lect_OptII.pdf.
• M. Epelman, Continuous Optimization Methods, lecture notes, available at http://www-personal.
umich.edu/~mepelman/teaching/IOE511/511notes.pdf.
• S. Bubeck, Theory of Convex Optimization for Machine Learning, monograph draft, available
at http://www.princeton.edu/~sbubeck/Bubeck14.pdf.
• A. Ben-Tal, A. Nemirovski, Lectures on Modern Convex Optimization, monograph draft, available at http://www2.isye.gatech.edu/~nemirovs/Lect_ModConvOpt.pdf
• S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein, Distributed Optimization and Statistical
Learning via the Alternating Direction Method of Multipliers, monograph draft, available at
http://stanford.edu/~boyd/papers/pdf/admm_distr_stats.pdf