Provably training overparameterized neural network classifiers with non-convex constraints
Abstract
Training a classifier under non-convex constraints has gotten increasing attention in the machine learning community thanks to its wide range of applications such as algorithmic fairness and class-imbalanced classification. However, several recent works addressing non-convex constraints have only focused on simple models such as logistic regression or support vector machines. Neural networks, one of the most popular models for classification nowadays, are precluded and lack theoretical guarantees. In this work, we show that overparameterized neural networks could achieve a near-optimal and near-feasible solution of non-convex constrained optimization problems via the project stochastic gradient descent. Our key ingredient is the no-regret analysis of online learning for neural networks in the overparameterization regime, which may be of independent interest in online learning applications.
Type
Publication
Electronic Journal of Statistics, 16(2)
Authors
PhD (2016-2021)
You-Lin Chen was a statistics PhD candidate at the University of Chicago advised by Mladen Kolar. He pursues his research interests in machine learning, stochastic and non-convex optimization, high-dimensional statistics.
Authors
Professor of Data Sciences and Operations
Mladen Kolar is a Professor of Data Sciences and Operations at the University of Southern California Marshall School of Business and a Visiting Professor of Statistics and Data Science at Mohamed bin Zayed University of Artificial Intelligence. Before joining USC, he was on the faculty of the University of Chicago Booth School of Business. His research is focused on high-dimensional statistical methods, graphical models, varying-coefficient models and data mining, driven by the need to uncover interesting and scientifically meaningful structures from observational data. He is a Fellow of the Institute of Mathematical Statistics.