Adaptive Batch Size Schedules for Distributed Training of Language Models with Data and Model Parallelism
Jan 1, 2025·
,,,
·
0 min read
Tim Tsz-Kit Lau
Weijian Li
Chenwei Xu
Han Liu
Mladen Kolar
Authors
Post-doc (2023-2024)
Tim Tsz-Kit Lau was a postdoctoral principal researcher at the University of Chicago Booth School of Business. He received his PhD in Statistics and Data Science from Northwestern University. His research lies at the intersection of statistics, machine learning and optimization, including optimizer design, batch size strategies and distributed stochastic optimization for large models.
Authors
Professor of Data Sciences and Operations
Mladen Kolar is a Professor of Data Sciences and Operations at the University of Southern California Marshall School of Business and a Visiting Professor of Statistics and Data Science at Mohamed bin Zayed University of Artificial Intelligence. Before joining USC, he was on the faculty of the University of Chicago Booth School of Business. His research is focused on high-dimensional statistical methods, graphical models, varying-coefficient models and data mining, driven by the need to uncover interesting and scientifically meaningful structures from observational data. He is a Fellow of the Institute of Mathematical Statistics.