Understanding and Improving Optimization Algorithms in Deep Learning

Abstract

Deep learning has achieved remarkable empirical success across domains such as natural language processing, computer vision, and generative modeling, driven in large part by scalable stochastic optimization methods. Yet, despite their widespread use, many fundamental aspects of these algorithms remain poorly understood. This thesis develops a unified perspective on understanding and improving optimization algorithms in deep learning, combining theoretical analysis with empirical investigation at modern scale. First, we study the implicit biases of gradient-based training in highly overparameterized models. We analyze how both the geometry of the objective and the choice of optimization algorithm influence the selection of solutions among many interpolating minima. Through structured tasks and theoretical characterization, we show how optimization induces specific representational biases that help explain the emergence of structured solutions in deep networks. To gain analytical insight into large-scale training dynamics, we analyze an infinite-dimensional quadratic model as a tractable abstraction. This framework isolates key phenomena—such as bias–variance trade-offs —while remaining amenable to precise analysis using elementary linear algebra tools. It serves as a foundation for understanding optimizer behavior in practical settings. We then turn to scalability. By systematically studying critical batch size (CBS), we characterize the limits of near-linear speedups in large-scale language model training. We disentangle the roles of model size and data size, showing that CBS grows primarily with training duration rather than width or depth once models are sufficiently large. We further propose principled batch size scheduling strategies for improved serial efficiency. Finally, we revisit two central components of modern optimizers: momentum and preconditioning. We clarify the role of momentum in stochastic regimes, analyze why classical acceleration does not directly translate to SGD, and reinterpret recent large-scale optimizers through a unifying lens. We develop a theoretical understanding of structured preconditioning methods, analyzing how existing preconditioners approximate the curvature matrix. Building on this perspective, we design improved optimizers and characterize the practical limits of second-order methods in large-scale deep learning. Together, these contributions advance both the theoretical understanding and practical design of optimization algorithms for deep learning.