Optimizing Synchronization for Large-scale Distributed Training

Abstract

The rapid growth of machine learning (ML) model parameter sizes and the floating-point operations required for training has made training workloads increasingly large in both the number of participating GPUs and geographic scope. Frontier model training jobs now span multiple data center buildings and may span multiple regions if current scaling trends continue. At the same time, the need to train on fresh client data without raising privacy concerns has made federated learning across geographically decentralized clients increasingly important. As training scales, synchronization becomes increasingly costly. Within each training round, workers exchange large volumes of model state or training intermediates. Synchronization that cannot be overlapped with useful computation leaves compute resources idle and prolongs the round time, especially over oversubscribed and high-latency networks. The cumulative cost of synchronization also depends on how many rounds are required to reach the target model quality. In centralized training, increasing the global batch size can reduce synchronization frequency, but the convergence benefit diminishes beyond a critical batch size. In federated learning, clients tend to have heterogeneous data with different statistical characteristics, and naive synchronization that aggregates per-client updates can slow or destabilize model convergence. Therefore, the end-to-end synchronization cost is the product of the visible synchronization cost within each round and the number of rounds. Optimizing this cost faces three recurring deployment challenges: algorithm optimizations incur system cost, partition granularity complicates computation-communication overlap, and network and workload conditions change across deployments. This thesis argues that end-to-end synchronization cost can be substantially reduced by targeting the dominant cost source at each deployment scale, while accounting for factors that affect the actual performance benefit of each optimization. To address the aforementioned challenges, we follow three design principles: cost-aware algorithm design to balance synchronization acceleration and system overhead, communication-aware computation partitioning that considers the resulting communication pattern to maximize overlap, and a general framework that adapts optimization choices to diverse network and workload conditions. To demonstrate the thesis statement, we present three systems. At the intra-building training scale, THC reduces the visible synchronization cost per round for data-parallel training. It introduces tensor homomorphic compression, which enables direct aggregation of compressed gradients and avoids paying the decompression and re-compression costs at parameter servers or in-network aggregators. At the cross-building and cross-region scale, ScaleAcross Explorer continues to reduce the visible synchronization cost per round. It proposes a workflow that combines testbed characterization, production-validated simulation, and cross-stack optimization that holistically configures parallelism placement, scheduling, microbatching, model chunking, and network-layer technologies to guide scale-across training configuration. At the geographically distributed federated learning scale, Fielding reduces the number of synchronization rounds needed to reach the target model quality under data heterogeneity and drift. It supports selecting client data representations based on the targeted data drift type and performs selective global re-clustering, and provides convergence guarantee with a heterogeneity bound estimated at runtime. Together, these systems reduce the visible synchronization costs in end-to-end distributed ML training across three deployment scopes: from data-parallel synchronous training within a data center building, to multi-dimensional parallel training for both dense and Mixture-of-Experts models across data center buildings and regions, to federated learning over heterogeneous data at geographically distributed clients.