Advancing Multi-Agent Systems with Scalable and Robust Learning and Control

Abstract

Many modern infrastructures—such as smart cities, power grids, and transportation networks—are inherently multi-agent systems. Designing effective coordination mechanisms in these settings is challenging due to model uncertainty, scalability constraints, and unaligned agent incentives. This dissertation addresses these challenges by developing scalable and efficient learning-based control algorithms for multi-agent systems with provable and verifiable performance guarantees. The work is organized into three major parts. Part I focuses on designing scalable control and reinforcement learning (RL) algorithms for networked systems. In large-scale cyber-physical systems—such as smart grids, intelligent buildings, and traffic networks—agents are often embedded in graph structures where coordination relies on local interactions and communication. Distributed control and RL become essential due to communication constraints and the need for scalability. This part delves into the fundamental capabilities and sample-based design of distributed control and RL algorithms for networked systems. By leveraging the underlying network topology, we demonstrate that distributed controllers can achieve near-optimal global performance (Chapter 2). Furthermore, we develop distributed RL algorithms that are both communication- and sample-efficient, providing theoretical guarantees alongside strong empirical results (Chapter 3). Part II investigates strategic behavior in multi-agent systems. In applications such as traffic, trading and energy market, systems are generally comprised of agents that may act non-cooperatively due to unaligned incentives. In such settings, the goal shifts from achieving global optimality to finding a Nash equilibrium. In Chapter 4, we develop efficient, data-driven algorithms for Nash equilibrium seeking using multi-agent reinforcement learning (MARL). Building on the insight that all first-order stationary points correspond to Nash equilibria in Markov potential games, we derive sample-based algorithms to compute them effectively using gradient-based methods. In Chapter 5, we take a step further by exploring equilibrium selection methods aiming at promoting socially optimal outcomes. We propose a unified framework that systematically integrates the sequential structure of multi-agent reinforcement learning (MARL) with equilibrium selection, enabling agents to converge to equilibria that are both stable and socially desirable. Part III addresses robustness and risk sensitivity in uncertain environments. Real-world systems often operate under imperfect models, noisy data, and external disturbances. To ensure reliable performance under such conditions, we develop robust and risk-sensitive RL algorithms. These include formulations of soft robust Markov Decision Processes (MDPs) and risk-aware policy optimization techniques with theoretical convergence guarantees. Together, these contributions advance the theoretical and practical frontiers of learning and control in multi-agent systems. The algorithms developed in this work are validated across a range of real-world-inspired applications, including robotics, smart buildings, and energy management. This thesis lays the groundwork for resilient, efficient, and cooperative autonomous systems in increasingly complex and uncertain environments.