No Free Lunch for Hungry Machines: A Systems Investigation of Reinforcement Learning

Abstract

LLMs have gained rapid popularity in recent years, with the ability to align outputs to human preferences and objectives. Eager end users have caused surging demand for LLM inference. This has motivated new research to improve efficiency, reduce costs, and better understand systems in inference serving, alongside decades of similar research in distributed training. Meanwhile, the wave of systems- and hardware-oriented reinforcement learning (RL) research has just begun. RL offers a unique paradigm that combines inference, preparation, and training phases. Although RL has been a popular arena for algorithmic inventions, there is still a need to better understand how RL policies and phase workloads affect underlying systems, beyond algorithms. This requires carefully designed experiments that vary key axes to test system effects. To address this gap, this thesis conducts experiments along four axes: model family/policy, reward paradigm, task/dataset, and scaling. We build profiling extensions atop VeRL and collect metrics such as power, latency, energy, and other system-level metrics at the granularity of phases and subphases—a novel contribution in RL. We find that different RL policies induce distinct phase signatures, driven both by time spent in phases and by extra computational machinery. We quantify the costs of different reward-scoring paradigms—a learned reward model or an explicit reward function—in data movement and extra computation. We find that different tasks and datasets alter the response-length structure and rollout imbalance, contributing to latency differences. Finally, we outline the best use cases for different scaling configurations on H200 and A100 GPUs, which are strongly dependent on phase and stability requirements. We hope that our work may motivate new RL algorithms that address the less desirable attributes of phases, advance asynchronous RL methods that lower transition costs between phases, and inform data centers and cloud operators about the distinct workloads of RL. Overall, we hope that the profiling suite and metrics pipeline will inspire further energy-aware innovation in RL.