Memory-centric AI: Optimizing Memory Systems for Large-Scale AI

Abstract

Artificial Intelligence (AI) has made rapid progress in both accuracy and efficiency over the past decade. Early deep learning models were limited to relatively simple tasks like image classification. In contrast, modern large language models (LLMs) now solve complex reasoning problems and enable autonomous systems in domains such as vision and navigation. At the same time, the cost of deploying frontier models has decreased dramatically. For example, inference with ChatGPT fell from $10 per million input tokens in 2023 to $1.25 in 2025, an 8×reduction within a year, demonstrating simultaneous advances in algorithms and systems efficiency. These improvements have been largely driven by scaling model size, which has unlocked emergent capabilities but also created significant computational and memory challenges. Training and serving large models require storing billions of parameters and moving data at extremely high throughput, making memory a critical bottleneck. While reducing memory overhead is essential, the ultimate goal of AI systems remains accuracy. To bridge this gap, this dissertation introduces a new metric: accuracy-per-byte, which quantifies the predictive accuracy obtained per unit of data movement. This metric reframes AI progress through the lens of memory efficiency, complementing traditional measures such as FLOPs-per-byte. Achieving higher accuracy-per-byte demands a holistic, cross-layer approach that spans algorithms, software systems, and hardware architectures.