Toward Bandwidth-Centric Architectures for Fast and Efficient Large Language Model Inference

Abstract

Along our journey toward Artificial General Intelligence, we have arrived at a new class of algorithms that enable AI systems to reason and act independently. Large language models (LLMs) are solving new problems through logic, rather than through memorization. At the heart of this shift is token generation, the fundamental unit through which intelligence is expressed, and increasingly, the currency of modern AI systems. The faster we can generate tokens, the more effective AI becomes. This means latency per token is the critical bottleneck, determining how quickly AI systems can execute multi-step reasoning, carry out agent workflows, and produce useful outputs. During my PhD, I studied how to design systems for low-latency inference. This required fundamentally rethinking our inference system architecture, shifting from compute-centric designs toward architectures optimized for bandwidth and efficient data movement. To achieve this, I developed bandwidth-centric approaches that treat data movement – not computation – as the primary design constraint. These approaches leverage key architectural ingredients, including chiplet-based designs to scale memory bandwidth through increased shoreline, and emerging memory technologies better aligned with the demands of AI inference. To fully utilize scalable memory bandwidth, I developed a dataflow architecture which decouples memory, compute, and network pipelines and enables low-latency inference. Building on these foundations, I designed systems across scales, from datacenter to edge, unified by a common objective: reducing latency per token by optimizing data movement rather than compute alone. These ideas have been validated through both research and real-world implementation. This work has led to a Best Paper award for the proposed Reasoning Processing Unit (RPU), a bandwidth-centric architecture for low-latency token generation in the datacenter. Beyond simulation, these principles have been realized in a taped-out edge-device, Pistil, where my custom instruction set and dataflow architecture were brought to silicon and used to execute a large language model end-to-end. Looking beyond single-model inference, this work further explores emerging “big-little” LLM workloads, showing how specialized memory hierarchies can extend bandwidth-centric principles to more complex, heterogeneous AI systems. Together, these results demonstrate that bandwidth-centric design is not only practical, but essential for enabling the next generation of AI systems, where the speed of token generation defines the speed of intelligence itself.