Resource Optimization of Emerging Machine Learning Workloads on Edge Devices

Abstract

The deployment of machine learning workloads is becoming increasingly prevalent, particularly on edge devices. However, emerging workloads such as Bayesian inference and Large Language Models (LLMs) present significant technical challenges in these resource-constrained environments. On edge devices, computation and memory are among the most critical resources for any workload. To optimize computational resources, this dissertation focuses on Bayesian inference as a case study. Unlike neural networks, the sampling kernels used in Bayesian inference often suffer from inefficient hardware architectures and lack scalable compute solutions. To address these challenges, we introduce FlexGibbs and CoopMC, which provide flexible architectures designed to overcome the computational bottlenecks and scalability issues inherent in Bayesian inference. For memory optimization, LLMs serve as a representative example due to their substantial size and memory demands, which are particularly problematic on edge devices. Model quantization has thus become a standard approach for deploying LLMs at the edge. However, existing research on quantized LLMs tends to prioritize accuracy and model size, often overlooking practical deployment considerations such as inference latency, deployment flexibility, and on-device fine-tuning. Our body of work addresses these overlooked aspects. In SpeedLimit, we propose a novel neural architecture search framework for quantized transformer models that enhances accuracy under fixed latency constraints. FlexQuant significantly improves the flexibility of LLM deployment on edge devices by increasing the diversity of available model sizes, reducing storage requirements, and supporting elastic hosting at runtime. To facilitate on-device fine-tuning, our FwC framework offers a practical solution for memory-constrained devices, mitigating performance degradation due to quantization. Beyond computation and memory optimization, we also explore performance prediction techniques to accelerate design exploration for edge workloads. The PerfSAGE framework delivers accurate predictions of key performance metrics—including latency, energy consumption, and memory usage—for any neural network architecture, outperforming existing methods in prediction accuracy. This provides researchers and engineers with a robust, generalizable methodology to expedite future design space exploration.