Quantization, Sparsity, Reliability, and Their Interactions

Abstract

In this thesis, we will explore and contribute to three seemingly disjoint areas of machine learning: quantization, sparsity, and reliability. While each of these areas are well-understood individually, there is very little existing literature on the interaction of these research areas. They are typically viewed as orthogonal to each other and solved independently. In this thesis, we would like to shed light on the fact that many practical systems employ elements from all three fields. Therefore, it is important to understand how they can influence and interact with each other. This thesis makes the following three key contributions to the existing literature. First, it presents GoldenEye, a functional simulator for modeling fault injections into machine learning models. In particular, it targets models that use novel number formats to quantize or emulate to different data types. The flexible API design makes it easy to add new number formats as the area evolves. Furthermore, we propose a mathematical framework for using reliability analysis to inform quantization. Second, we present our work on EdgeBERT, an accelerator for accelerating transformer inference. EdgeBERT features both quantization and sparsity optimizations that together achieve strong speedups when profiled on true silicon. We also write compiler code that allows other transformer workloads to be compiled to EdgeBERT. Finally, we attempt to understand the problem of sequential applications of quantization and sparsity. We prove a theoretical result that shows that at the tensor-level sparsity before quantization is preferred over quantization before sparsity. However, we show that at the model level, the order is not too important, so long as the algorithms are "synergetic". Then, we propose a novel quantization-aware sparsity algorithm that consider the problem of sparsity when the weights are already quantized. Together, we believe that these contributions provide valuable insights to understanding the interactions between these distinct problems.