Computation-Cautious Machine Learning Systems

Abstract

Deriving knowledge from data is central to how we live, learn, and decide: Machine learning and data science pipelines are extensively applied to extract knowledge from an ever-increasing amount of data across all fields, including high-energy physics, astronomy, and genetics. These pipelines consist of multiple stages from data exploration to model design, training, and deployment. Different stages have their own set of algorithms and techniques, yet they share a common challenge -- they involve repeated computation on huge data sets. This bottleneck slows down machine learning pipelines, which is problematic not only for latency-sensitive applications (such as self-driving cars and medical diagnosis), but as a result of this bottleneck, only a fraction of the generated data can be processed leading to lower quality models, fewer decisions per time unit, and overall, limited applicability of machine learning.

We introduce Computation-Cautious Machine Learning Systems -- Data Canopy, Deep Collider, and MotherNets -- that address the bottleneck of repeated computation and data movement across four critical stages of machine learning pipelines: (i) data exploration, (ii) model design, (iii) model training, and (iv) model deployment. During data exploration, Data Canopy enables reuse of computation and data movement across different statistical queries leading to several orders of magnitude (10x to 100x) improvement in the speed of data exploration and machine learning algorithms. Deep Collider reconsiders conventional model design wisdom and enables drastically better model design by balancing simultaneously accuracy, training time, deployment time, and memory resources. Finally, MotherNets enables fast and accurate training and deployment of ensembles of deep neural networks (2 to 3 percent reduced absolute test error rate and up to 35 percent faster training as compared to state-of-the-art approaches). MotherNets also establishes a new and navigable Pareto frontier for the accuracy-training cost tradeoff of deep neural network ensembles.