Increasing Model Accuracy, System Efficiency and Data Security of Distributed Computing for Deep Neural Networks

Abstract

Deep Neural Networks (DNNs) have demonstrated efficacy in numerous applications, such as image recognition and natural language processing. Consequently, connected edge devices see increased demand for DNN computation while requiring to live with stringent constraints in hardware resources. Transferring raw input data from edge devices to cloud servers for cloud computing will introduce expensive communication overhead. Distributed computing, which executes a part of DNNs’ computing on edges and offloads the remaining to the cloud, emerges as a prevalent strategy in this context. The main focus of this thesis is to adapt the DNN architecture and training procedure to increase system security and efficiency as well as model accuracy in a distributed computing environment. First, we introduce SplitNets, that derives DNNs and network splittings to distribute the inference workload across multiple connected compute units. In order to optimize systems performance, we propose a split-aware neural architecture search framework that enable the co-design of the model, network splitting, and communication compression modules. In comparison to existing approaches, our proposed framework results in a substantial (10x) reduction in end-to-end latency without compromising task performance. We then examine the risk to data security in the context of distributed computing. Prior research has typically assumed that distributed computing offers data privacy benefits since private data is not transmitted from edge devices to the cloud; instead, only intermediary features extracted by early layers on edge devices are exchanged. In this work, we debunk this protection by presenting a novel data-free model inversion method and demonstrating sample inversion where private data from edge devices can still be leaked with high fidelity from the shared feature even after tens of neural network layers. Further, we show that our method can be used as a tool to analyze inversion vulnerability at different layer depths as well as the trade-off between utility and defense. Lastly, we discuss how to train DNNs jointly across edge devices using distributed data that cannot be shared with the server due to privacy protection and legal regulation reasons. A key challenge lies in handling of heterogeneous data across multiple devices that may induce disparities of their local features. We present the Hyperspherical Federated Learning (SphereFed) framework to allow use of distributed heterogeneous training data by constraining learned representations of data points to be on a unit hypersphere spanned by a fixed classifier shared by edge devices. After federated training in improving the global model, this classifier is further calibrated with a closed-form solution by minimizing a mean squared loss. Experiments indicate that our SphereFed approach is able to improve the accuracy of multiple existing federated learning algorithms by a considerable margin (up to 6% on challenging datasets) with enhanced computation and communication efficiency across datasets and model architectures.