On Neural Linear Model Prediction, with Applications to Nonstationary Settings

Abstract

Neural Linear Models (NLMs) are deep Bayesian machine learning models that appear in a variety of contexts due to their data adaptivity and model flexibility, including many settings where Gaussian Processes (GPs) traditionally perform well. However, the possibility of utilizing NLMs for nonstationary data is not well studied, where nonstationary data generally refers to datasets in which some salient properties of the data vary with the input space. Our first contribution is that we benchmark the NLM against GPs on nonstationary data and demonstrate that the NLM is able to perform well in many settings. However, there remain areas for improvement in the NLM, especially in its predictive uncertainty. Our second contribution is our analysis of NLM predictions based on pruning basis functions, illustrating potential redundancies in certain NLMs with Leaky ReLU or ReLU-based activations in the last hidden layer. Finally, we propose novel methods with the potential to remedy problems of poor predictive uncertainty in NLMs by incorporating elements of a GP into the NLM.