Misspecification, Nonstationarity, and Approximate Inference in Gaussian Processes and Bayesian Neural Networks

Abstract

In many fields there is a need to estimate nonlinear functions and provide reliable uncertainty quantification. Gaussian processes (GPs) enable easy specification of prior assumptions and rigorous guarantees of posterior properties, but scale poorly with the number of observations. On the other hand, neural networks have achieved remarkable empirical success on large, high-dimensional datasets but lack dependable uncertainty estimation. Although Bayesian neural networks (BNNs) provide a conceptually elegant solution, it is fundamentally difficult to aggregate priors on large quantities of parameters into collectively meaningful properties of the model they define. This dissertation presents GPs and BNNs under a common framework, highlighting similarities and exposing failures of each. Chapter 1 focuses on the consequences of misspecifying a kernel, the key choice in a GP model, and illustrates how a BNN can overcome this misspecification. Next, chapter 2 explains how to encode a common property of a GP kernel (stationarity) in a BNN. Finally, chapter 3 proves a critical failure of overparameterized BNNs under approximate inference.