Variational inference in high-dimensional Bayesian regression models

Abstract

In modern applications of Bayesian Statistics, the posterior distribution is typically high-dimensional and analytically intractable. Variational Inference (VI) has emerged as an attractive option to approximate these intractable distributions, facilitating fast, parallel computations. The simplest version of VI is the Naive Mean-field approximation (NMF), where the distribution of interest is approximated by a product distribution. In recent years, another strategy rooted in statistical physics, the Thouless-Anderson-Palmer (TAP) formulation, started to garner increasing attention from theorists and practitioners alike. However, despite the rapidly growing popularity of variational approximations in Statistics and Machine Learning, the corresponding theoretical guarantees for these approximations remain largely unexplored. This dissertation addresses this challenge through three main contributions:

TAP Approximation in Bayesian Linear Regression: In Chapter 1, a variational representation for the log-normalizing constant of the posterior distribution in Bayesian linear regression is derived in the proportional asymptotic regime, assuming a uniform spherical prior and i.i.d. Gaussian designs.

Performance of NMF in Linear Regression: Chapter 2 investigates the NMF approximation in linear regression under proportional asymptotics. It confirms the inaccuracy of NMF for approximating the log-normalizing constant and supports empirical observations of NMF being overconfident.

NMF in Generalized Linear Models (GLMs): Chapter 3 identifies conditions under which the NMF approximation is valid in high-dimensional GLMs. Algorithmic insights and probabilistic properties of the high-dimensional posteriors were also investigated.