Analysis and Generalization of Several Information Processing Methods Related to Stein’s Lemma

Abstract

Stein's lemma was proposed by the statistician Charles Stein as a beautiful and simple lemma about normal distributions. In spite of its simplicity, it has applications to many important methods of information processing and statistical inference, such as principal Hessian directions, Stein's unbiased risk estimator (SURE), the James-Stein estimator and empirical Bayes methods. In this dissertation, we study and generalize several information processing methods based on Stein's lemma. We begin by studying the spectral method, which was first proposed in the statistics literature as principal Hessian directions. We solve the optimal design problem of preprocessing functions which play a significant role in the performance of spectral methods. We show that under mild technical conditions, there exists a uniformly optimal preprocessing function for all sampling ratios. Next, we move to the analysis of the multi-rank spectral method for estimating a low-rank subspace. We focus on the regime where both the signal dimension and the number of measurements go to infinity with a fixed ratio and provide an asymptotic characterization of and performance of the method. Our analysis reveals a phase transition phenomenon where both the performance of spectral methods and the computational complexity are significantly different in the two phases. Finally, we consider another important application of Stein's lemma, namely, Stein's unbiased risk estimator (SURE). SURE was first derived for the Gaussian observation channel and later extended to distributions in the exponential family. We further generalize the SURE formalism to discrete observation models and propose approximate Stein's unbiased risk estimator (A-SURE). We prove the approximate consistency of the proposed A-SURE and apply it to improve the performance of binary imaging with quantum image sensors.