Talk by Sean Plummer (Texas A&M University)

Title: Statistical and Computational Guarantees for Variational Inference

Abstract: Over the last few years there has been an explosion of research pertaining to the statistical accuracy of the estimators that arise from mean-field variational inference. In contrast, the properties of the coordinate ascent algorithm, known as CAVI, used to compute the optimal mean-field variational approximation remain relatively mysterious. The convergence properties of CAVI are only known for specific models, where the proceeding analysis is heavily dependent on the model's structure. This is in part due to the non-convexity of the optimization objective which prevents the use of more familiar optimization based approaches to studying convergence. Furthermore, there is little known about the statistical accuracy of estimators that arise from more expressive variational families such as implicit variational inference, auto-encoders, and normalizing flows.

This talk is an overview of my recent work on developing statistical and computational properties for variational inference. The first part of the talk is focused on addressing questions related to the computational properties of the CAVI algorithm in mean-field variational inference. Using the Ising model in dimension 2 as a simple illustrative example, we demonstrate an alternative approach to studying the convergence behavior of CAVI based on dynamical systems. This approach allows us to fully classify the convergence behavior of both the sequential and parallel implementations of the Ising CAVI algorithm in dimension 2 for the entire parameter regime. We further illustrate the usefulness of the dynamical systems approach by showing the evidence lower bound correctly recovers the leading order of the log-marginal likelihood for singular models in dimension 2. The last part of the talk focuses on developing risk bounds and approximation quality of estimators arising from a Gaussian process based implicit variational inference framework deemed as GP-IVI.