Andrew Barron, Yale University

Rapid Bayesian Computation and Estimation for Neural Networks via Log-Concave Coupling
Date
Feb 18, 2025, 4:30 pm5:30 pm

Details

Event Description

We study single hidden-layer artificial neural networks with l_1 controls on the weights of the neurons.  These networks are known to provide accurate approximations to a flexible class of high-dimensional functions.  With K neurons and d input variables, the parameter dimension is of the order Kd. Suppose N observations are provided, with input coordinates bounded by 1. With a uniform prior on the constrained parameters, we show that the posterior distribution has a log-concave coupling, when Kd > N^2.  That is, we exhibit auxiliary variables such that their density is log-concave and the conditional density of the parameters given the auxiliary variables is also log-concave.  Accordingly, Markov chain Monte Carlo strategies for sampling these coupled variables are rapidly mixing, permitting accurate draws from the posterior in low-order polynomial time.  Moreover, we demonstrate, with a discrete uniform prior, that the posterior mean networks provide arbitrary sequence regret controls for squared error loss of order a fixed fractional power of [log d]/N.  Similar conclusions are obtained for mean square generalization error and for Kullback divergence of the predictive distribution in the case of independent observations. Thus accurate estimation is available even in extremely high dimensions, with moderately large sample sizes, and polynomial-time computation.  This is joint work with Curtis McDonald.

Event Category
ORFE Department Colloquia