Statistics and Probability seminar
Speaker: Dr. Abhinek Shukla, Centre for Biomedical Data Science, Duke-NUS Medical School
Host: Koushik Saha
Title: Improving Inference in Stochastic Gradient Descent Based on Equal Batch-Size Batch-Means Estimator
Time, day and date: 12:00:00 PM - 1:00:00 PM, Wednesday, March 18
Venue: Online (https://meet.google.com/wjs-deag-ysh)
Abstract: Stochastic gradient descent (SGD) is a widely used technique for solving various optimization problems ranging from regression on a large dataset employed in statistics, to training deep neural networks for high dimensional models in machine learning. Performing tasks such as inference in SGD is a challenging problem due to its time-inhomogeneous Markovian nature. An underlying asymptotic normality of the averaged SGD (ASGD) estimator allows for the construction of a batch-means estimator of the asymptotic covariance matrix. In contrast to the existing increasing batch-size (IBS) strategy proposed for reducing correlation between far-apart batches, we propose a memory efficient equal batch-size (EBS) estimator and show consistency of the proposed batch-means estimator under mild conditions. The proposed EBS technique offers bias-correction of the variance at no additional cost to memory and is also shown to outperform the IBS estimator in extensive simulations. Further, since joint inference for high dimensional problems may be difficult, we present marginal-friendly simultaneous confidence intervals, and demonstrate improved predictions based on the proposed covariance estimators of ASGD.