Uncertainty Quantification using Simulation Output: Batching as an Inferential Device

Abstract: We present batching as an omnibus device for uncertainty quantification using simulation output. We consider the classical context of a simulationist performing uncertainty quantification on an estimator $\theta_n$ (of an unknown fixed quantity $\theta$) using only the output data $(Y_1,Y_2,\ldots,Y_n)$ gathered from a simulation. By uncertainty quantification, we mean approximating the sampling distribution of the error $\theta_n-\theta$ toward: (A) estimating an assessment functional $\psi$, e.g., bias, variance, or quantile; or (B) constructing a $(1-\alpha)$-confidence region on $\theta$. We argue that batching is a remarkably simple and effective device for this purpose, and is especially suited for handling dependent output data such as what one frequently encounters in simulation contexts. We demonstrate that if the number of batches and the extent of their overlap are chosen appropriately, batching retains bootstrap's attractive theoretical properties of strong consistency and higher-order accuracy. For constructing confidence regions, we characterize two limiting distributions associated with a Studentized statistic. Our extensive numerical experience confirms theoretical insight, especially about the effects of batch size and batch overlap.