An Improved Yau-Yau Algorithm for High Dimensional Nonlinear Filtering Problems

Abstract: Nonlinear state estimation under noisy observations is rapidly intractable as system dimension increases. We introduce an improved Yau-Yau filtering framework that breaks the curse of dimensionality and extends real-time nonlinear filtering to systems with up to thousands of state dimensions, achieving high-accuracy estimates in just a few seconds with rigorous theoretical error guarantees. This new approach integrates quasi-Monte Carlo low-discrepancy sampling, a novel offline-online update, high-order multi-scale kernel approximations, fully log-domain likelihood computation, and a local resampling-restart mechanism, all realized with CPU/GPU-parallel computation. Theoretical analysis guarantees local truncation error (O(\Delta t^2 + D^*(n))) and global error (O(\Delta t + D^*(n)/\Delta t)), where (\Delta t) is the time step and (D^*(n)) the star-discrepancy. Numerical experiments, spanning large-scale nonlinear cubic sensors up to 1000 dimensions, highly nonlinear small-scale problems, and linear Gaussian benchmarks, demonstrate sub-quadratic runtime scaling, sub-linear error growth, and excellent performance that surpasses the extended and unscented Kalman filters (EKF, UKF) and the particle filter (PF) under strong nonlinearity, while matching or exceeding the optimal Kalman-Bucy filter in linear regimes. By breaking the curse of dimensionality, our method enables accurate, real-time, high-dimensional nonlinear filtering, opening broad opportunities for applications in science and engineering.