Can a Transformer Represent a Kalman Filter?

Abstract: Transformers are a class of autoregressive deep learning architectures which have recently achieved state-of-the-art performance in various vision, language, and robotics tasks. We revisit the problem of Kalman Filtering in linear dynamical systems and show that Transformers can approximate the Kalman Filter in a strong sense. Specifically, for any observable LTI system we construct an explicit causally-masked Transformer which implements the Kalman Filter, up to a small additive error which is bounded uniformly in time; we call our construction the Transformer Filter. Our construction is based on a two-step reduction. We first show that a softmax self-attention block can exactly represent a Nadaraya-Watson kernel smoothing estimator with a Gaussian kernel. We then show that this estimator closely approximates the Kalman Filter. We also investigate how the Transformer Filter can be used for measurement-feedback control and prove that the resulting nonlinear controllers closely approximate the performance of standard optimal control policies such as the LQG controller.

• The paper demonstrates that a specially designed Transformer Filter can approximate the Kalman Filter within an epsilon range.
• It employs a two-phase approach where softmax self-attention first replicates Gaussian kernel smoothing before closely approximating Kalman filtering.
• The study extends the method to measurement-feedback control, showing the Transformer Filter mimics LQG controller dynamics with minimal error.

Introduction

Transformers, initially popularized through their success in natural language processing, have demonstrated remarkable flexibility by delivering state-of-the-art performances in various fields, such as computer vision and robotics. A fundamental question addressed in this paper is whether Transformers can perform Kalman Filtering, a pivotal process in optimal control and signal processing. The investigation of whether the nonlinear nature of Transformers harmonizes with the linear dynamics of a Kalman Filter is of considerable interest within the sphere of deep learning theory.

Transformer Filter Construction

A principal contribution of this research is the design of a specific Transformer, hereinafter referred to as the "Transformer Filter." This architecture demonstrates it can implement the Kalman Filter approximately. Employing a causally-masked structure, it eschews positional embeddings, which means the ordering of historical data does not influence the current state estimation. This is achieved in two phases. Initially, it is established that softmax self-attention can represent a Gaussian kernel smoothing estimator precisely; then, it is demonstrated that this estimator closely approximates the Kalman filter. The Transformer Filter's estimates are shown to be within an epsilon range of the Kalman Filter's estimates, indicating a strong approximation capability.

Control Application

Beyond filtering, the Transformer Filter's applicability extends to measurement-feedback control systems. A particular focus of the study is on analyzing the closed-loop dynamics induced by the Transformer Filter in such systems. An established outcome is that the Transformer Filter can approximate an LQG (Linear-Quadratic-Gaussian) controller to within an epsilon range. This approximation holds for the state sequences generated by both the control system and the LQG controller, reinforcing the potential of Transformers for use in control systems and their capability to stabilize the system within a specified error margin.

Analysis and Implications

On a broader note, the paper's findings suggest that while Transformers are intrinsically nonlinear systems, they can be tailored to approximate key linear filtering and control mechanisms closely. This paves the way for further exploration of Transformers within domains that were previously dominated by linear dynamic models. The results also characterize the conditions under which such an approximation holds in terms of the underlying parameters of the system, illustrating the depth of interplay between the complexity of Transformers and the precision of classical control models.