From system models to class models: An in-context learning paradigm
Abstract: Is it possible to understand the intricacies of a dynamical system not solely from its input/output pattern, but also by observing the behavior of other systems within the same class? This central question drives the study presented in this paper. In response to this query, we introduce a novel paradigm for system identification, addressing two primary tasks: one-step-ahead prediction and multi-step simulation. Unlike conventional methods, we do not directly estimate a model for the specific system. Instead, we learn a meta model that represents a class of dynamical systems. This meta model is trained on a potentially infinite stream of synthetic data, generated by simulators whose settings are randomly extracted from a probability distribution. When provided with a context from a new system-specifically, an input/output sequence-the meta model implicitly discerns its dynamics, enabling predictions of its behavior. The proposed approach harnesses the power of Transformers, renowned for their \emph{in-context learning} capabilities. For one-step prediction, a GPT-like decoder-only architecture is utilized, whereas the simulation problem employs an encoder-decoder structure. Initial experimental results affirmatively answer our foundational question, opening doors to fresh research avenues in system identification.
- Deep convolutional networks in system identification. In IEEE 58th conference on decision and control (CDC), pages 3670–3676, 2019.
- Can transformers learn optimal filtering for unknown systems? arXiv preprint arXiv:2308.08536, 2023.
- Deep subspace encoders for nonlinear system identification. Automatica, 156, 2023.
- Least costly identification experiment for control. Automatica, 42(10):1651–1662, 2006.
- Meta-learning of neural state-space models using data from similar systems. In World Congress of the International Federation of Automatic Control (IFAC), July 2023.
- A survey for in-context learning. arXiv preprint arXiv:2301.00234, 2022.
- Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pages 1126–1135. PMLR, 2017.
- Continuous-time system identification with neural networks: Model structures and fitting criteria. European Journal of Control, 59:69–81, 2021.
- dynoNet: A neural network architecture for learning dynamical systems. International Journal of Adaptive Control and Signal Processing, 35(4):612–626, 2021.
- From system models to class models: An in-context learning paradigm. IEEE Control Systems Letters, X(Y):ZZ—WW, 2023.
- What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583–30598, 2022.
- Block-oriented nonlinear system identification, volume 1. Springer, 2010.
- Meta-learning probabilistic inference for prediction. In International Conference on Learning Representations, 2019.
- Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(9):5149–5169, 2021.
- Andrej Karpathy. nanoGPT. https://github.com/karpathy/nanoGPT, 2023. Accessed: June 6, 2023.
- General-purpose in-context learning by meta-learning transformers. arXiv preprint arXiv:2212.04458, 2022.
- Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
- LPV modelling and identification: An overview. Robust Control and Linear Parameter Varying Approaches: Application to Vehicle Dynamics, pages 3–24, 2013.
- Learning nonlinear state–space models using autoencoders. Automatica, 129, 2021.
- Identification of hybrid and linear parameter-varying models via piecewise affine regression using mixed integer programming. International Journal of Robust and Nonlinear Control, 30(15):5802–5819, 2020.
- Performance-oriented model learning for data-driven MPC design. IEEE control systems letters, 3(3):577–582, 2019.
- Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
- Jürgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta-… hook. PhD thesis, Technische Universität München, 1987.
- Subspace identification for linear systems: Theory—Implementation—Applications. Springer Science & Business Media, 2012.
- Attention is all you need. Advances in neural information processing systems, 30, 2017.
- A survey on curriculum learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):4555–4576, 2021.
- Calibrating building simulation models using multi-source datasets and meta-learned bayesian optimization. Energy and Buildings, 270:112278, 2022.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.