Chronos: Learning the Language of Time Series

Abstract: We introduce Chronos, a simple yet effective framework for pretrained probabilistic time series models. Chronos tokenizes time series values using scaling and quantization into a fixed vocabulary and trains existing transformer-based LLM architectures on these tokenized time series via the cross-entropy loss. We pretrained Chronos models based on the T5 family (ranging from 20M to 710M parameters) on a large collection of publicly available datasets, complemented by a synthetic dataset that we generated via Gaussian processes to improve generalization. In a comprehensive benchmark consisting of 42 datasets, and comprising both classical local models and deep learning methods, we show that Chronos models: (a) significantly outperform other methods on datasets that were part of the training corpus; and (b) have comparable and occasionally superior zero-shot performance on new datasets, relative to methods that were trained specifically on them. Our results demonstrate that Chronos models can leverage time series data from diverse domains to improve zero-shot accuracy on unseen forecasting tasks, positioning pretrained models as a viable tool to greatly simplify forecasting pipelines.

• The paper introduces Chronos, which tokenizes time series data through scaling and quantization to adapt transformer models for forecasting.
• It employs synthetic augmentation techniques, including TSMixup and KernelSynth, to address data scarcity and enhance model generalization.
• The model demonstrates strong in-domain and zero-shot performance using metrics like MASE and WQL, highlighting its competitive edge over traditional methods.

Introduction to Chronos

Chronos is introduced as an innovative framework designed to leverage transformer-based LLMs for the task of probabilistic time series forecasting. By tokenizing the values in a time series through scaling and quantization, Chronos converts these continuous values into discrete tokens that can be processed by pre-existing LLM architectures. The transformer models are subsequently trained using a cross-entropy loss, capitalizing on this novel tokenization to efficiently handle the time series data. The unique approach of Chronos involves augmenting the training data with synthetic datasets generated from Gaussian processes to further boost generalization.

Figure 1: High-level depiction of Chronos.

Tokenization and Model Training

Chronos employs a two-step process for tokenizing time series data: scaling, followed by quantization. Each time series is first scaled to reduce its variance, which is crucial for handling diverse datasets with varying scales. The scaled values are then quantized into a predefined number of bins, representing the time series as a sequence of tokens.

These tokens, treated as language tokens, are fed into a transformer model that can be either an encoder-decoder or decoder-only architecture. The model is trained using a categorical cross-entropy loss between the predicted token distribution and the ground truth, allowing it to learn the sequential structure of the time series data effectively. Notably, this method requires no alterations to the model architecture beyond adjusting the vocabulary size to match the number of quantization bins.

Handling Data Scarcity with Synthetic Augmentation

A significant challenge in utilizing LLMs for time series forecasting is the limited availability of diverse and extensive time series datasets. Chronos addresses this by incorporating two key forms of data augmentation:

1. TSMixup: This approach generates new time series by forming convex combinations of multiple existing time series.
2. KernelSynth: Synthetic time series are generated using Gaussian processes with kernels sampled from a predefined bank, allowing for customization of variability and structural complexity through compositional rules.

Figure 2: (a) Illustration of KernelSynth synthetic time series generation. (b) Example synthetic time series.

Performance Evaluation

The performance of Chronos is evaluated across 42 different datasets, which are categorized into two benchmarks: Benchmark I (in-domain) and Benchmark II (zero-shot). Chronos models demonstrate strong performance in terms of point forecasting (using MASE) and probabilistic forecasting (using WQL) on both benchmarks. Importantly, even without task-specific fine-tuning, Chronos models exhibit superior zero-shot forecasting capabilities compared to traditional local models and are competitive with task-specific deep learning models.

Figure 3: Model performance on in-domain datasets.

Analysis of Model Parameters

The efficacy of Chronos is further exemplified through the detailed analysis of model parameters, such as model size and initial pretraining weights. Larger Chronos models exhibit improved performance, affirming the potential of scaling the model size to enhance forecasting accuracy. However, initializations from pre-trained LLM weights did not yield substantial improvements, indicating that random initializations might be preferable for time series tasks.

Qualitative Insights and Limitations

Despite its effectiveness, Chronos faces challenges when predicting time series with strong trends or sparse data due to the intrinsic limitations of tokenization within a finite range. Performance can also degrade when high-frequency data severely taxes the transformer’s context length.

Figure 4: In-domain versus zero-shot performance of varied Chronos configurations.

Conclusion

Chronos presents a powerful methodology for leveraging LLMs in time series forecasting by innovatively reimagining the tokenization process. By employing synthetic data for augmentation, it effectively navigates data scarcity issues, positioning itself as a competitive alternative to traditional time series forecasting paradigms.

Chronos' performance, particularly in zero-shot scenarios, implies significant potential for simplifying and enhancing the deployment of forecasting models across diverse applications. Further, its compatibility with scalable LLM architectures suggests promising avenues for broader application beyond univariate forecasting.