When Does Multimodality Lead to Better Time Series Forecasting?

Abstract: Recently, there has been growing interest in incorporating textual information into foundation models for time series forecasting. However, it remains unclear whether and under what conditions such multimodal integration consistently yields gains. We systematically investigate these questions across a diverse benchmark of 16 forecasting tasks spanning 7 domains, including health, environment, and economics. We evaluate two popular multimodal forecasting paradigms: aligning-based methods, which align time series and text representations; and prompting-based methods, which directly prompt LLMs for forecasting. Our findings reveal that the benefits of multimodality are highly condition-dependent. While we confirm reported gains in some settings, these improvements are not universal across datasets or models. To move beyond empirical observations, we disentangle the effects of model architectural properties and data characteristics, drawing data-agnostic insights that generalize across domains. Our findings highlight that on the modeling side, incorporating text information is most helpful given (1) high-capacity text models, (2) comparatively weaker time series models, and (3) appropriate aligning strategies. On the data side, performance gains are more likely when (4) sufficient training data is available and (5) the text offers complementary predictive signal beyond what is already captured from the time series alone. Our study offers a rigorous, quantitative foundation for understanding when multimodality can be expected to aid forecasting tasks, and reveals that its benefits are neither universal nor always aligned with intuition.

• The paper demonstrates that leveraging auxiliary text improves forecasting when models pair high-capacity text encoders with effective alignment strategies.
• It compares aligning-based methods, which integrate representations through concatenation or residual projection, against prompting-based techniques to highlight context-dependent benefits.
• Empirical results reveal that MMTS models excel when text adds unique predictive signals and sufficient training data is available, although strong unimodal baselines still often prevail.

Introduction to Multimodality in Time Series Forecasting

The paper "When Does Multimodality Lead to Better Time Series Forecasting?" investigates the conditions under which integrating auxiliary textual information enhances time series forecasting. It examines two approaches: aligning-based methods, which integrate time series and text representations, and prompting-based methods, which leverage LLMs to make predictions. The study covers 16 datasets across various domains, revealing key modeling and data conditions for effective multimodal integration.

Criteria for Successful Multimodal Integration

The study identifies several factors critical to the success of multimodal time series (MMTS) models. These findings include:

• High-Capacity Text Models: Integrating larger text models can significantly improve the performance of multimodal models, particularly when the time series models are weaker.
• Appropriate Alignment Strategies: The choice of alignment, such as concatenation or residual projection mechanisms, impacts performance. Additive fusion with average pooling generally outperforms others.
• Sufficient Training Data: The availability of ample training instances is crucial for the effective learning of multimodal representations.
• Text Augmentation with Novel Signals: Gains from text incorporation are most substantial when text data provides additional, complementing predictive signals not apparent from the time series data alone.

  Figure 1: An overview of findings highlighting the conditions for effective MMTS.

Testing the Effectiveness of Integration Methods

The paper provides an in-depth comparative analysis of several modeling approaches. A significant observation is the context-dependent efficacy of MMTS. Among the datasets tested, aligning-based methods generally outperform prompting-based techniques, indicating the limitation of current LLMs in handling numerical reasoning and temporal patterns without specially designed mechanisms.

Figure 2: Illustration of unimodal forecasting compared to multimodal methods: aligning-based and prompting-based, the former aligning representations and the latter using direct prompting.

Another critical insight is that incrementally increasing the capacity of the text encoder helps, but only to an extent, as aligning mechanisms that preserve interactions with time series inputs are equally important.

Empirical Performance and Data Characteristics

The findings reveal that MMTS models fail to consistently outperform strong unimodal baselines. The paper systematically dissects MMTS performance across datasets and identifies the following:

• Modeling Complexity: Incorporating textual data with misaligned temporal characteristics or redundancy does not guarantee improved forecasting accuracy.
• Trade-Offs in Text Encoder Capacity: A larger text model can improve capability, but other factors like alignment mechanism play crucial roles.
• Dataset Properties: The study highlights that optimizing MMTS models for datasets where text adds unique, complementary information tends to result in the greatest improvements.

Conclusion

This paper makes significant contributions to understanding when and how multimodal approaches benefit time series forecasting. The extensive benchmarks provide a clear picture that multimodal integrations must be strategically designed based on model capability and data characteristics. Future research should consider exploring broader multimodal datasets, including visual data, and further refining alignment strategies to leverage additional signal modalities effectively.

These insights are invaluable for researchers and practitioners aiming to deploy MMTS models in real-world settings, suggesting that while multimodal integration has potential, its application must be carefully managed to truly enhance forecasting tasks.