OpenCity: Open Spatio-Temporal Foundation Models for Traffic Prediction

Abstract: Accurate traffic forecasting is crucial for effective urban planning and transportation management, enabling efficient resource allocation and enhanced travel experiences. However, existing models often face limitations in generalization, struggling with zero-shot prediction on unseen regions and cities, as well as diminished long-term accuracy. This is primarily due to the inherent challenges in handling the spatial and temporal heterogeneity of traffic data, coupled with the significant distribution shift across time and space. In this work, we aim to unlock new possibilities for building versatile, resilient and adaptive spatio-temporal foundation models for traffic prediction. To achieve this goal, we introduce a novel foundation model, named OpenCity, that can effectively capture and normalize the underlying spatio-temporal patterns from diverse data characteristics, facilitating zero-shot generalization across diverse urban environments. OpenCity integrates the Transformer architecture with graph neural networks to model the complex spatio-temporal dependencies in traffic data. By pre-training OpenCity on large-scale, heterogeneous traffic datasets, we enable the model to learn rich, generalizable representations that can be seamlessly applied to a wide range of traffic forecasting scenarios. Experimental results demonstrate that OpenCity exhibits exceptional zero-shot predictive performance. Moreover, OpenCity showcases promising scaling laws, suggesting the potential for developing a truly one-for-all traffic prediction solution that can adapt to new urban contexts with minimal overhead. We made our proposed OpenCity model open-source and it is available at the following link: https://github.com/HKUDS/OpenCity.

• The paper presents OpenCity, a foundation model leveraging Transformers and graph neural networks to tackle zero-shot spatio-temporal traffic prediction challenges.
• It pre-trains on large-scale, heterogeneous traffic datasets, enabling robust and generalizable representations across varying urban contexts.
• Experimental results demonstrate that OpenCity outperforms state-of-the-art methods, offering an efficient, cost-effective solution for urban traffic management.

Overview of the Paper: OpenCity – Open Spatio-Temporal Foundation Models for Traffic Prediction

The paper "OpenCity: Open Spatio-Temporal Foundation Models for Traffic Prediction" introduces a novel foundation model, OpenCity, designed to address the intricacies of traffic forecasting. The motivation behind this research stems from the pressing need to improve urban planning and transportation management through accurate traffic prediction. Conventional models often falter when faced with zero-shot prediction challenges on unknown regions or cities and exhibit limited generalization over extended periods. Such difficulties arise due to the inherent spatial and temporal heterogeneity in traffic data, as well as distribution shifts across different contexts.

Methodology

OpenCity endeavors to redefine traffic prediction by harnessing the combined strengths of Transformers and graph neural networks for modeling intricate spatio-temporal dependencies. This model is pre-trained on large-scale, heterogeneous traffic datasets, enabling it to learn robust, generalizable representations. Consequently, OpenCity demonstrates adept zero-shot predictive performance across diverse urban environments. It showcases an innovative approach by creating a versatile, resilient, and adaptive framework for traffic prediction.

Key components of OpenCity include:

• Zero-Shot Spatio-Temporal Embedding Layer: This involves instance normalization and a patch embedding strategy to accommodate variations across regions and efficiently manage prediction tasks over longer time horizons.
• Spatio-Temporal Context Encoder: It captures essential temporal features, such as the time of day and day of the week, alongside spatial contexts embedded using region-specific eigenvectors.
• Spatio-Temporal Dependencies Modeling: By employing a novel TimeShift Transformer architecture, the model captures both periodic and dynamic traffic patterns. Graph convolutional networks further elucidate spatial dependencies, enhancing predictive accuracy.

Experimental Evaluation

The effectiveness of OpenCity was tested against multiple large-scale datasets from the US and China. The researchers examined various scenarios like cross-regional zero-shot evaluation, cross-city zero-shot evaluation, and long-term forecasts, demonstrating OpenCity's capacity to generalize across distinct datasets without additional fine-tuning. Notably, the model outperformed various state-of-the-art methods in several metrics, even matching full-shot scenarios where baselines require training on target datasets.

The results articulate OpenCity's potential in providing competitive and reliable predictions for complex traffic forecasting tasks, showcasing its adaptability to diverse spatial and temporal patterns inherent in traffic systems.

Implications and Future Directions

The OpenCity model bears significant practical implications for urban planners and transportation agencies. Its ability to deliver accurate predictions without extensive training or sensor deployment reduces both time and costs associated with model deployment and maintenance. The seamless adaptability across variable urban contexts suggests its utility in developing unified traffic management systems on a global scale.

Theoretically, OpenCity advances the discourse on foundation models by offering insights into their scaling capabilities for spatio-temporal tasks and the essence of learning universal representations. Future research could explore refining model efficiency, especially for real-time deployment settings, and exploring similar foundation model applications in other large-scale spatio-temporal datasets beyond traffic prediction.

In closing, OpenCity represents a commendable stride toward developing scalable, efficient, and generalizable models, aligning closely with the overarching objectives of mitigating the challenges faced by urban traffic systems worldwide.