How Far Do Time Series Foundation Models Paint the Landscape of Real-World Benchmarks ?

Abstract: Recent evaluations of time-series foundation models (TSFMs) have emphasized synthetic benchmarks, leaving real-world generalization less thoroughly examined. This work proposes a novel benchmarking approach that bridges synthetic and realistic data by extracting temporal signals from real-world video using optical flow and curating datasets reflecting everyday temporal dynamics. Building upon this pipeline, we introduce REAL-V-TSFM, a novel dataset designed to capture rich and diverse time series derived from real-world videos. Experimental results on three state-of-the-art of TSFMs under zero-shot forecasting shows that, despite strong performance on conventional benchmarks, these models predominantly exhibit performance degradation on the proposed dataset, indicating limited generalizability in these foundation models. These findings highlight the urgent need for data-centric benchmarking and diverse model structure to advance TSFMs toward genuine universality, while further validating the effectiveness of our video-based time series data extraction pipeline.