Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking

Abstract: LLMs have demonstrated notable capabilities across financial tasks, including financial report summarization, earnings call transcript analysis, and asset classification. However, their real-world effectiveness in managing complex fund investment remains inadequately assessed. A fundamental limitation of existing benchmarks for evaluating LLM-driven trading strategies is their reliance on historical back-testing, inadvertently enabling LLMs to "time travel"-leveraging future information embedded in their training corpora, thus resulting in possible information leakage and overly optimistic performance estimates. To address this issue, we introduce DeepFund, a live fund benchmark tool designed to rigorously evaluate LLM in real-time market conditions. Utilizing a multi-agent architecture, DeepFund connects directly with real-time stock market data-specifically data published after each model pretraining cutoff-to ensure fair and leakage-free evaluations. Empirical tests on nine flagship LLMs from leading global institutions across multiple investment dimensions-including ticker-level analysis, investment decision-making, portfolio management, and risk control-reveal significant practical challenges. Notably, even cutting-edge models such as DeepSeek-V3 and Claude-3.7-Sonnet incur net trading losses within DeepFund real-time evaluation environment, underscoring the present limitations of LLMs for active fund management. Our code is available at https://github.com/HKUSTDial/DeepFund.

Evaluation of DeepFund for Real-time Fund Investment Benchmarking with LLMs

The paper titled "Time Travel is Cheating: Going Live with DeepFund for Real-Time Fund Investment Benchmarking" introduces DeepFund, a benchmark tool specifically designed to evaluate the effectiveness of Large Language Models (LLMs) in real-time fund investment scenarios. This work seeks to address significant limitations present in existing benchmarks, which primarily utilize historical back-testing and are susceptible to information leakage, consequently yielding artificially optimistic results.

In the context of financial applications, LLMs have shown capability across various operations, including financial report summarization and asset classification. However, the authors identify a gap in evaluating these models against real-world fund management challenges. Empirical testing employing DeepFund reveals that even advanced models like DeepSeek-V3 and Claude-3.7-Sonnet experience net trading losses under live market conditions. This finding emphasizes the challenges still faced by LLMs in active fund management.

DeepFund utilizes real-time market data to model realistic investment environments, explicitly preventing information leakage by interacting only with data produced post each model's pretraining cutoff. The methodology involves a multi-agent architecture composed of specialized roles that mimic investor behaviors within an operational ecosystem. Key components include strategic decision-making by financial planners, detailed analyses by a spectrum of analysts (such as Fundamental, Technical, and Policy), and real-time investment decisions by the portfolio manager.

Quantitative results from live market analysis highlight the complexity of real-time market prediction by LLMs. In detailed comparisons, Grok demonstrated the most significant potential within DeepFund's structure, achieving a positive cumulative return, while several other flagship models incurred losses. Evaluations encompass a breadth of financial metrics, with Grok emerging with notably higher adaptability and resilience under unpredictable market conditions compared to its counterparts.

The implications of this research are substantial for both the practical application of LLMs in financial sectors and the theoretical development of AI models. By showcasing the limitations and active performance of LLMs in realistic investment scenarios, the study urges future research towards more sophisticated and reliable AI tools for fund management. These insights can drive the development and training of more adaptive and context-aware models that may eventually align with financial industry standards.

From a methodological standpoint, the shift from static, historical data benchmarks to dynamic benchmarking represents a critical evolution in evaluating AI competency, not only within finance but potentially extending to other industries reliant on real-time data interactions. By open sourcing their code, the authors of DeepFund underscore a commitment to reproducibility and collaborative improvement, inviting the broader research community to engage with and extend upon this pivotal work. This aspect ensures continuous evolution and adaptation in benchmarking methodologies as AI models grow in complexity and variety.

Overall, DeepFund sets a benchmark framework that is acutely relevant in modern AI and financial contexts. Its ability to map LLMs' real-time decision-making capabilities against actual market conditions encourages continued exploration and refinement, essential for AI's future role in complex financial ecosystems.