Large Language Models Can Learn Temporal Reasoning

Abstract: While LLMs have demonstrated remarkable reasoning capabilities, they are not without their flaws and inaccuracies. Recent studies have introduced various methods to mitigate these limitations. Temporal reasoning (TR), in particular, presents a significant challenge for LLMs due to its reliance on diverse temporal concepts and intricate temporal logic. In this paper, we propose TG-LLM, a novel framework towards language-based TR. Instead of reasoning over the original context, we adopt a latent representation, temporal graph (TG) that enhances the learning of TR. A synthetic dataset (TGQA), which is fully controllable and requires minimal supervision, is constructed for fine-tuning LLMs on this text-to-TG translation task. We confirmed in experiments that the capability of TG translation learned on our dataset can be transferred to other TR tasks and benchmarks. On top of that, we teach LLM to perform deliberate reasoning over the TGs via Chain-of-Thought (CoT) bootstrapping and graph data augmentation. We observed that those strategies, which maintain a balance between usefulness and diversity, bring more reliable CoTs and final results than the vanilla CoT distillation.

• The paper introduces TG-LLM, a framework that improves LLMs' temporal reasoning by translating text into structured temporal graphs.
• It employs chain-of-thought bootstrapping and graph data augmentation to enhance reasoning accuracy across benchmarks.
• Experimental results demonstrate superior token-level F1 scores and exact match rates, highlighting the framework’s effectiveness in TR tasks.

LLMs and Temporal Reasoning

Introduction

The paper "LLMs Can Learn Temporal Reasoning" (2401.06853) addresses the challenge of temporal reasoning (TR) in LLMs. While LLMs are renowned for their reasoning capabilities, temporal reasoning remains a formidable task due to the complexity of temporal logic and expressions required. This paper proposes the TG-LLM framework as a novel approach to enhance TR capabilities in LLMs by leveraging temporal graphs (TG) as latent representations, providing a comprehensive structure for reasoning tasks.

TG-LLM Framework

The TG-LLM framework comprises two primary steps: text-to-temporal graph translation and temporal graph reasoning. The synthesis of these components allows for efficient learning and application of temporal reasoning. The construction of a synthetic dataset, TGQA, serves as the basis for fine-tuning LLMs in translating text into TGs, enabling their capabilities to be generalized across various TR tasks.

Figure 1: Our TG-LLM framework has two steps: text-to-temporal graph translation and temporal graph reasoning.

Text-to-Temporal Graph Translation

The text-to-temporal graph translation is the cornerstone of TG-LLM, transforming narrative contexts into structured temporal graphs. The paper highlights the importance of high-quality TG generation, leveraging controlled datasets to ensure accurate translation aligned with temporal logic benchmarks. These TGs provide the foundation for subsequent reasoning tasks, addressing the common shortcomings found in conventional TR approaches where intrinsic temporal logic is often overlooked.

Temporal Graph Reasoning

Once translated into TG, LLMs engage in deliberate reasoning, facilitated by Chain of Thought (CoT) bootstrapping and graph data augmentation. CoT bootstrapping involves generating intermediate reasoning steps, ensuring the LLM's decision processes are both logical and accurate, further refined by contrastive learning scores.

Figure 2: In Chain of Thoughts (CoTs) bootstrapping, we accept CoTs leading to correct final answers and sample them according to their contrastive learning scores.

The graph data augmentation introduces strategic disturbances to improve robustness in TR tasks. These entail removing irrelevant edges and employing relation synonyms, alongside strategies to enhance the diversity and applicability of training data. This ensures LLMs can proficiently handle temporal reasoning without being hindered by traditional data insufficiencies typical in reasoning tasks.

Figure 3: We have several strategies for graph data augmentation: remove irrelevant edges, use relation synonyms and change entities/times.

Experimental Results

Experiments conducted demonstrate the efficacy of the TG-LLM framework across various TR benchmarks including TGQA, TimeQA, and TempReason. The results, which measure token-level F1 scores, exact match (EM) rates, and perplexity-based accuracy, indicate superior performance of TG-LLM relative to existing LLM-based strategies, particularly demonstrating enhancement in reasoning reliability through CoT bootstrapping and graph data augmentation techniques.

Figure 4: Performance comparison between different CoT generation strategies on TGQA.

Implications and Future Directions

The implications of this research extend both practically and theoretically. Practically, TG-LLM provides a scalable method for improving TR in LLMs, offering potential applications in fields requiring complex temporal reasoning such as planning and causal discovery. Theoretically, this framework opens avenues for integrating graph-based reasoning in LLMs, promoting a structured approach to reasoning tasks that surpasses traditional methods.

Future research could explore extending TG-LLM to more intricate reasoning forms, including inductive and abductive reasoning, given the robustness of graph-based methodologies. Moreover, the adaptability of TG-LLM suggests potential for broadening its application to diverse domains where temporal understanding plays a pivotal role.

Conclusion

In summary, the paper achieves a significant stride in enhancing temporal reasoning capabilities within LLMs via the TG-LLM framework. This approach, centered on text-to-graph translation followed by deliberate reasoning, brings forth a structured methodology facilitating advanced reasoning capabilities. Its success across varied benchmarks underscores the promise of integrating temporal graphs into LLMs. Future work may focus on expanding these methodologies to encompass wider reasoning types and applications, further solidifying the role of LLMs in complex problem-solving scenarios.

Please refer to the provided arXiv paper ID (2401.06853) for detailed insights and empirical findings presented in this essay.