FinTral: A Family of GPT-4 Level Multimodal Financial Large Language Models

Abstract: We introduce FinTral, a suite of state-of-the-art multimodal LLMs built upon the Mistral-7b model and tailored for financial analysis. FinTral integrates textual, numerical, tabular, and image data. We enhance FinTral with domain-specific pretraining, instruction fine-tuning, and RLAIF training by exploiting a large collection of textual and visual datasets we curate for this work. We also introduce an extensive benchmark featuring nine tasks and 25 datasets for evaluation, including hallucinations in the financial domain. Our FinTral model trained with direct preference optimization employing advanced Tools and Retrieval methods, dubbed FinTral-DPO-T&R, demonstrates an exceptional zero-shot performance. It outperforms ChatGPT-3.5 in all tasks and surpasses GPT-4 in five out of nine tasks, marking a significant advancement in AI-driven financial technology. We also demonstrate that FinTral has the potential to excel in real-time analysis and decision-making in diverse financial contexts. The GitHub repository for FinTral is available at \url{https://github.com/UBC-NLP/fintral}.

• The paper presents FinTral, a multimodal LLM suite that achieves GPT-4 level performance using domain-specific pretraining on 20B tokens.
• It employs advanced prompting, instruction tuning, and reduction of hallucinations to improve financial document analysis.
• Empirical evaluations on the FinSET benchmark show superior results compared to GPT-3.5 and GPT-4 in multiple financial tasks.

FinTral: A Family of GPT-4 Level Multimodal Financial LLMs

The paper "FinTral: A Family of GPT-4 Level Multimodal Financial LLMs," presents a suite of advanced multimodal LLMs developed specifically for financial applications. The FinTral models have been designed to outpace contemporary models like GPT-3.5 and GPT-4 across various tasks within the financial domain.

Introduction and Motivation

Financial document analysis and interpretation present significant challenges due to the complexity of the financial language and the necessity for intricate numerical processing. Traditional NLP models often falter in this domain, primarily due to the scarcity of annotated data and the need for real-time analysis capabilities. Responding to these challenges, the FinTral model family is constructed on the Mistral-7b base, extending GPT-4 level competencies to a specialized financial context. This integration aims to address the processing of text, numerical, tabular, and image data collectively.

Methodology

Model Architecture and Training

FinTral extends the Mistral-7b model with domain-specific pretraining that includes instruction fine-tuning and RLAIF training, incorporating a vast amount of curated textual and visual datasets. The pretraining phase utilizes a filtered 20 billion tokens dataset rich in financial data, ensuring the model's ability to comprehend and generate accurate financial narratives without hallucinations.

Figure 1: FinSET, a Financial Training and Evaluation Benchmark.

Prompting and Instruction Tuning

FinTral employs a sophisticated prompting method tailored for financial analysis, designed to elicit precise financial insights. This involves the use of a memetic proxy as a financial expert integrated with task-based questions and strategic retrieval of pertinent information, which aligns the model's focus with query requirements.

Figure 2: FinTral prompting method.

Evaluation and Benchmarking

FinTral's performance was evaluated against a comprehensive benchmark, FinSET, encompassing multiple financial tasks, including sentiment analysis, named entity recognition, and stock movement prediction. Notably, FinTral outperforms GPT-3.5 and surpasses GPT-4 in selected financial tasks, demonstrating superior zero-shot performance capabilities.

Figure 3: Comparative Performance Analysis on text-based tasks of Key Financial AI Models.

Addressing Hallucinations

A critical advancement in FinTral's deployment is the reduction of hallucinations, a common challenge faced by LLMs in finance. Through rigorous training on domain-specific data and AI feedback mechanisms, FinTral effectively minimizes erroneous interpretations and maintains high accuracy in financial tasks.

Figure 4: Performance comparison of various models on the FinanceBench dataset.

Practical and Theoretical Implications

The introduction of FinTral represents a significant step forward in aligning the capabilities of LLMs with the nuanced requirements of financial analysis and decision-making. The model's proficiency in handling diverse modalities and its enhanced real-time analytical capabilities suggest potential applications in live market analysis and financial advisement.

Figure 5: Human Evaluation on FinTerms Dataset.

Conclusion

The FinTral models showcase a substantial leap in the application of LLMs to financial data analysis. By leveraging a tailored training pipeline and employing direct policy optimization, FinTral manifests as a robust tool capable of advancing the current capabilities of financial AI technology. Future research may explore extending these methodologies to broader domains and integrating live data stream analysis to further enhance financial decision-making processes.