FLAG-Trader: LLM-RL Fusion for Trading

• FLAG-Trader is a unified architecture that fuses LLM linguistic reasoning with reinforcement learning to enable goal-oriented financial decision-making.
• It employs parameter-efficient fine-tuning by freezing 90% of the model and updating only top layers and task-specific heads to optimize trading strategies.
• Empirical evaluations show that a compact FLAG-Trader model outperforms larger baselines by achieving superior Sharpe ratios and cumulative returns.

FLAG-Trader is a unified architecture designed to integrate the linguistic processing and reasoning capabilities of LLMs with gradient-based reinforcement learning (RL) policy optimization for financial trading. The approach addresses the limitations of LLMs in complex, sequential, goal-oriented financial environments by enabling a partially fine-tuned LLM to act as the trading policy network. This design leverages pre-trained linguistic knowledge while achieving financial-domain adaptation through parameter-efficient techniques. FLAG-Trader introduces an RL objective with trading-specific rewards—most notably, increment in Sharpe ratio—and demonstrates competitive empirical results with a relatively small model size, outperforming significantly larger baselines in algorithmic trading settings (Xiong et al., 17 Feb 2025).

1. Architecture and Multimodal Fusion

FLAG-Trader employs a pre-trained LLM whose parameter set is divided into a frozen portion (θ_frozen), comprising the lower N transformer layers, and a trainable portion (θ_train), consisting of the top M layers. On top of these, two small feed-forward heads are added: a policy head (θ_P) and a value head (θ_V). The Policy_Net and Value_Net share θ_train but each maintains its own head for respective outputs.

At each decision time step $t$, the market state $s_t$—composed of numerical prices, technical indicators, sentiment features, and account balance—is formatted into a structured text prompt via a fixed template, denoted as $\text{lang}(s_t)$. This prompt is tokenized and embedded by the frozen LLM layers, producing hidden representations $h^{(1)}$, which are further processed by the trainable layers to produce $h^{(2)}$. The final outputs are:

• $\pi_\theta(a_t \mid s_t)$: the policy distribution over actions, generated via a softmax policy head (with action masking as appropriate).
• $V_\theta(s_t)$: state-value estimation, from the value head.

The agent interacts with the environment by sampling actions from $\pi_\theta$, receiving rewards, and triggering RL-based updates.

2. Mathematical Formalization

The trading environment is formalized as a Markov decision process (MDP) with the following elements:

• State space: $S = X \times Y$, where $s_t = (m_t, b_t)$ combines market observations $s_t$0 (price vector $s_t$1 and indicators $s_t$2) and account features $s_t$3 (cash $s_t$4, holdings $s_t$5).
• Action space: $s_t$6.
• Transitions: Portfolio transitions update cash and holdings based on the action.
• Reward: Defined as the increment in Sharpe ratio,

$s_t$7

$s_t$8

The optimization seeks $s_t$9 maximizing expected discounted returns: $\text{lang}(s_t)$0 with the policy gradient update: $\text{lang}(s_t)$1

Training employs Proximal Policy Optimization (PPO) with value and entropy regularization: $\text{lang}(s_t)$2

$\text{lang}(s_t)$3

$\text{lang}(s_t)$4

The total loss: $\text{lang}(s_t)$5 An optional cross-entropy language-modeling loss may be included for stability.

3. Parameter-Efficient Fine-Tuning

FLAG-Trader adopts a LoRA-style parameter-efficient fine-tuning strategy in which approximately 90% of the LLM parameters (θ_frozen) are kept fixed. Only the top M transformer layers (θ_train) and the two policy/value heads (θ_P, θ_V)—altogether about 5–10% of the full model—are updated during RL-driven training. The main training objective is the PPO RL loss; a small ($\text{lang}(s_t)$6) language-modeling (MLE) loss on trainable layers may optionally be incorporated to preserve language alignment.

This split reduces GPU memory and compute requirements while allowing effective adaptation to the financial domain. Ablation experiments show that updating only a small subset of parameters is sufficient for trading-specific generalization and robust policy learning, while maintaining the LLM’s linguistic generality.

4. Training Methodology

The training procedure is structured as follows:

1. Initialize $\text{lang}(s_t)$7 using LLM pre-training; randomly initialize $\text{lang}(s_t)$8, $\text{lang}(s_t)$9, and $h^{(1)}$0.
2. For each time step $h^{(1)}$1:
   • Observe market state $h^{(1)}$2; generate prompt $h^{(1)}$3.
   • Forward pass: frozen layers $h^{(1)}$4 trainable layers $h^{(1)}$5 heads, yielding $h^{(1)}$6, $h^{(1)}$7.
   • Sample and execute action $h^{(1)}$8; observe reward $h^{(1)}$9 and next state $h^{(2)}$0.
   • Store transition $h^{(2)}$1 in the replay buffer.
   • Every $h^{(2)}$2 steps, compute advantages $h^{(2)}$3 (e.g., via GAE), and perform $h^{(2)}$4 PPO epochs:
     • Update $h^{(2)}$5 using $h^{(2)}$6,
     • Update $h^{(2)}$7 using $h^{(2)}$8,
     • Update $h^{(2)}$9 using $\pi_\theta(a_t \mid s_t)$0.
3. Repeat until convergence or resource budgets are exhausted.

A plausible implication is that this decoupled and memory-efficient training enables rapid iteration on financial tasks with limited compute.

5. Empirical Evaluation and Performance

The empirical evaluation includes five U.S. equities (MSFT, JNJ, UVV, HON, TSLA) and a cryptocurrency (BTC), with the following baselines:

• Buy-and-hold passive strategy.
• INVESTORBENCH LLM-agentic framework with 13 proprietary/open models.

Primary evaluation metrics are Cumulative Return (CR) and Sharpe Ratio (SR), with Annualized Volatility (AV) and Max Drawdown (MDD) as secondary metrics.

Key results using a 135M-parameter SmolLM2 model with FLAG-Trader:

• Average SR improvement of 20–50% across assets (e.g., SR ≈ 1.37 vs. 1.04 on MSFT).
• CR improved by 10–30% on most tickers.
• The 135M-parameter FLAG-Trader configuration outperforms baseline models with up to 70B parameters.

Ablation findings:

• Freezing lower LLM layers preserves pre-trained language skills; domain adaptation occurs with minimal parameter updates.
• Pure RL fine-tuning (without MLE regularization) produces stable trading policies that become less sensitive to prompt details.

6. Insights and Implications

FLAG-Trader demonstrates that fusing LLM reasoning for multimodal time-series and text with RL-based policy optimization can yield domain-specialized agents effective at sequential financial decision-making. The framework’s parameter-efficient adaptation achieves robust results at low computational cost.

Empirical outcomes show that "intelligence + rewards"—interpreted as effective RL optimization of a capable LLM backbone—can yield superior financial performance compared to mere scale, with small open-source models surpassing much larger commercial baselines in trading tasks. The modular approach, leveraging LLMs for structured prompt engineering and efficient RL updates, supports extensibility to other financial-domain objectives and agentic settings (Xiong et al., 17 Feb 2025).