Enhancing Speech Large Language Models through Reinforced Behavior Alignment

Abstract: The recent advancements of LLMs have spurred considerable research interest in extending their linguistic capabilities beyond text to other modalities, which leads to emergence of speech-based LLMs (SpeechLMs) with capability of processing user request in either speech or textual formats. However, owing to inter-modal discrepancies, these SpeechLMs still exhibit a significant performance gap compared to their text-based LLM counterparts in instruction-following, particularly when confronted with the dynamic and variable nature of user speech. To address this challenge, this paper introduces a framework termed Reinforced Behavior Alignment (RBA), designed to bolster the language generation proficiency of SpeechLMs. Instead of relying on supervised fine-tuning from human annotations, RBA employs a self-synthesis methodology to generate extensive, high-fidelity alignment data by a powerful teacher LLM. Then SpeechLMs is aligned its behavior with that of a teacher using a reinforcement learning-based approach. Experimental results demonstrate that this method effectively enhances the instruction-following capabilities of SpeechLMs that outperform conventional distillation baselines. Crucially, we demonstrate that RBA can be seamlessly extended to tasks such including spoken question answering and speech-to-text translation, attaining state-of-the-art performance on open benchmarks with only self-generated data.

• The paper improves Speech LLMs by introducing Reinforced Behavior Alignment, aligning them with advanced text-based models to enhance instruction-following.
• It employs self-synthesis data generation and reinforcement learning methodologies to bridge inter-modal performance gaps with high-quality synthetic instruction-response pairs.
• Results demonstrate significant performance gains in spoken question answering and speech-to-text translation, underlining its potential in multimodal applications.

Enhancing Speech LLMs through Reinforced Behavior Alignment

Introduction

The paper "Enhancing Speech LLMs through Reinforced Behavior Alignment" (2509.03526) presents an innovative approach to improving the capability of Speech LLMs (SpeechLMs) to perform instruction-following tasks using a framework called Reinforced Behavior Alignment (RBA). This research addresses the notable performance gap between SpeechLMs and their text-based counterparts in handling instructions, a discrepancy largely attributed to inter-modal differences. SpeechLMs often struggle with the variability inherent to spoken language. To bridge this gap, the authors propose aligning SpeechLM behavior with that of advanced text-based teacher models through reinforcement learning techniques, facilitated by high-quality synthetic data generation.

Methodology

The study introduces RBA, a two-step framework that significantly enhances SpeechLM performance:

1. Self-Synthesis Data Generation:

RBA constructs a large-scale instruction dataset through self-synthesis, eliminating the need for manual annotation. This involves using a text-based LLM, specifically a high-capacity aligned LLM, to generate pairs of instructions and their corresponding responses.

• Instruction Sampling: By employing a query template with a defined structure, the LLM autonomously generates diverse instructions, ensuring a broad representation of potential user queries without direct supervision.
• Response Completion: The teacher LLM provides corresponding responses to the generated instructions, ensuring alignment with human-like preferences and high-quality instruction-response pairs. Filtering criteria are applied to maintain practical and realistic speech-based interaction data, avoiding complex or overly technical instructions.
• User Speech Generation: Utilizing a pre-trained TTS model, multiple speaker variations are synthesized to create a diverse auditory instruction set, thereby enhancing the SpeechLM's capability to handle varying speech inputs.

  Figure 1: Frameworks of RBA. Step 1: generate text user instruction by modifying pre-defined query templates, followed by generating spoken instructions by TTS model. Step 2: complete text response by teacher LLMs.

2. Reinforcement Learning for SpeechLMs:

The second phase employs reinforcement learning to align the behavior of SpeechLMs with that of the teacher LLM:

• Reward Modeling: The alignment process uses a pre-trained reward model to evaluate speech-generated responses. Two strategies, RBA-Group and RBA-Single, are introduced for positive-negative sampling based on multi-speaker inputs. RBA-Group focuses on choosing the best result from a set, while RBA-Single uses teacher LLM-generated data as a baseline for comparison.
• Optimization Process: Reinforcement learning techniques adjust the SpeechLM to align closely with the sophisticated responses of LLMs, overcoming biases and improving the model's performance in generating human-like responses across different speech tasks.

Results and Discussion

The paper reports significant improvements in accuracy and robustness of SpeechLMs, outperforming traditional text-to-speech fine-tuning approaches. Key performance gains are observed across various benchmarks in spoken question answering (SQA) and speech-to-text translation (S2TT) tasks. Notably, the self-synthetic dataset used in RBA facilitates substantial speech model improvements without requiring extensive annotated data, showcasing the efficacy of using self-synthesized data for behavioral alignment.

• Instruction-Following: The RBA framework, particularly the RBA-Single variant, achieves higher win rates and better generalization across domains when evaluated against baseline models.
• Adaptation to Downstream Tasks: The method's adaptability allows it to perform exceptionally well in SQA and S2TT tasks, highlighting its effectiveness in multimodal applications.

Conclusion

The proposed RBA method successfully addresses the challenge of inter-modal discrepancies in SpeechLMs by aligning their behavior with advanced LLMs through reinforcement learning. This alignment significantly enhances instruction-following capabilities, effectively closing the gap between speech-based and text-based models. The study demonstrates the potential of self-synthesized data in avoiding the limitations of conventional annotated datasets and opens pathways for adapting RBA to other speech and multimodal tasks. Future research could explore expanding this framework to include additional modalities and refining model efficiency further.