Gained in Translation: Privileged Pairwise Judges Enhance Multilingual Reasoning

Abstract: When asked a question in a language less seen in its training data, current reasoning LLMs (RLMs) often exhibit dramatically lower performance than when asked the same question in English. In response, we introduce \texttt{SP3F} (Self-Play with Privileged Pairwise Feedback), a two-stage framework for enhancing multilingual reasoning without \textit{any} data in the target language(s). First, we supervise fine-tune (SFT) on translated versions of English question-answer pairs to raise base model correctness. Second, we perform RL with feedback from a pairwise judge in a self-play fashion, with the judge receiving the English reference response as \textit{privileged information}. Thus, even when none of the model's responses are completely correct, the privileged pairwise judge can still tell which response is better. End-to-end, \texttt{SP3F} greatly improves base model performance, even outperforming fully post-trained models on multiple math and non-math tasks with less than of the training data across the single-language, multilingual, and generalization to unseen language settings.

• The paper introduces SP3F, a novel two-stage framework combining supervised fine-tuning on translated responses with reinforcement learning guided by privileged pairwise judges.
• It employs a unique method where an LLM judge, with privileged access to English reference answers, reliably rewards correct reasoning chains even in low-resource languages.
• The approach achieves significant gains, with over 3x accuracy improvements in languages like Indonesian and Bengali while using only 1/8 of the typical training data.

SP3F: Advancing Data-Efficient Multilingual Reasoning with Privileged Pairwise Judges

Introduction

Reasoning performance of LLMs often degrades precipitously when deployed in lower-resource languages due to insufficient data and weak cross-lingual transfer from predominantly English-centric training pipelines. The paper "Gained in Translation: Privileged Pairwise Judges Enhance Multilingual Reasoning" (2601.18722) introduces SP3F (Self-Play with Privileged Pairwise Feedback), a two-stage post-training framework that improves LLM reasoning performance for target languages even when no data in those languages is available. SP3F combines supervised finetuning (SFT) over translated reference responses and reinforcement learning (RL) guided by a privileged LLM judge leveraging English source answers for feedback.

The SP3F Framework

SP3F comprises two algorithmic phases:

1. SFT on Translated Reference Responses:

Given plentiful English question-answer pairs $(x, y^*)$, both questions and reference solutions are translated into the target language. SFT maximizes the likelihood of generating the translated solution under the model given a translated question, facilitating better mode selection for subsequent RL. The translation uses automated systems and the approach is applied single-language or across a multilingual set.

2. RL with Privileged Pairwise Judge Feedback:

RL updates are driven by a composite reward, integrating standard verifiable signals (correctness, formatting, language fidelity) and preference judgments from an LLM "judge" possessing privileged access to the original English reference response. For each batched prompt, multiple candidate responses are sampled, and the judge is tasked with selecting the response closest in reasoning and correctness to the reference. This setup enhances reliability in rewarding correct chains-of-thought (CoT), even when the final answer is incorrect—a crucial step for mitigating the cold-start problem in low-resource languages.

Figure 1: SP3F outperforms Qwen2.5-7B-Instruct across 4 tasks, achieving higher accuracy and language fidelity while using only $\frac{1}{8}$ as much training data.

Figure 2: The RL pipeline uses a pairwise judge informed by English reference responses, comparing $N$ outputs per prompt and rewarding based on empirical win-rate.

Experimental Setup

SP3F-7B is derived from Qwen2.5-7B, trained using questions translated into 18 different languages and the corresponding solutions. Datasets include both in-domain (math: MGSM, MT-Math100) and out-of-domain (Belebele, Global MMLU Lite) benchmarks. Evaluation metrics are accuracy (final answer correctness) and language fidelity (fraction of response in target language). Baselines include Qwen2.5-7B-Instruct (1M post-training examples) and "Translate Test" (inference-time translation boosting).

Results: Gains of SP3F

Single-Language and Multilingual Performance

SP3F demonstrates marked improvements over post-trained models—especially in lower-resource languages (e.g., Indonesian, Swahili, Bengali), where accuracy multipliers exceeding $3\times$ over Qwen2.5-7B-Instruct are observed. Both SFT and RL phases are essential; SFT alone raises baseline accuracy, but RL with judge feedback brings robust reasoning gains.

Figure 3: SP3F trained in a single language outperforms Qwen2.5-7B-Instruct in all tasks; performance deltas are largest in low-resource languages.

Figure 4: SP3F-7B gains over Qwen2.5-7B-Instruct are particularly pronounced in math reasoning and out-of-domain tasks for lower-resource languages.

SP3F-7B, trained with multilingual data and SP3F pipeline, consistently surpasses Qwen2.5-7B-Instruct across all in-language and cross-lingual benchmarks—achieving higher scores with $1/8$th the post-training data. Translate Test inference-time boosts do not close this gap.

Generalization to Unseen Languages

Crucially, SP3F-7B generalizes to unseen languages beyond its training set, outperforming Qwen2.5-7B-Instruct with an average absolute improvement of $18\%$ in reading comprehension and $3.4\%$ in math accuracy.

Privileged Pairwise Judges: Technical Insights

Improving Judge Reliability and Reducing Intransitivity

Providing privileged information (English reference responses) to the LLM judge reduces noise and intransitivity in feedback. Intransitive preferences of LLM judges—where circular ranking of responses undermines reward modeling—are diminished, and the judge achieves better detection of correct CoT even if final answers are incorrect.

Figure 5: Privileged judges demonstrate lower intransitivity compared to non-privileged judges, yielding more reliable pairwise preference orderings.

Privileged feedback especially aids the early RL cold start, when correct reasoning chains are sparse. Empirical ablations show privileged judges are better at ranking responses with correct intermediate steps, thus increasing sample efficiency and robustness in model updates.

Figure 6: Models trained with non-privileged judge feedback still prefer generations of models trained with privileged feedback, implying broad qualitative improvements.

Training with privileged judges yields superior models in both in-domain and (to a lesser extent) out-of-domain tasks. The effect is validated by comparative pairwise evaluations—models trained with privileged feedback are favored even by non-privileged judges, indicating generalized improvements rather than narrow reward optimization.

Implications and Future Directions

SP3F substantiates that (1) data-efficient cross-lingual reasoning can be unlocked by leveraging privileged information in RL reward modeling, and (2) pairwise self-play protocols are robust against LLM judge intransitivity, allowing learning to be driven by dense, reliable relative feedback. By requiring only English reference responses, SP3F makes it feasible to bring high reasoning fidelity to resource-poor languages, advancing the practical deployment of LLMs globally. The demonstrated generalization to unseen languages and tasks, alongside robust sample efficiency, indicates significant promise for scaling multilingual reasoning without prohibitive data curation costs.

Potential future extensions include broader adoption of privileged pairwise judges in other RL fine-tuning paradigms, integration with advanced regularization for further domain generalization, and tighter analysis of judge composition for optimal feedback reliability.

Conclusion

The SP3F framework sets forth a principled, efficient, and broadly applicable methodology for post-training LLMs for multilingual reasoning in the absence of target-language data. By reusing English reference solutions in both SFT and privileged RL feedback, SP3F-7B models outperform considerably larger baselines with drastically less data. Theoretical justifications and extensive empirical analysis of privileged judges elucidate their advantages in feedback consistency and sample efficiency. These insights will inform future developments in multilingual LLM reasoning and RL-based model post-training.