Transferability of decomposed supervision to the authentic Japanese bar exam format

Determine whether large language models trained on decomposition-based datasets that reformulate Japanese bar examination multiple-choice questions into independent true/false statements (for example, the Japanese Bar Exam Question Answering (JBE-QA) paradigm) can successfully answer intact multiple-choice questions from the Japanese bar examination when evaluated under the original joint-proposition format and official scoring scheme requiring reasoning over multiple interacting propositions.

Background

The Japanese bar examination’s multiple-choice section requires joint evaluation of several statements and strict adherence to an answer format in which errors in any constituent can invalidate the full response. Recent work such as JBE-QA simplifies this structure by decomposing each exam question into independent true/false judgments, which stabilizes learning but alters the original task.

Because the decomposed formulation optimizes binary classification rather than constrained selection under combinatorial rules, it is unclear whether models trained under such supervision can generalize to intact questions evaluated by original scoring criteria. The paper explicitly raises this uncertainty and positions its format-faithful dataset and self-verification approach as a direct test of this question.