Source of gains from arbitrary-order decoding in diffusion language models

Determine whether the performance gains attributed to arbitrary-order token generation in diffusion language models primarily arise from improved exploitation of existing solution patterns encoded in the training data and model, or from enabling qualitatively new reasoning strategies that are unattainable under purely autoregressive left-to-right decoding. Clarify the causal contribution of order arbitrariness to reasoning performance in standard domains such as mathematics and code generation.

Background

Diffusion LLMs (dLLMs) enable arbitrary-order generation, a capability hypothesized to benefit complex reasoning by relaxing the strict left-to-right constraint of autoregressive models. Several works have reported behaviors suggestive of non-standard reasoning strategies and increased diversity tied to order arbitrariness.

At the same time, evidence remains mixed regarding whether observed improvements reflect genuinely new reasoning capabilities or better exploitation of existing solution patterns already learned by the model. Establishing the true origin of these gains is important for deciding whether preserving arbitrary-order mechanisms is necessary for training and inference in dLLMs.