Explain the DR19 random-vs-sorted ordering performance reversal for IC-DQN and IC-IQL

Determine the underlying cause of the observation that IC-DQN (In-Context Twin Deep Q Network) and IC-IQL (In-Context Implicit Q-Learning) achieve higher performance when trained on randomly ordered trajectories than when trained on trajectories sorted by discounted return in the Dark Room 19x19 environment using complete datasets with one learning history per target and no structured learning-history ordering, and ascertain why this phenomenon appears exclusively in the Dark Room 19x19 environment.

Background

Algorithm Distillation relies on ordered, progressively improving learning histories, which may not be available in practice. The authors evaluated RL-based In-Context approaches (IC-DQN, IC-CQL, IC-IQL) when the trajectory data were either randomly shuffled (no inherent improvement order) or sorted by discounted return to impose a heuristic order.

In Dark Room 19x19 with complete datasets and one history per target, the authors observed that IC-DQN and IC-IQL performed better with randomly ordered trajectories than with sorted-by-return samples—contrary to expectations and unlike other environments tested. They explicitly state that they currently lack an explanation for this effect and why it appears only in the simpler DR19 setting.