Efficiently recovering post-training capabilities after hybrid distillation

Develop an efficient method to recover the instruction-following and alignment capabilities introduced by post-training in the original Transformer base models after they are converted via distillation into RNN–attention hybrid architectures using pre-training-style data.

Background

The paper presents HALO, a distillation pipeline that converts pre-trained Transformer models into RNN–attention hybrid architectures (HypeNet) using primarily pre-training-style data. While this conversion yields strong long-context efficiency and competitive performance, it can diminish capabilities that are typically introduced during post-training, such as instruction-following and alignment.

The authors note that this degradation is a common shortcoming across existing distillation-based conversions to hybrid models, not just their own approach. They explicitly identify the efficiency-focused recovery of these post-training capabilities in the converted hybrid models as an open question.