Mean BERTs make erratic language teachers: the effectiveness of latent bootstrapping in low-resource settings

Abstract: This paper explores the use of latent bootstrapping, an alternative self-supervision technique, for pretraining LLMs. Unlike the typical practice of using self-supervision on discrete subwords, latent bootstrapping leverages contextualized embeddings for a richer supervision signal. We conduct experiments to assess how effective this approach is for acquiring linguistic knowledge from limited resources. Specifically, our experiments are based on the BabyLM shared task, which includes pretraining on two small curated corpora and an evaluation on four linguistic benchmarks.