Sparse Autoencoders Reveal Interpretable Structure in Small Gene Language Models

Abstract: Sparse autoencoders (SAEs) have recently emerged as a powerful tool for interpreting the internal representations of LLMs, revealing latent latent features with semantical meaning. This interpretability has also proven valuable in biological domains: applying SAEs to protein LLMs uncovered meaningful features related to protein structure and function. More recently, SAEs have been used to analyze genomics-focused models such as Evo 2, identifying interpretable features in gene sequences. However, it remains unclear whether SAEs can extract meaningful representations from small gene LLMs, which have fewer parameters and potentially less expressive capacity. To address it, we propose applying SAEs to the activations of a small gene LLM. We demonstrate that even small-scale models encode biologically relevant genomic features, such as transcription factor binding motifs, that SAEs can effectively uncover. Our findings suggest that compact gene LLMs are capable of learning structured genomic representations, and that SAEs offer a scalable approach for interpreting gene models across various model sizes.