Origin of cross-genre differences in code length and conditional entropy

Ascertain whether the observed cross-genre differences in the dependence of code length L(N) and conditional entropy s(N) on context length N arise from inherent properties of the underlying texts in each genre or from each genre’s relationship to the training sets used to train the large language models employed for estimation.

Background

Across genres (e.g., internet corpus, Wikipedia, poetry), the authors find that code lengths and conditional entropies behave differently as functions of context length. Poetry shows longer code lengths and an apparent plateau, while other corpora show continued decline.

The authors explicitly state uncertainty about whether these differences reflect intrinsic properties of the genres themselves or are artifacts of how those genres were represented in the models’ training data, highlighting a key unresolved issue in interpreting LLM-based entropy measurements of language.