GePpeTto Carves Italian into a Language Model

Abstract: In the last few years, pre-trained neural architectures have provided impressive improvements across several NLP tasks. Still, generative LLMs are available mainly for English. We develop GePpeTto, the first generative LLM for Italian, built using the GPT-2 architecture. We provide a thorough analysis of GePpeTto's quality by means of both an automatic and a human-based evaluation. The automatic assessment consists in (i) calculating perplexity across different genres and (ii) a profiling analysis over GePpeTto's writing characteristics. We find that GePpeTto's production is a sort of bonsai version of human production, with shorter but yet complex sentences. Human evaluation is performed over a sentence completion task, where GePpeTto's output is judged as natural more often than not, and much closer to the original human texts than to a simpler LLM which we take as baseline.