Modelling the semantics of text in complex document layouts using graph transformer networks

Abstract: Representing structured text from complex documents typically calls for different machine learning techniques, such as LLMs for paragraphs and convolutional neural networks (CNNs) for table extraction, which prohibits drawing links between text spans from different content types. In this article we propose a model that approximates the human reading pattern of a document and outputs a unique semantic representation for every text span irrespective of the content type they are found in. We base our architecture on a graph representation of the structured text, and we demonstrate that not only can we retrieve semantically similar information across documents but also that the embedding space we generate captures useful semantic information, similar to LLMs that work only on text sequences.