Interpretable Sentence Representation with Variational Autoencoders and Attention

Abstract: In this thesis, we develop methods to enhance the interpretability of recent representation learning techniques in NLP while accounting for the unavailability of annotated data. We choose to leverage Variational Autoencoders (VAEs) due to their efficiency in relating observations to latent generative factors and their effectiveness in data-efficient learning and interpretable representation learning. As a first contribution, we identify and remove unnecessary components in the functioning scheme of semi-supervised VAEs making them faster, smaller and easier to design. Our second and main contribution is to use VAEs and Transformers to build two models with inductive bias to separate information in latent representations into understandable concepts without annotated data. The first model, Attention-Driven VAE (ADVAE), is able to separately represent and control information about syntactic roles in sentences. The second model, QKVAE, uses separate latent variables to form keys and values for its Transformer decoder and is able to separate syntactic and semantic information in its neural representations. In transfer experiments, QKVAE has competitive performance compared to supervised models and equivalent performance to a supervised model using 50K annotated samples. Additionally, QKVAE displays improved syntactic role disentanglement capabilities compared to ADVAE. Overall, we demonstrate that it is possible to enhance the interpretability of state-of-the-art deep learning architectures for language modeling with unannotated data in situations where text data is abundant but annotations are scarce.

• The paper refines semi-supervised VAEs to build faster, smaller, and simpler models for clear sentence representation.
• It introduces two models—ADVAE and QKVAE—with QKVAE effectively disentangling syntactic and semantic information.
• Transfer experiments show QKVAE matching supervised performance, emphasizing its capability with unannotated data.

The paper "Interpretable Sentence Representation with Variational Autoencoders and Attention" focuses on enhancing the interpretability of representation learning techniques in NLP, particularly under conditions where annotated data is unavailable. The study leverages Variational Autoencoders (VAEs) for their effectiveness in learning data-efficient and interpretable representations.

Contributions and Methodology

1. Optimizing VAEs:
   • The authors begin by refining the semi-supervised VAEs, aiming to streamline their functionality by removing unnecessary components. This optimization results in models that are faster, smaller, and simpler to design.
2. Models for Interpretability:
   • Two main models are introduced:
     • Attention-Driven VAE (ADVAE): This model is crafted to distinctly represent and control information related to syntactic roles within sentences. It employs attention mechanisms to separate this syntactic information.
     • QKVAE: Built upon a novel use of VAEs and Transformers, QKVAE utilizes separate latent variables for forming keys and values in a Transformer decoder, tasked with disentangling syntactic from semantic information in the representations.

Results and Experiments

• Transfer Experiments:
  • QKVAE achieves notable performance, comparable to supervised models, even when using an equivalent amount of unannotated data as a model trained on 50K annotated samples.
  • The model exhibits superior capabilities in disentangling syntactic roles compared to ADVAE.

Impact and Implications

The research underscores the potential for developing interpretable models using unannotated data, an essential advancement when dealing with ample text data but limited annotations. The paper highlights the feasibility of improving the interpretability of advanced deep learning architectures for language modeling—illustrating that it’s possible to extract meaningful and understandable latent representations without relying heavily on annotated datasets.

This work contributes to the broader field by providing methods to facilitate the interpretability of complex models, thereby making them more accessible for various NLP applications where interpretability and data-efficiency are paramount.