ATEB: Evaluating and Improving Advanced NLP Tasks for Text Embedding Models

Abstract: Traditional text embedding benchmarks primarily evaluate embedding models' capabilities to capture semantic similarity. However, more advanced NLP tasks require a deeper understanding of text, such as safety and factuality. These tasks demand an ability to comprehend and process complex information, often involving the handling of sensitive content, or the verification of factual statements against reliable sources. We introduce a new benchmark designed to assess and highlight the limitations of embedding models trained on existing information retrieval data mixtures on advanced capabilities, which include factuality, safety, instruction following, reasoning and document-level understanding. This benchmark includes a diverse set of tasks that simulate real-world scenarios where these capabilities are critical and leads to identification of the gaps of the currently advanced embedding models. Furthermore, we propose a novel method that reformulates these various tasks as retrieval tasks. By framing tasks like safety or factuality classification as retrieval problems, we leverage the strengths of retrieval models in capturing semantic relationships while also pushing them to develop a deeper understanding of context and content. Using this approach with single-task fine-tuning, we achieved performance gains of 8\% on factuality classification and 13\% on safety classification. Our code and data will be publicly available.

• The paper introduces ATEB, a benchmark that reformulates advanced NLP tasks into retrieval problems using contrastive loss.
• It demonstrates that fine-tuning with label augmentation improves performance by 8% on factuality and 13% on safety classifications.
• The study highlights the efficiency of adapter-based fine-tuning, achieving competitive results with minimal computational costs.

ATEB: Evaluating and Improving Advanced NLP Tasks for Text Embedding Models

Introduction

The paper "ATEB: Evaluating and Improving Advanced NLP Tasks for Text Embedding Models" introduces a benchmark specifically designed to evaluate the capabilities of text embedding models in handling complex NLP tasks beyond traditional semantic similarity assessments. These tasks, such as reasoning, safety, factuality, and instruction following, require deep contextual understanding and often involve sensitive content handling or the verification of factual statements against authoritative sources.

Benchmark Design

The ATEB benchmark comprises a diverse set of tasks that simulate real-world scenarios requiring advanced capabilities from embedding models. The benchmark encompasses four main categories:

1. Factuality Classification: It includes tasks like ESNLI, DialFact, and VitaminC, aimed at assessing models' performance in natural language inference (NLI) and fact verification.
2. Safety Classification: Tasks such as BeaverTails Safety Classification and HH-RLHF Harmlessness Classification focus on binary classification of inputs as safe or unsafe.
3. Instruction Following as Reranking: Reformulated tasks from instruction-following domains, including Stanford Human Preference, AlpacaFarm, LMSys, and others, require ranking model-generated responses based on human preferences.
4. Reasoning as Retrieval: Embedding models are tasked with retrieving correct answers from data pools for challenges like HellaSwag, Winogrande, PIQA, and ARCChallenge.

Additionally, the benchmark encompasses document-level paraphrasing (DIPPER) as pairwise-classification and document-level machine translation (Europarl, IWSLT17, NC2016) as bitext-mining tasks.

Methodology

The authors proposed a novel fine-tuning approach that reformulates various classification tasks into retrieval-based settings using contrastive loss. This approach represents each instance as a triplet: an input, a positive target, and multiple negative targets, allowing the development of dual encoder embedding models without architectural changes. By providing detailed explanations with label texts, the model focuses on learning semantic meanings rather than unique ID tokens.

Results

The empirical evaluation of advanced embedding models on ATEB highlights several findings:

• Baseline Models: Embedding models such as Gemma-2B and Google Gecko, when evaluated against the benchmark, exhibited near-random performance on many reranking, retrieval, and classification tasks, indicating the need for more targeted fine-tuning and intelligent adaptation strategies.
• Improvement via Fine-Tuning: The use of label augmentation during fine-tuning resulted in significant performance improvements on both factuality and safety classification tasks. Specifically, performance gains of 8% on factuality classification and 13% on safety classification were observed.
• Adapter-based Fine-Tuning: A lightweight approach leveraging adapters yielded competitive results with minimal computational cost, offering both efficiency and accuracy in model training.

Conclusion

ATEB serves as a comprehensive benchmark to expose the limitations of current embedding models in addressing complex NLP tasks. By transforming these tasks into retrieval problems augmented with label explanations, embedding models can better harness their capability in semantic relationship capture and contextual understanding. This approach can significantly improve performance, especially in factuality and safety tasks. The study underscores the necessity for specialized benchmarks and tailored fine-tuning methods to advance the capabilities of embedding models.

Future Directions

Moving forward, integrating more diverse datasets relating to advanced NLP tasks, including those focusing on cultural nuances, ethics, and dynamic reasoning, could further enhance the robustness and applicability of ATEB. Additionally, exploring adaptive learning techniques that dynamically adjust to task-specific requirements may offer improved model performance and efficiently scale across varied domains.