Fabricator: An Open Source Toolkit for Generating Labeled Training Data with Teacher LLMs

Abstract: Most NLP tasks are modeled as supervised learning and thus require labeled training data to train effective models. However, manually producing such data at sufficient quality and quantity is known to be costly and time-intensive. Current research addresses this bottleneck by exploring a novel paradigm called zero-shot learning via dataset generation. Here, a powerful LLM is prompted with a task description to generate labeled data that can be used to train a downstream NLP model. For instance, an LLM might be prompted to "generate 500 movie reviews with positive overall sentiment, and another 500 with negative sentiment." The generated data could then be used to train a binary sentiment classifier, effectively leveraging an LLM as a teacher to a smaller student model. With this demo, we introduce Fabricator, an open-source Python toolkit for dataset generation. Fabricator implements common dataset generation workflows, supports a wide range of downstream NLP tasks (such as text classification, question answering, and entity recognition), and is integrated with well-known libraries to facilitate quick experimentation. With Fabricator, we aim to support researchers in conducting reproducible dataset generation experiments using LLMs and help practitioners apply this approach to train models for downstream tasks.

• The paper introduces a teacher-student paradigm where large LLMs generate datasets for training smaller, task-specific models.
• It demonstrates comparable performance in binary classification tasks against manually labeled benchmarks on datasets like IMDB.
• The study highlights the importance of prompt engineering and few-shot exemplars to optimize label quality and model efficacy.

Insights on Labeled Training Data Generation using Teacher LLMs

The paper presents an open-source toolkit designed to facilitate the generation of labeled datasets using LLMs as "teachers" to train smaller, downstream NLP models. Recognizing the cost and time associated with manual data labeling, this work addresses these challenges by innovating new approaches to automatically generate labeled training data using LLMs.

Key Methodological Insights

The proposed toolkit implements a workflow utilizing a teacher-student paradigm in dataset generation. Specifically, an LLM serves as a teacher that creates datasets based on task-specific prompts. These datasets are subsequently employed to train smaller, task-specific models. The authors integrate the toolkit with familiar libraries like HuggingFace, which allows easy access and sharing of generated datasets.

The study emphasizes three main workflows in dataset generation:

1. Unlabeled Data Generation: Users provide a prompt instructing the LLM to generate content within a specified domain, useful for assembling training corpora without labels.
2. Label-Conditioned Data Generation: Designed for tasks like classification, where the LLM generates samples corresponding to predefined classes using label-informed prompts.
3. Annotating Unlabeled Data: Augments existing unlabeled datasets with labels synthesized through LLM-generated prompts.

Experimental Evaluation

The empirical evaluation involved several popular NLP benchmarks, including IMDB for sentiment analysis and MRPC for textual similarity. Pretrained LLMs, when tasked with generating training data, enabled downstream models to achieve competitive performance relative to those trained on manually-annotated datasets, particularly in straightforward binary classification tasks.

Crucially, the efficacy of generated data varied across different tasks. For instance, binary sentiment analysis tasks like IMDB exhibited negligible performance drops, contrasting with more complex tasks like extractive question answering where a substantial discrepancy remained. This underscores the need for further refinement in employing LLMs for complex data generation.

Another noteworthy experiment highlighted the role of few-shot learning by varying the number of few-shot examples included in prompts to improve the quality of generated data. As observed, increasing few-shot exemplar augmentation sometimes improved model performance, although an optimal balance was necessary to prevent diminishing returns.

Limitations and Future Directions

The study acknowledges several limitations, including the varying effectiveness of LLM-generated datasets across different NLP tasks and the challenges surrounding prompt design and other generation parameters. Indeed, effective prompt engineering is a non-trivial aspect of efficiently harnessing LLMs for various applications, as optimal models must balance between data volume and label reliability.

The authors suggest that future work could extend the support of the toolkit to cover a broader range of NLP tasks, especially those that are resource-intensive in terms of data requirements. Potential advancements in combining LLM-generated datasets with small amounts of high-quality human-annotated data could offer robust solutions to data sparsity challenges, especially in domain-specific applications.

Practical and Theoretical Implications

The research provides a compelling perspective on leveraging LLMs for dataset generation, significantly reducing the manual effort necessary in developing NLP applications. The immediate practical implications involve facilitating research adaptability and scalability, especially in low-resource settings. Furthermore, the study opens theoretical avenues on the teacher-student paradigm in machine learning, encouraging exploration into the pedagogical effectiveness of LLMs as data teachers.

Overall, this study contributes a valuable framework for both the practical application of LLM-driven data generation in NLP and advances our understanding of data efficiency paradigms in machine learning.