STELLAR-E: a Synthetic, Tailored, End-to-end LLM Application Rigorous Evaluator

Abstract: The increasing reliance on LLMs across diverse sectors highlights the need for robust domain-specific and language-specific evaluation datasets; however, the collection of such datasets is challenging due to privacy concerns, regulatory restrictions, and the time cost for manual creation. Existing automated benchmarking methods are often limited by relying on pre-existing data, poor scalability, single-domain focus, and lack of multilingual support. We present STELLAR-E - a fully automated system to generate high-quality synthetic datasets of custom size, using minimal human inputs without depending on existing datasets. The system is structured in two stages: (1) We modify the TGRT Self-Instruct framework to create a synthetic data engine that enables controllable, custom synthetic dataset generation, and (2) an evaluation pipeline incorporating statistical and LLM-based metrics to assess the applicability of the synthetic dataset for LLM-based application evaluations. The synthetic datasets reach an average difference of +5.7% in terms of LLM-as-a-judge scores against existing language-specific benchmarks, demonstrating comparable quality for comprehensive assessment of big and small LLMs. While real datasets remain slightly more challenging for LLMs especially for smaller models, this work establishes a scalable and domain-adaptable benchmarking framework that supports fair evaluation of LLM applications, offering a faster alternative to manual approaches and enabling high-efficiency automated quality assurance cycles.

• The paper introduces STELLAR-E, a fully automated pipeline that generates synthetic instruction-answer datasets, providing scalable LLM evaluation with multilingual support.
• The system employs rigorous multi-stage quality controls, including adversarial paraphrasing and embedding-based filtering, to ensure dataset fidelity and diversity.
• Experiments reveal that synthetic datasets achieve near-human benchmark scores, demonstrating practical benefits for LLMOps, regulatory compliance, and quality assurance.

Formal Analysis of "STELLAR-E: a Synthetic, Tailored, End-to-end LLM Application Rigorous Evaluator" (2604.24544)

Motivation and Background

The operational proliferation of LLMs across regulated industries underscores the necessity for rigorous and customizable evaluation datasets. Data collection in such contexts is impeded by privacy, compliance, and annotation cost constraints. Existing automated benchmarks rely heavily on pre-existing corpora, translation-based augmentation, or narrow domain focus, suffering from limited scalability, insufficient quality assurance, and linguistic bias. The paper addresses these gaps by establishing STELLAR-E—a fully automated end-to-end pipeline for generating and evaluating high-quality synthetic instruction-answer (I{content}A) datasets independent of existing data sources and with enhanced support for multilingual and domain-specific requirements.

Figure 1: An architectural overview of the synthetic I{content}A generation pipeline, illustrating its modular blocks and quality control loops.

System Architecture and Methodology

The STELLAR-E pipeline incorporates a modular, multi-stage process that offers granular control over generated dataset properties. Key stages include:

• Question Type Definition: Manual selection of semantically rich question types forms the starting point, aligning with adversarial and generalization challenges in LLM evaluation.
• Topic Generation and Filtering: For each question type, an LLM produces a diverse set of concise topics. Custom G-Eval scoring ensures semantic and contextual fidelity, discarding topics below threshold $\tau=8$.
• Instruction Generation and Quality Improvement: Instructions are produced for sampled topics using prompts maximizing diversity and conciseness, followed by iterative feedback loops for quality enhancement. Instructions are filtered along dimensions of diversity, relevance, conciseness, and correctness.
• Difficulty Enhancement (DFE): Adversarial paraphrasing increases instruction complexity via dedicated prompts and filtering, ensuring content demands reasoning and non-trivial computation for LLMs.
• Answer Generation and Quality Improvement: Answers are generated sequentially and tested for correctness, relevance, conciseness, completeness, and safety using custom G-Eval.
• Diversity Enhancement (DVE): Embedding-based selection ensures semantic and linguistic diversity. Multilingual embeddings (e.g., bge-m3) are used to deduplicate and enrich dataset variability.
• Meta-evaluation Pipeline: Generated datasets are validated against human-annotated benchmarks via a rigorous suite of metrics (ROUGE-L, BertScore, custom G-Eval), facilitating quantitative fidelity analysis.

  Figure 2: Quality improvement cycle for instruction generation, demonstrating the iterative feedback loop architecture and filtering criteria.

  

  Figure 3: Schema of language datasets utilized, indicating dialectal and translation-based variances for multilingual performance assessment.

Experimental Design

Experiments target evaluation fidelity using the Mintaka dataset (English/Italian, human-curated and machine-translated). Synthetic datasets are generated matching the domain and language specifications. DVE and DFE modules are toggled to assess their impact.

Model selection incorporates a high-performing LLM (Gemini 2.5 Flash) and a smaller model (Llama 2 Chat 13B), facilitating cross-capability validation and bias mitigation. Metrics enforce strict assessment protocols: lexical, semantic, and relevance scoring are aggregated for a nuanced performance profile.

Generation parameters are tuned (e.g., number of question types, topics, DVE thresholds) for balance between diversity and topic coherence. Embedding-based DVE ensures redundancy mitigation while preserving domain coverage.

Results

Strong numerical results confirm that DVE/DFE synthetic datasets yield average G-Eval scores within +5.7% of real benchmarks for both strong and weak models. Synthetic datasets generally inflate LLM-as-a-judge performance, especially for weaker models—Gemini shows +5.7% G-Eval gap to human data, but Llama 2's inflation is +10.9% in English. DVE effectively narrows this gap, and DFE adjusts dataset difficulty, decreasing over-simplicity.

Breakdown analysis reveals synthetic datasets yield higher completeness and accuracy, but lower answer relevance and ROUGE-L compared to real benchmarks. BertScore trends diverge, especially in cross-language experiments, confirming semantic shifts due to adversarial paraphrasing and syntactic diversity.

Machine-translated datasets are less challenging than real ones, indicating translation artifacts versus cultural/linguistic authenticity. Synthetic pipeline produces datasets competitive in complexity and quality, supporting domain adaptability and multilingual scaling.

Implications and Future Directions

STELLAR-E demonstrates that fully synthetic evaluation datasets—augmented by robust quality enhancements and diversity mechanisms—can substitute for expert-crafted benchmarks, supporting scalable LLM benchmarking, monitoring, and QA testing cycles. The architecture minimizes risks of bias, contamination, and linguistic exclusion inherent to manual corpora acquisition.

Practical implications span LLMOps, CI/CD pipeline integration, regulatory compliance validation, and automated QA. The modularity of STELLAR-E enables repurposing for RLHF, RAG, and domain adaptation scenarios, with risk mitigation strategies for dataset contamination.

Theoretically, the approach supports rigorous, adversarial evaluation paradigms, facilitating emergent abilities detection and out-of-distribution robustness analysis. Future research avenues include multi-LLM generation/judge ensembles, broader dataset and task coverage, domain/cultural specificity benchmarking, and targeted human evaluation to expose pipeline-induced systemic bias.

Conclusion

STELLAR-E establishes a scalable, customizable, end-to-end system for synthetic evaluation dataset generation and benchmarking in LLM applications. Experiments confirm synthetic datasets can closely replicate complexity and quality of human benchmarks, with modular enhancements compensating for known simplifications. The pipeline's independence from real data enables rapid adaptation for multilingual and domain-specific applications within regulated sectors, supporting operational efficiency and automated quality assurance. Further development is warranted to address cultural nuances, ensemble bias, and broader domain integration.