LLM-as-an-Interviewer: Beyond Static Testing Through Dynamic LLM Evaluation

Abstract: We introduce LLM-as-an-Interviewer, a novel paradigm for evaluating LLMs. This approach leverages multi-turn interactions where the LLM interviewer actively provides feedback on responses and poses follow-up questions to the evaluated LLM. At the start of the interview, the LLM interviewer dynamically modifies datasets to generate initial questions, mitigating data contamination. We apply the LLM-as-an-Interviewer framework to evaluate six models on the MATH and DepthQA tasks. Our results show that the framework effectively provides insights into LLM performance, including the quality of initial responses, adaptability to feedback, and ability to address follow-up queries like clarification or additional knowledge requests. The framework also addresses key limitations of conventional methods like LLM-as-a-Judge, including verbosity bias and inconsistency across runs. Finally, we propose the Interview Report, which aggregates insights from the interview process, providing examples and a comprehensive analysis of the LLM's strengths and weaknesses. This report offers a detailed snapshot of the model's real-world applicability. The code for our framework is publicly available at https://github.com/interview-eval/.

• The paper introduces a dynamic multi-turn evaluation that refines LLM responses through interactive feedback.
• It mitigates data contamination and bias by adapting benchmarks to simulate realistic user-model dialogues.
• Experiments reveal stabilized performance improvements in models like GPT-4 and Llama 3.1 70B through iterative interaction.

LLM-as-an-Interviewer: Dynamic Evaluation of LLMs

The presented research extends the current methodologies for evaluating LLMs with the introduction of the "LLM-as-an-Interviewer" framework. This evaluation paradigm goes beyond static testing methods by simulating dynamic interaction scenarios akin to a human interview process, thereby attempting to offer a comprehensive assessment of an LLM's performance.

Methodological Framework

The LLM-as-an-Interviewer framework is a two-phase assessment strategy. Initially, it involves adapting existing benchmark datasets to create varied and contextually relevant initial queries. Subsequently, it engages in an interactive evaluation with the model using feedback and follow-up questions derived from the model's responses. This multi-turn interaction is designed to evaluate the model's adaptability and depth of understanding in real-world scenarios.

Key differentiators from the traditional "LLM-as-a-Judge" approach include:

• Data Contamination Mitigation: The framework adjusts benchmark questions to circumvent issues of test data leakage in training datasets.
• Robustness to Bias: By involving multi-turn interactions, the evaluation is less susceptible to verbosity and self-enhancement biases.
• Insight into Model Capabilities: The process provides detailed insights into the model’s ability to handle multi-step interactions, refine responses, and generate clarifications, which are crucial skills for practical deployments.

Numerical Results and Observations

The results derived using the LLM-as-an-Interviewer framework demonstrate its viability and advantages over static evaluation metrics. Specifically, models such as GPT-4, Llama 3.1 70B, and others were assessed, showing consistent performance improvements throughout the iterative feedback-driven process.

Standard deviations across multiple runs decreased during interactions, pointing toward a stabilization of performance as models are given opportunities to refine and adapt their responses. Moreover, both proprietary and open-source models manifested this trend, underscoring the robustness of the framework.

In experiments, LLM-as-an-Interviewer effectively highlighted discrepancies by simulating common user-model interactions in practical applications, such as required clarifications in completion or revelation of models' failure types.

Implications and Future Directions

The introduction of LLM-as-an-Interviewer presents several implications for the development and deployment of LLMs:

1. Practical Applicability: The framework simulates realistic conditions where models are expected to iterate on responses, which aligns with potential use cases in customer support, tutoring systems, and more.
2. Enhanced Evaluation: It provides a more nuanced evaluation by capturing dynamic interaction capabilities and model suitability in real-world contexts.
3. Informing Model Design: Results can guide researchers and developers in refining architectures to enhance adaptability and accuracy in user interactions.

Future developments in this space may include expanding the framework’s application across different domains and task types to fully leverage the potential of interaction-based evaluations. Furthermore, LLM-as-an-Interviewer could play a pivotal role in the ongoing improvement of LLMs' ability to handle complex tasks requiring multi-step reasoning and adaptability to evolving dialogues.

In conclusion, the LLM-as-an-Interviewer framework offers a significant contribution to the methodological toolkit available to researchers, providing a dynamic and comprehensive approach to LLM evaluation that addresses some of the critical limitations inherent in static benchmarking methods.