MLLM-as-a-Judge Protocol

• MLLM-as-a-Judge protocol is a framework that formalizes the use of multimodal LLMs as automated evaluators to assess model output quality.
• It integrates human subject matter expert insights with LLM judgments using pairwise comparisons and aspect-specific evaluations.
• Empirical results reveal domain-specific alignment challenges, underscoring the need for hybrid SME-LLM workflows in high-stakes applications.

The MLLM-as-a-Judge protocol formalizes the use of multimodal LLMs (MLLMs) as automated evaluators ("judges") for model output quality across domains involving text, image, audio, and video modalities. The aim is to scale, unify, and partially automate the historically costly and subjective process of evaluating complex model outputs, including in expert knowledge tasks such as medicine, psychology, and dietetics. Protocols combining LLM judges and human subject matter experts (SMEs) have become widespread, but substantial challenges persist in aligning machine-generated judgments with expert-level human evaluation, especially for tasks requiring specialized expertise (Szymanski et al., 2024).

1. Protocol Foundations and Workflow Design

A canonical MLLM-as-a-Judge protocol incorporates both SME and LLM judges in a mixed evaluation workflow. Task setup begins with selecting one or more expert domains (e.g., dietetics, mental health) and recruiting appropriate SMEs—such as registered dietitians or clinical psychologists—whose educational background, domain experience, and AI familiarity are documented. The evaluation corpus comprises a curated set of domain-specific instructions (e.g., dietary management scenarios, mental health interventions) paired with candidate responses from competing models (e.g., GPT-4o, GPT-3.5-turbo, both at fixed temperature).

Each instruction is further annotated with randomly sampled aspect-evaluation questions representing key domain dimensions (Accuracy, Clarity, Personalization, Professional Standards, Educational Context). Pairwise comparison tasks are then constructed by presenting unordered response pairs, querying judges for both overall preference and aspect-specific judgments, and collecting short free-text explanations. The LLM judge operates under two persona settings—generic and expert persona (explicitly stated)—and is deployed within the AlpacaEval framework for discrete preference prediction (A/B) and justification, using log-probabilities as confidence signals.

Example Prompt Template (AlpacaEval)

<system> You are a clinical psychologist evaluating two model responses for adherence to professional standards. Provide which response (A or B) is better and a 2–3 sentence justification. </system> <user> Instruction: "Describe first-line therapy for mild depression." Aspect question: "Which response uses an empathetic yet clinical tone?" Response A: "..." Response B: "..." </user>

2. Evaluation Metrics and Statistical Formulation

Performance assessment employs both simple agreement metrics and more nuanced measures. Let $N$ be the total number of comparisons and $n_{\text{agree}}$ those where SME and LLM judges concur:

$$\text{PctAgree}(J,S) = \frac{n_{\text{agree}}}{N} \times 100\%$$

Cohen’s kappa quantifies inter-annotator reliability by correcting for chance agreement:

$\kappa = \frac{p_o - p_e}{1 - p_e}$

where $p_o = \frac{n_{\text{agree}}}{N}$ and $p_e = \sum_{c \in \{A,B\}} p_{c,1} p_{c,2}$ across label classes.

Additional analyses include win-rate aggregation per output and inferential comparisons of SME–LLM versus lay–LLM agreement using chi-squared or two-proportion z-tests:

$$z = \frac{p_1 - p_2}{\sqrt{p(1-p)\left(\frac{1}{N_1}+\frac{1}{N_2}\right)}}, \quad p = \frac{n_1 + n_2}{N_1 + N_2}$$

Agreement is also dissected by aspect category (see Table below).

Table: SME–LLM Agreement by Aspect (%)

Aspect	Dietetics (Gen/Ex)	Mental Health (Gen/Ex)
Clarity	55 / 60	70 / 40
Accuracy	56 / 67	80 / 80
Professional Standards	80 / 80	64 / 73
Educational Context	55 / 45	60 / 70
Personalization	56 / 44	67 / 67

3. Empirical Results and Domain-Specific Agreement

Key findings across case studies reveal incomplete but significant alignment between SME and LLM judges in knowledge-specific tasks. For overall preference:

• Dietetics: SME–LLM agreement 68%; SME–SME 75%; lay user–LLM 80%
• Mental Health: SME–LLM agreement 64%; SME–SME 72%; lay user–LLM 80%

Aspect-level agreement varies widely, with Professional Standards generally higher and Clarity more volatile across domains.

These results underscore the limitations of MLLM judgment in capturing nuance, especially for domain-specific aspect queries, and point to the necessity of retaining human experts in high-stakes evaluations.

4. Identified Failure Modes and Alignment Challenges

Systematic analysis of failure modes reveals several model limitations:

• Shallow accuracy checks: LLM judges often reproduce surface-level details and may overlook outdated or dangerous medical/dietetic claims.
• Misaligned clarity definitions: LLMs prefer exhaustive explanation, whereas SMEs value concise, patient-adapted language.
• Persona over-specialization: Explicit expert personas in LLM can induce jargon or reduce accessibility, particularly in educational contexts.
• Domain variability: Heterogeneous guidelines in dietetics and sensitivity to emotional tone in mental health introduce additional misalignment.
• RLHF overfitting: LLMs trained with reinforcement learning from human feedback based on lay judgments achieve higher agreement with lay annotators (80%) versus SMEs (64–68%).

5. Protocol Recommendations for Hybrid Evaluation

Robust protocol recommendations favor a hybrid, SME-in-the-loop approach:

• Use LLM judges for large-scale initial filtering of poor outputs.
• Escalate low-confidence or borderline cases (small log-probability difference $|\log P(A) - \log P(B)| < \tau$) to SME adjudication.
• SMEs audit a stratified sample of LLM judgments, focusing on aspects with historically lower alignment (e.g., Clarity).
• Calibrate confidence thresholds $\tau$ on held-out SME judgments.
• Design explicit prompts including persona details, evidence-weighing instructions, and requests for discrete preference plus rationale.
• Implement conflict-resolution workflows: accept high-confidence LLM decisions; route uncertain ones to SMEs; use SME feedback for prompt/model recalibration.

6. Implementation Details and Workflow Generalization

Implementation proceeds as a structured pipeline:

for instruction in instruction_set: outputs = {A: modelA(instruction), B: modelB(instruction)} shuffle_pair(outputs) llm_pref, llm_logp_diff, llm_expl = LLM_judge(outputs, persona, aspects) human_pref, human_expl = collect_human_judgment(instruction, outputs, aspects) records.append({ 'instruction': instruction, 'llm_pref': llm_pref, 'llm_conf': abs(llm_logp_diff), 'human_pref': human_pref, 'llm_expl': llm_expl, 'human_expl': human_expl }) pct_agree = compute_pct_agreement(records) kappa = compute_cohens_kappa(records) report_results(pct_agree, kappa)

Flowchart summary:

1. Task/data curation
2. Candidate output generation
3. LLM judge + persona deployment
4. Human (SME/lay) judging
5. Agreement analysis
6. Escalation & calibration
7. Final ranking & feedback loop

This protocol is extensible to any expert domain by swapping instruction corpus, aspect queries, and SME pool as needed. It achieves scalable coverage without sacrificing critical depth and expert validation.

7. Context, Significance, and Limitations

MLLM-as-a-Judge protocols offer significant potential for scalable, semi-automated expert evaluation, especially in domains where human annotation is costly or inconsistent. However, empirical evidence shows that for complex, high-stakes, and knowledge-intensive tasks, current LLMs alone do not reach SME-level depth and rigor. While general agreement rates are nontrivial, aspect-specific and domain-specific calibration gaps remain evident. Hybrid approaches—where LLM judges handle bulk filtering and humans provide final arbitration for sensitive cases—present a best-practice pathway, contingent upon rigorous protocol calibration, regular SME audits, and domain-adapted prompt design (Szymanski et al., 2024).

This blueprint is a foundation for future system designs aimed at balancing evaluation efficiency, statistical robustness, and the preservation of expert standards in medical, psychological, and other high-stakes applications leveraging advanced generative models.