Separating Stylistic and Content Quality in LLM-as-a-Judge

Design methods that classify and separately evaluate stylistic quality and content quality when LLM-as-a-Judge assesses responses, preventing stylistic presentation from masking factual inaccuracies.

Background

Stylistic features such as fluency, formatting, and rhetorical structure can unduly influence judgments, independent of correctness. This conflation risks inflating scores for well-presented but incorrect content.

The paper proposes distinguishing style from content quality to ensure that factual accuracy carries primary weight in automated evaluations.