Position-Invariant Evaluation Frameworks for LLM-as-a-Judge

Design a position-invariant evaluation framework for LLM-as-a-Judge that eliminates dependence on the ordering of candidate responses in pairwise and listwise judgments.

Background

LLM judges can exhibit positional bias, favoring candidates based on their placement in the prompt rather than intrinsic quality. Such bias can be exploited by adversaries who manipulate ordering to gain advantage.

The paper identifies the need for frameworks that neutralize ordering effects to ensure fair and consistent judgments.