Evaluation of Multi-Agent Video Recommender Systems

Establish a rigorous, multi-dimensional evaluation framework for multi-agent video recommender systems that goes beyond offline metrics such as nDCG and MRR to capture context-aware, conversational benefits and coordination effects, and develop robust validation procedures to assess whether LLM-based user simulation ensembles (such as Agent4Rec and VRAgent-R1) faithfully reproduce real human behavior.

Background

The paper argues that conventional offline metrics like nDCG and MRR, while necessary, are insufficient for capturing the unique properties of collaborative, agentic recommendation frameworks, including coordination, reasoning quality, and user-centric conversational benefits.

The authors further note that ensembles of LLM-driven user simulators (e.g., Agent4Rec, VRAgent-R1) face an alignment and validation challenge: without robust procedures to confirm that simulated behavior matches real human behavior, it is difficult to trust findings derived from such simulations or to rely on them for offline training.