arxiv preprint - Evaluating Large Language Models as Generative User Simulators for Conversational Recommendation

In this episode, we discuss Evaluating Large Language Models as Generative User Simulators for Conversational Recommendation by Se-eun Yoon, Zhankui He, Jessica Maria Echterhoff, Julian McAuley. The paper presents a new protocol with five tasks to assess the performance of synthetic users, generated by large language models, aiming to mimic human behavior in conversational recommender systems. The tasks evaluate essential features such as discussing items, stating preferences, asking for recommendations, and providing feedback. Initial evaluations show that these tasks can identify how language models differ from actual human behavior and suggest how model tuning and prompting can improve the synthetic users' resemblance to real users.