The Connection Between User Goals and Evaluation Metrics in Recommender Systems

In episode 21 of Recsperts, we welcome Martijn Willemsen, Associate Professor at the Jheronimus Academy of Data Science and Eindhoven University of Technology. Martijn's researches on interactive recommender systems which includes aspects of decision psychology and user-centric evaluation. We discuss how users gain control over recommendations, how to support their goals and needs as well as how the user-centric evaluation framework fits into all of this.

In our interview, Martijn outlines the reasons for providing users control over recommendations and how to holistically evaluate the satisfaction and usefulness of recommendations for users goals and needs. We discuss the psychology of decision making with respect to how well or not recommender systems support it. We also dive into music recommender systems and discuss how nudging users to explore new genres can work as well as how longitudinal studies in recommender systems research can advance insights.

Towards the end of the episode, Martijn and I also discuss some examples and the usefulness of enabling users to provide negative explicit feedback to the system.

Enjoy this enriching episode of RECSPERTS - Recommender Systems Experts.
Don't forget to follow the podcast and please leave a review

• (00:00) - Introduction
• (03:03) - About Martijn Willemsen
• (15:14) - Waves of User-Centric Evaluation in RecSys
• (19:35) - Behaviorism is not Enough
• (46:21) - User-Centric Evaluation Framework
• (01:05:38) - Genre Exploration and Longitudinal Studies in Music RecSys
• (01:20:59) - User Control and Negative Explicit Feedback
• (01:31:50) - Closing Remarks

• Martijn Willemsen on LinkedIn
• Martijn Willemsen's Website
• User-centric Evaluation Framework
• Behaviorism is not Enough (Talk at RecSys 2016)
• Neil Hunt: Quantifying the Value of Better Recommendations (Keynote at RecSys 2014)
• What recommender systems can learn from decision psychology about preference elicitation and behavioral change (Talk at Boise State (Idaho) and Grouplens at University of Minnesota)
• Eric J. Johnson: The Elements of Choice
• Rasch Model
• Spotify Web API

Papers:

• Ekstrand et al. (2016): Behaviorism is not Enough: Better Recommendations Through Listening to Users
• Knijenburg et al. (2012): Explaining the user experience of recommender systems
• Ekstrand et al. (2014): User perception of differences in recommender algorithms
• Liang et al. (2022): Exploring the longitudinal effects of nudging on users’ music genre exploration behavior and listening preferences
• McNee et al. (2006): Being accurate is not enough: how accuracy metrics have hurt recommender systems

General Links:

• Follow me on LinkedIn
• Follow me on X
• Send me your comments, questions and suggestions to marcel.kurovski@gmail.com
• Recsperts Website