A Novel Approach of Bench marking Recommendation Systems

Recommendation systems are growing progressively popular due to their ability to offer a personalized experience in a unique way. Leading them to become a very useful tool for different domains from commercially to research. The algorithms used in recommendation systems perform differently depending on the domain and task. To determine what to use, the designer must choose between a set of candidate approaches. Within this process, it must decide which properties of the application should be focused on when making the choice. As recommendation systems have different properties that affect user experience from accuracy, scalability, robustness, and etc.. This causes us to evaluate recommendation systems in many, often incomparable ways. Within this paper, we will discuss an optimal way of evaluating recommendation systems based on different properties and application scenarios. Based on our research, we will propose a novel evaluation concept for recommendation systems based on ideas from research and industry to ensure quality when deployed to the user. The goal is to ensure the recommendation system is significant and goes beyond the conventional accuracy criteria.

View the slides for this session