Risk-sensitive optimization of neural deep learning ranking models with applications in ad-hoc retrieval and recommender systems

Abstract

We answer open research questions regarding the (hard) problem of incorporating risk-sensitiveness measures into Deep Neural Networks for ranking models of retrieval and recommender systems. Risk-sensitive measures are important for controlling the bias towards the average when optimizing ranking solutions’ effectiveness. In previous work, we proposed the RiskLoss function which presents two important adaptations for neural network ranking in ad-hoc retrieval: a differentiable loss function and the use of networks’ sub-portions, obtained via dropout, as baseline systems for optimizing risk sensitiveness. However, questions remained to be answered regarding the generality, cost, and applicability of our solution. In this article, we respond to these questions by (i) applying RiskLoss to ranking in recommender systems, (ii) analyzing the execution cost of RiskLoss and (iii) providing an experimental evaluation of RiskLoss’ resilience to overfitting. Our experiments, comparing seven loss functions on three benchmark recommendation datasets (AIV, ML35M, ML25M, ML100K and ML1M) and four Learning To Rank datasets (WEB30K, WEB10K, YAHOO and MQ2007), with thousands to millions of interactions, reveal that RiskLoss presents the most consistent risk sensitiveness behavior, with gains up to 4.5% in GeoRisk@10 without significant losses in effectiveness. In particular, RiskLoss can reduce the number of bad recommendations by over 11% for “hard to recommend” users. We also show that RiskLoss is not much affected by overfitting.