[Academic Paper] To Recommend or Not: Recommendability Identification in Conversations with Pre-trained Language Models

Abstract: Most current recommender systems primarily focus on what to recommend, assuming users always require personalized recommendations. However, with the widely spread of ChatGPT and other chatbots, a more crucial problem in the context of conversational systems is how to minimize user disruption when we provide recommendation services for users. While previous research has extensively explored different user intents in dialogue systems, fewer efforts are made to investigate whether recommendations should be provided. In this paper, we formally define the recommendability identification problem, which aims to determine whether recommendations are necessary in a specific scenario. First, we propose and define the recommendability identification task, which investigates the need for recommendations in the current conversational context. A new dataset is constructed. Subsequently, we discuss and evaluate the feasibility of leveraging pre-trained language models (PLMs) for recommendability identification. Finally, through comparative experiments, we demonstrate that directly employing PLMs with zero-shot results falls short of meeting the task requirements. Besides, fine-tuning or utilizing soft prompt techniques yields comparable results to traditional classification methods. Our work is the first to study recommendability before recommendation and provides preliminary ways to make it a fundamental component of the future recommendation system.

Lay summary (by Claude 3 Sonnet): When using chatbots or digital assistants, sometimes you may want recommendations (like for a movie or product), but other times you just want to have a conversation. Current recommendation systems always try to give personalized recommendations without considering if you actually want one in that moment. This can disrupt the natural flow of conversation. Researchers have defined a new problem called “recommendability identification” which aims to determine if a recommendation is actually needed before providing one. They created a dataset for this task and explored using large language models like ChatGPT to identify when recommendations are wanted versus just having a friendly chat. Their experiments showed that simply using these language models doesn’t work well, but techniques like fine-tuning or prompting can allow them to learn this new task. This work introduces recommendability as an important first step before making recommendations to avoid awkwardly shoehorning them into conversations.