arXiv：cs.AI· Junling Liu, Chao Liu, Peilin Zhou, Qichen Ye, Dading Chong, Kang Zhou, Yueqi Xie, Yuwei Cao, Shoujin Wang, Chenyu You, Philip S. Yu·· 21 小时前
LLMRec：在推荐任务上评测大语言模型
LLMRec: Benchmarking Large Language Models on Recommendation Task
AI 导读
研究者提出基于 LLM 的推荐系统 LLMRec，在评分预测、序列推荐、直接推荐、解释生成和评论摘要五项任务上评测 ChatGPT、LLaMA、ChatGLM 等现成大语言模型。基准结果显示，LLM 在序列推荐和直接推荐等准确性任务上仅表现中等，但在可解释性任务上达到与 SOTA 方法相当的水平。团队还评估了监督微调对提升 LLM 指令遵循能力的效果，并开源了代码、处理后的数据与基准结果。
正文
Authors:Junling Liu, Chao Liu, Peilin Zhou, Qichen Ye, Dading Chong, Kang Zhou, Yueqi Xie, Yuwei Cao, Shoujin Wang, Chenyu You, Philip S.Yu
Abstract:Recently, the fast development of Large Language Models (LLMs) such as ChatGPT has significantly advanced NLP tasks by enhancing the capabilities of conversational models. However, the application of LLMs in the recommendation domain has not been thoroughly investigated. To bridge this gap, we propose LLMRec, a LLM-based recommender system designed for benchmarking LLMs on various recommendation tasks. Specifically, we benchmark several popular off-the-shelf LLMs, such as ChatGPT, LLaMA, ChatGLM, on five recommendation tasks, including rating prediction, sequential recommendation, direct recommendation, explanation generation, and review summarization. Furthermore, we investigate the effectiveness of supervised finetuning to improve LLMs' instruction compliance ability. The benchmark results indicate that LLMs displayed only moderate proficiency in accuracy-based tasks such as sequential and direct recommendation. However, they demonstrated comparable performance to state-of-the-art methods in explainability-based tasks. We also conduct qualitative evaluations to further evaluate the quality of contents generated by different models, and the results show that LLMs can truly understand the provided information and generate clearer and more reasonable results. We aspire that this benchmark will serve as an inspiration for researchers to delve deeper into the potential of LLMs in enhancing recommendation performance. Our codes, processed data and benchmark results are available at this https URL.
| Subjects: | Information Retrieval (cs.IR); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2308.12241 [cs.IR] |
| (or arXiv:2308.12241v2 [cs.IR] for this version) | |
| https://doi.org/10.48550/arXiv.2308.12241 arXiv-issued DOI via DataCite |
Submission history
From: Junling Liu [view email]
[v1]
Wed, 23 Aug 2023 16:32:54 UTC (3,233 KB)
[v2]
Tue, 13 Oct 2026 09:53:32 UTC (640 KB)
来源：arXiv：cs.AI · arxiv.org