arXiv：cs.CL· Xinyi Liu, Hooshang Nayyeri, Emine Yilmaz, Dilek Hakkani-Tur, Charith Peris, Hari Thadakamalla·· 7 小时前
PersEvalBench：LLM 评估器能判断人设信息的语境适用性吗？
PersEvalBench: Can LLM Evaluators Judge Contextually Appropriate Use of Persona Information?
AI 导读
研究者推出 PersEvalBench，用 403 个多轮情景测试 LLM 评估器能否识别用户历史信息该延续、迁移还是失效，覆盖欠个性化、过度个性化和状态误用三类共九种错误子类型。在五种 LLM 评估器上，即使提供完整对话历史，受控个性化错误的识别率仍然有限：人类能识别 94.4% 的欠个性化错误，最强评估器仅 63.9%，且常把方向判反。高亮相关历史轮次仅带来小幅提升。
正文
Abstract:Large language model (LLM) evaluators are increasingly used to assess personalized assistants, yet whether they can reliably judge personalization has rarely been tested. We introduce PersEvalBench, a benchmark testing whether an evaluator can recognize appropriate and inappropriate uses of user history. It contains 403 multi-episode scenarios where user information should persist, transfer, or cease to apply. The benchmark covers nine controlled error subtypes across three families: under-personalization, over-personalization, and misapplied-state errors. Human audits confirm that generated errors match their intended types and that the prescribed uses of user information are reasonable.
Across five LLM evaluators, recognition of controlled personalization errors remains limited even with the full dialogue history. On the human-evaluated subset, under-personalization shows the largest gap: humans recognize 94.4% of these errors, compared with 63.9% for the strongest reported evaluator. Evaluators often recognize that a response is problematic but reverse the error direction, mistaking failure to follow an applicable preference (UNDER) for use of an inapplicable preference (OVER). Easier evidence access offers little help: highlighting the relevant history turns raises error recognition only slightly, suggesting that the remaining difficulty lies in judging what role that information should play in the response.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19314 [cs.CL] |
| (or arXiv:2610.19314v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19314 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xinyi Liu [view email]
[v1]
Tue, 13 Oct 2026 23:44:58 UTC (922 KB)
来源：arXiv：cs.CL · arxiv.org