arXiv：cs.AI· Stephane Hatgis-Kessell, Myra Cheng, Xiaoxuan Hou, Qian Hu, Rahul Gupta, Natasha Jaques, Emma Brunskill·· 6 小时前AI 评分46
通过多元偏好优化缓解社交谄媚
Mitigating Social Sycophancy via Pluralistic Preference Optimization
AI 导读
研究者提出多元偏好优化（PlurPO），让语言模型在人际冲突场景中识别并模拟相关利益方，训练其生成各方都能接受的回复，从而缓解社交谄媚。在四个数据集和四个模型家族上，PlurPO 对意图伤害类陈述的认可率平均降低 89%；在一般建议问题上将人类认可率差距从 17.8% 缩小至 8.0%。为 8B 模型构建的偏好数据集还能有效迁移到 32B 模型。
正文
Abstract:Personal advice, including relationship advice, now ranks among the most common uses of generative AI. But language models (LMs) exhibit sycophancy: they affirm users much more often than humans do, which can make people overconfident and less willing to repair their relationships after a conflict. Prior work on mitigating sycophancy has focused on factual settings where a response can be checked against a ground truth answer, while mitigations for social sycophancy (e.g., personal advice, where there is no ground truth) have relied on simple prompting and post-training methods with limited effectiveness. Our insight is that social sycophancy occurs in part because LMs overly center on the user and fail to consider the perspectives of other stakeholders impacted by the user's behavior. To address this problem we propose Pluralistic Preference Optimization (PlurPO): given inputs describing interpersonal conflicts, the LM identifies and simulates the relevant stakeholders, and is then trained to prefer and generate responses acceptable to all stakeholders. PlurPO uses only signals the model produces about its own outputs, without ground-truth labels. PlurPO substantially reduces social sycophancy across four datasets and four model families compared to prior methods. For example, on statements of intent to cause harm, where the users' actions should not be endorsed, PlurPO reduces the endorsement rate by 89% on average across four models. On general advice questions, where the target is to match the endorsement rate of human responses, it closes the gap by more than half, from 17.8% to 8.0% on average. The preference dataset constructed by PlurPO for an 8B model also effectively transfers to mitigating sycophancy in a larger (32B) model. Our results indicate that social sycophancy can be reduced by leveraging a model's own capabilities to simulate a plurality of relevant perspectives.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.02568 [cs.AI] |
| (or arXiv:2610.02568v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.02568 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Stephane Hatgis-Kessell [view email]
[v1]
Thu, 1 Oct 2026 23:00:49 UTC (8,047 KB)
来源：arXiv：cs.AI · arxiv.org