arXiv：cs.LG· Ruomeng Ding, Tianwei Gao, Lianrui Geng, Mohit Bansal, Zhun Deng·· 3 小时前
WAPA：面向多元对齐的福利感知偏好采集方法
Who Should We Listen to More? Welfare-Aware Preference Acquisition for Pluralistic Alignment
AI 导读
研究提出 Welfare-Aware Preference Acquisition（WAPA），用辅助参考模型估计每个查询的边际福利增益与个体监督效用，在固定查询预算下按估计增益选择查询并证明其精确最优性。
正文
Abstract:Pluralistic alignment needs to accommodate diverse preferences under limited human feedback. Using AI simulators to label preference comparisons reduces annotation costs, but variation in simulation accuracy across individuals can leave some people's preferences poorly represented in training. We study how to allocate human feedback across people to construct training data for pluralistic alignment, and how this allocation affects the trained policy. To make this allocation explicit and controllable, we define a social welfare objective over individual supervision utilities that measure label quality. Each query's marginal welfare gain depends on its expected supervision improvement, the recipient's current utility, and application-designated priority. Welfare-Aware Preference Acquisition (WAPA) uses auxiliary reference models to estimate query gains and individual utilities, then selects queries by estimated marginal welfare gain while updating recipient utilities. We prove exact optimality of the acquisition algorithm for the estimated welfare objective under a fixed query budget and model predictions. Experiments on six Roleplay collections and four OpinionQA regions demonstrate improved supervision and controllable allocation tradeoffs. Policy training on four Roleplay collections improves preference prediction for both seen and unseen individuals. When prioritizing a designated group and poorly represented individuals, acquiring human labels for 20% of comparisons recovers 75-86% of the average predictive utility gap between policies trained entirely on AI-generated labels and entirely on ground-truth labels.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19035 [cs.LG] |
| (or arXiv:2610.19035v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19035 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ruomeng Ding [view email]
[v1]
Tue, 13 Oct 2026 18:27:17 UTC (2,128 KB)
来源：arXiv：cs.LG · arxiv.org