arXiv：cs.CL· Florian Kutzner, Celina Kacperski, Laura de Moli\`ere, Edoardo Chidichimo, Min Jun Jung, Felix P. S. Wallis, James K. He·· 5 小时前AI 评分40
LLM 合成调研受访者如何验证？论文提出兼顾后果行为与表征公平的验证框架
Consequential Behaviour and Representational Fairness in the Validation of Synthetic Research
AI 导读
针对用 LLM 驱动的合成受访者替代人类样本的研究，arXiv 论文指出当前与人类调查的临时对比在多数应用场景中验证了错误的对象。作者提出一个验证框架，要求每项效度声明说明与人类数据的对应程度及所涉及的 location、dispersion、response process、structure 四类诊断，并须为子群体单独报告效度。
正文
Abstract:Researchers in industry and academia use synthetic survey respondents powered by large language models as substitutes for human samples. These synthetic populations require validation against real-world data, so researchers often address them using ad hoc comparisons with human surveys. Inspired by the intention-behaviour gap in behavioural science, we argue that these validations test the wrong thing for most applied cases where decision makers commission synthetic research to anticipate consequential behaviour. To address this problem, we propose a validation framework with two requirements. First, every validity claim must state its level of correspondence with human data: does the sample predict what the represented people do, which of four diagnostics (location, dispersion, response process and structure) does the validation address, and does the validation compare against experimental effects? Second, researchers must report validity claims for subgroups, since these groups are often the most affected by consequential decisions and aggregate accuracy hides their misrepresentation. Our validation framework operationalises three justice dimensions (distributional, procedural, and recognition) as measurable quantities and treats within-persona counterfactual experiments as a design that itself requires validation. We then apply the framework to electric vehicle charging tariffs, before closing with a reporting checklist that researchers can use to make convincing validity claims.
| Comments: | 19 pages, 1 figure |
| Subjects: | Computation and Language (cs.CL); Computers and Society (cs.CY) |
| Cite as: | arXiv:2609.27690 [cs.CL] |
| (or arXiv:2609.27690v3 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2609.27690 arXiv-issued DOI via DataCite |
Submission history
From: Edoardo Chidichimo [view email]
[v1]
Wed, 23 Sep 2026 11:10:16 UTC (75 KB)
[v2]
Thu, 24 Sep 2026 10:09:06 UTC (75 KB)
[v3]
Tue, 6 Oct 2026 15:58:16 UTC (93 KB)
来源：arXiv：cs.CL · arxiv.org