arXiv：cs.AI· Mantas Lukauskas, Viktorija \v{S}arkauskait\.e·· 6 小时前
LLM 作为合成问卷受访者的心理测量审计：看似合理但并不可靠
Plausible but Not Valid: A Psychometric Audit of LLMs as Synthetic Survey Respondents
AI 导读
研究用立陶宛组织心理学数据集（263 名员工、68 个条目、12 个子量表）对 37 个 LLM 进行心理测量审计，提出 Psychometric Similarity Score（PSS）并与五个统计基线、人类上限对比。
正文
Abstract:Large language models (LLMs) are increasingly used as synthetic survey respondents, but existing evaluations mostly ask whether individual answers look plausible. We ask a stricter, psychometric question: do LLMs preserve the joint distribution, latent structure, reliability, mediation pathways, and demographic effects of real human survey data? We introduce an open Lithuanian organisational-psychology dataset (n=263 employees; three validated instruments; 68 items, 12 subscales) and condition 37 LLMs on real respondent profiles under a persona-disclosure ladder, presentation, reasoning-effort and cross-language ablations, counterfactual demographic swaps, and a memorization probe. The resulting Psychometric Similarity Score (PSS) is anchored against five statistical baselines, a held-out human ceiling, and the real answers of the same respondents. LLMs reproduce the direction of human psychometric relationships, but statistical baselines with no language model match or beat them. The best LLM (PSS 0.81) is level with a Gaussian copula fit only on other respondents (0.82) and below the real answers of the same 100 people (0.89). A simple classifier separates every LLM sample from humans (median AUC 0.99), and verbatim recall of the items is rare (at most 4.7%) and unrelated to PSS. Counterfactual swaps move answers in the direction of human group differences and by a similar amount (role 0.49 and education 0.33 human SD, vs. 0.46 and 0.44 in humans). Gender swaps move them by 0.06 SD. Tucker's phi on UWES falls inside an item-permutation null for 10 of 37 models. Synthetic respondents compress the response scale (within-respondent SD 0.79 vs. 1.19), lose predictive validity on held-out humans (mean R^2=-0.14 vs. 0.28), and produce significant indirect effects on 4 of 7 mediation paths that are null in humans. LLM samples are therefore not a drop-in replacement for human survey data.
| Comments: | v2: corrects reverse-keyed item scoring for LLM answers, the distribution component of the statistical baselines, the acquiescence index, and the sign of paired differences in the counterfactual swaps (the role swap now has the largest effect), and drops three part-whole paths from the mediation negative control; all numbers, tables and figures recomputed |
| Subjects: | Computers and Society (cs.CY); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Applications (stat.AP) |
| Cite as: | arXiv:2608.14606 [cs.CY] |
| (or arXiv:2608.14606v2 [cs.CY] for this version) | |
| https://doi.org/10.48550/arXiv.2608.14606 arXiv-issued DOI via DataCite |
Submission history
From: Mantas Lukauskas [view email]
[v1]
Mon, 6 Jul 2026 17:11:16 UTC (216 KB)
[v2]
Sat, 10 Oct 2026 07:22:55 UTC (222 KB)
来源：arXiv：cs.AI · arxiv.org