arXiv：cs.CL· Anjali Kantharuban, Jonas Mueller·· 4 小时前AI 评分44
CUE：面向多轮基准测试的校准用户嵌入框架
CUEing User Simulators: Calibrated User Embeddings for Multi-Turn Benchmarking
AI 导读
研究者提出 Calibrated User Embeddings（CUE）框架，通过编码真实会话并采样连续表示、再解码为 persona 指令来驱动 LLM 扮演用户模拟器，无需训练。
正文
Abstract:Recent benchmarks rely on user simulators to evaluate AI agents in multi-turn interaction. While existing simulation techniques demonstrate surface fidelity to human style and behavior, ecologically valid interactive benchmarking also requires alignment in when and how agents fail across simulated and real user populations. We find that existing simulators lack outcome calibration: agreement with observed success rates and failure patterns when real users interact with the same agent. We introduce Calibrated User Embeddings (CUE), a framework that both encodes observed sessions and samples continuous representations, then decodes them into persona commands to steer LLMs to act as user simulators without training. Through this, we evaluate user-conditioned replay of past sessions and aggregate metric agreement when sampling novel personas for the same tasks. On $\tau^2$-Bench, CUEd simulators commit fewer simulator-attributed errors and more faithfully reproduce real-user agent failure modes, aggregate success rates, and outcomes for specific task-user pairs than other persona-based simulation methods. These gains coexist with competitive user fidelity as measured using metrics established in prior work. After being fit to mostly customer support interactions, the same CUEd simulators generalize to document creation, math tutoring, and casual conversation, and remain effective across different simulator LLMs without CUE retraining.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.02460 [cs.CL] |
| (or arXiv:2610.02460v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.02460 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Anjali Kantharuban [view email]
[v1]
Thu, 1 Oct 2026 20:34:16 UTC (585 KB)
来源：arXiv：cs.CL · arxiv.org