面向陪伴型智能体评测的披露门控用户模拟方法

View PDF HTML (experimental)

Abstract:Playing the user with a large language model is standard in scalable evaluation, and has a repeatedly diagnosed failure: the simulated user is excessively cooperative, so a system under test can score by the number of questions it asks rather than by making the user willing to speak. We answer with a disclosure gate conditioning information release on the companion agent's behaviour: its state is a ladder of five ordered gates, merged onto three observable depth layers. We specify, ablate, and audit it. The gate is a load-bearing component of the environment: on a published companion-agent benchmark's English corpus, withdrawing per-example gate statements from training displaces the ranking of the 12 systems beyond the noise band of a seed re-run, while per-system scores show no detectable change. We state two acceptance criteria: a ranking must be order-preserving, and absolute scores must be scale-stable. Only one candidate passes both -- the simulator we release -- correlating at 0.993 with the published leaderboard. Prompting a frontier model instead barely moves the ranking while shifting every score up, invisible to ranking-only checks; we supply the specification, ablations, and downstream sensitivity analysis the benchmark's environment lacked.

Subjects:	Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC)
Cite as:	arXiv:2609.00982 [cs.CL]
	(or arXiv:2609.00982v2 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2609.00982 arXiv-issued DOI via DataCite

Submission history

From: Yao Liu [view email]
[v1] Tue, 1 Sep 2026 09:39:41 UTC (285 KB)
[v2] Mon, 12 Oct 2026 03:17:02 UTC (280 KB)