arXiv：cs.CL· Yilin Jiang, Fei Tan, Xuanyu Yin, Guanchu Wang, Xia Hu, Yutao Yue·· 7 小时前
LLM Agent 回复可用性研究：54% 的回复无法使用，如何降低
Measuring and Reducing Unusable Replies of LLM Agents
AI 导读
研究提出"回复可用性"（reply usability）指标，用 reader model 从请求者视角判断 Agent 回复能否被理解和使用，其与五名标注者多数意见的一致性落在单个标注者区间内。
正文
Abstract:People delegate multi-step work to LLM agents and read only the reply that the agent hands back. Current evaluation scores task completion and the faithfulness of reports, and it leaves open whether the requester can understand and use the reply. We call this property reply usability and measure it with a reader model that takes the requester's position and tests whether the reply can be interpreted from what the requester has seen, is in the requester's language and answers the request. Its agreement with the majority of five annotators lies within the range of the individual annotators. Across eight models from seven families, 54% of 480 replies fail, and the best model still fails on 30%. Failed replies mostly point to material the requester has not seen or use labels that the agent coined during its work. Stating explicit requirements for the reply reduces the failure rate from 0.52 to 0.43 together with errors and omissions, and a secondary model that rewrites the reply reduces it to 0.36, which human annotators confirm in a blind comparison. Reply usability can thus be scored alongside task success and improved with one appended message or one call to a secondary model.
| Comments: | 17 pages, 4 figures, 12 tables |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Human-Computer Interaction (cs.HC) |
| Cite as: | arXiv:2610.19659 [cs.CL] |
| (or arXiv:2610.19659v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19659 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yilin Jiang [view email]
[v1]
Wed, 14 Oct 2026 06:35:49 UTC (168 KB)
来源：arXiv：cs.CL · arxiv.org