arXiv：cs.AI· Hanqi Duan, Xiang Li·· 23 小时前
TADDLE：检测 LLM 生成同行评审缺陷的工具增强智能体
TADDLE: A Tool-Augmented Agent for Detecting Deficient LLM-Generated Peer Reviews
AI 导读
TADDLE 是用于检测 LLM 生成同行评审缺陷的工具增强智能体，并配套首个专家标注基准，包含 50 篇 ICLR 2025 论文的 1,800 条评审，由 18 位领域专家按六类缺陷（另加非缺陷标签）进行多标签标注。
正文
Abstract:LLM-generated peer reviews are increasingly common at major venues, yet their deficiencies are hard to detect because they are uniformly fluent and well-structured. Existing work either classifies authorship without judging quality, or scores quality with features designed for human-written reviews; no prior system detects deficiencies in LLM-generated reviews at the level of individual defect types. To bridge the gap, we introduce TADDLE, a Tool-Augmented Agent for Detecting Deficient LLM-Generated Peer Reviews, together with the first expert-annotated benchmark for this task. Our benchmark comprises 1,800 reviews on 50 ICLR 2025 papers, multi-label-annotated by 18 domain experts against a taxonomy of six defect categories (plus a non-deficient label). TADDLE decomposes detection into four specialized analysis tools -- Verify, Correct, Complete, and Transform -- orchestrated by an agent; an integrator synthesizes their outputs into binary and multi-label classifications via two-stage semi-supervised learning. Experiments show that TADDLE performs strongly on binary detection and the multi-label classification task. We release the benchmark and code at this https URL.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.26911 [cs.AI] |
| (or arXiv:2605.26911v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.26911 arXiv-issued DOI via DataCite |
Submission history
From: Hanqi Duan [view email]
[v1]
Tue, 26 May 2026 12:07:44 UTC (1,506 KB)
[v2]
Tue, 13 Oct 2026 11:11:04 UTC (446 KB)
来源：arXiv：cs.AI · arxiv.org