arXiv：cs.AI· Anahita Sharma, Apaar Shanker, Rishav Chakravarti, Sneha Srinivasan, Vijay Kalmath, Dylan Wilbur, Yuan Li, Sara Bolouki, Yuan Xue·· 16 小时前
CoverSynth：面向检索评估集的覆盖率引导生成方法
CoverSynth: Coverage-Guided Generation of Retrieval Evaluation Sets
AI 导读
CoverSynth 是一种覆盖率引导的检索评估集生成器，从无标注语料出发，针对检索需求与系统能力覆盖生成评估数据，用确定性检查定位证据引文、用模型检查评估语义有效性。
正文
Abstract:Enterprise agents need retrieval evaluation sets that expose domain-specific failures and guide developers towards efficient deployment tradeoffs. Neither expert-human-authored sets nor current synthetic generation methods provide a methodology for achieving the necessary coverage, reliability, efficiency, and discriminability required for such decision-making. Simultaneously, the field does not have an assessment framework for determining when an evaluation set has satisfied these properties. We introduce an evaluation set assessment framework for these criteria and CoverSynth -- a generator specifically designed to satisfy these criteria without incurring the high cost of human expert labeling. Starting from an unannotated corpus, CoverSynth targets coverage across retrieval demands and system capabilities. Deterministic checks locate evidence quotes in their sources, while model-based checks assess semantic validity to guarantee reliable ground truth. Across five public enterprise-style corpora, our method provides informative variation in difficulty, revealing unseen failure modes such as an inability of agents to handle underspecified or ambiguous requests. On the RepLiQA corpus, 91\% of human-generated queries can be answered by all members on a panel of 12 retrieval systems, compared to only 12\% of CoverSynth-generated queries. Finally, in a repeated TechQA/RepLiQA study, 12 of 12 within-harness model contrasts have positive nominal 95\% intervals on generated sets, compared with 4 of 12 on human sets. CoverSynth makes coverage and evidence quality explicit construction objectives, producing corpus-grounded evaluations that reveal retrieval headroom and differences between agent configurations.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.15409 [cs.AI] |
| (or arXiv:2610.15409v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15409 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Apaar Shanker [view email]
[v1]
Sun, 11 Oct 2026 07:26:00 UTC (2,201 KB)
来源：arXiv：cs.AI · arxiv.org