arXiv：cs.AI（全量分类）· Katrina Drozdov, Oliver Chen, Langston Nashold, Rayan Krishnan·· 5 小时前AI 评分48
Legal Research Bench：衡量长时程法律研究智能体的端到端可靠性
Legal Research Bench: Measuring End-to-End Reliability in Long-Horizon Legal Research Agents
AI 导读
研究者推出 Legal Research Bench（LRB），一个由专家编写的 413 道美国法律研究开放题基准，每题配有标准答案、支撑权威和二元评分标准。在带网页搜索、判例检索等工具的测试框架下，13 个前沿模型表现最好的 Claude Opus 4.8 全对率仅 42.9%。全对率因法律领域而异，需协调冲突权威的问题更低，且更多轮次、工具调用和推理成本并不能预测更高准确率。
正文
Abstract:Legal research is a core and time-consuming legal workflow. Lawyers must identify controlling authority, verify that it remains valid, reconcile statutes and cases, and synthesize a grounded answer. Language model agents are a natural fit for this retrieval-intensive workflow, and automating even part of it would be valuable. But that value depends on reliability: a single missing authority, stale citation, or wrong legal conclusion can make an otherwise plausible answer unusable. We introduce \textbf{Legal Research Bench} (LRB), a benchmark of 413 open-ended U.S. legal research questions written by experts, each paired with a gold answer, supporting authorities, and a binary grading rubric. We evaluate thirteen frontier models in a harness with web search, case-law search, page parsing, and retrieval tools. We score agent responses through all-pass grading with source verification, where a response is correct only if every required criterion is satisfied and its cited authorities verify. We also validate the LLM judge against expert attorneys ensuring that benchmark scores track attorney judgment. Agents remain far from reliable: among the models we tested, the strongest, Claude Opus 4.8, is fully correct on 42.9\% of questions. Performance also varies substantially by task setting: all-pass rates differ across areas of law and are lower on questions requiring reconciliation of conflicting authorities. Across models, more turns, tool calls, and inference cost do not predict higher accuracy.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Computers and Society (cs.CY) |
| Cite as: | arXiv:2610.00609 [cs.AI] |
| (or arXiv:2610.00609v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.00609 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Katrina Drozdov (Evtimova) [view email]
[v1]
Wed, 30 Sep 2026 19:13:16 UTC (238 KB)
来源：arXiv：cs.AI（全量分类） · arxiv.org