arXiv：cs.CL· Filip Dorm, Leonora Vesterbacka·· 6 小时前AI 评分36
用四象限方法评估 Redpine Science
Leveraging a four-quadrant approach for evaluating Redpine Science
AI 导读
报告对 Redpine Science 进行了四项评估：在 ScholarQABench SciFact 上，配备 Redpine Science 的智能体正确回答 94.4% 的论断，无检索时为 87.6%；在专家验证问题集上，其陈述所需论断的比例为 80.1%，仅用网页搜索的智能体为 70.2%。
正文
Abstract:Redpine Science gives models and agents a single access point to a wide range of peer-reviewed literature, queried directly through the Model Context Protocol (MCP) and an API. This report evaluates Redpine Science on two levels: the relevance of the retrieved chunks, and a model's answer when it has access to Redpine Science compared to web search. Both public and expert-validated benchmarks are used. Public benchmarks are a widely accepted way to test model development and are comparable across labs, but risk saturation and memorization. To address this, we complement them with an expert-validated question set. In total, this report presents four evaluations. On ScholarQABench SciFact, the public answer-quality benchmark reported here, an agent with Redpine Science answers 94.4% of claims correctly against 87.6% with no retrieval. On the expert-validated question set, an agent with Redpine Science states 80.1% of the required claims against 70.2% for an agent restricted to web search. On the 668 queries of a public retrieval benchmark whose gold paper Redpine holds, stripped of any model reasoning, Redpine Science places the correct source paper in its top ten results for 83.1% of queries (Recall@10), against 79.3% for the benchmark's creator. A blinded expert relevance panel places Redpine Science's Precision@5 at 75.2% against 39.8% for the PubMed search tool. We release the expert-validated question set and instructions to reproduce every headline result above, at this https URL.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.07937 [cs.CL] |
| (or arXiv:2610.07937v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07937 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Leonora Vesterbacka [view email]
[v1]
Tue, 6 Oct 2026 08:11:30 UTC (88 KB)
来源：arXiv：cs.CL · arxiv.org