arXiv：cs.LG· Shiv Muthupandiyan, Tolga H. Dur, Mac Walker, Timo Flesch, Mahmoud Ghanem, Steph Suddell, Giles Harper-Donnelly, Sophie Rose, Jasper G\"otting, Coleman Breen, Jake Pencharz·· 10 小时前
ReproBAIT：编码智能体能否自主复现生物 AI 工具？
ReproBAIT: Can coding agents build their own biological AI tools?
AI 导读
研究团队提出 ReproBAIT 评测，在无互联网的沙箱环境中考察编码智能体能否根据有限描述复现已发表的生物 AI 工具。测试覆盖 ProteinMPNN、RFdiffusion3 及一款闭源分子设计工具，并设置方法检测预训练记忆。结果显示，编码智能体对部分工具具备较强复现能力，但对高度复杂工具能力有限，且存在程度不一的记忆迹象。
正文
Authors:Shiv Muthupandiyan, Tolga H. Dur, Mac Walker, Timo Flesch, Mahmoud Ghanem, Steph Suddell, Giles Harper-Donnelly, Sophie Rose, Jasper Götting, Coleman Breen, Jake Pencharz
Abstract:Large language models are increasingly competent at leveraging scientific knowledge and existing computational tools to execute complex workflows. At present, little information exists on their ability to reconstruct tools when given limited descriptions of such tools. We introduce an evaluation titled ReproBAIT (Reproducing Biological AI Tools), which measures the ability of coding agents to design, implement, and train biological AI tools taken from published research across a range of information access conditions. The coding agent is securely sandboxed without internet access and receives training data, a fixed time and compute budget, and limited information about the tool to replicate. We then score the reproduced biological AI tool based on its performance (tool-specific metrics) on held-out data. We evaluated a set of biological AI tools of varying complexities: ProteinMPNN, an inverse protein folding tool, RFdiffusion3, an all-atom generative diffusion tool, and a closed-source molecular design tool. We institute a methodology to check for memorisation, as the open source tools are likely to be included in models' pretraining corpus. Across a range of leading closed- and open-source coding agents, we find substantial reproduction capability for some selected tools, though limited capability for a highly complex tool. We find moderate evidence consistent with memorisation, varying across coding agents. These results suggest that tool developers need to be increasingly cautious when releasing information about the tools publicly; re-creation of the tool based on such information could undermine a tool's commercial value or invalidate dual-use managed access strategies.
| Subjects: | Artificial Intelligence (cs.AI); Computers and Society (cs.CY); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.19810 [cs.AI] |
| (or arXiv:2610.19810v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19810 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tolga H. Dur [view email]
[v1]
Wed, 14 Oct 2026 08:50:11 UTC (170 KB)
来源：arXiv：cs.LG · arxiv.org