arXiv：cs.AI· Kaiyue Feng, Yilun Zhao, Yixin Liu, Tianyu Yang, Chen Zhao, John Sous, Arman Cohan·· 5 小时前
Physics：面向大学水平物理问题求解的基础模型基准测试
Physics: Benchmarking Foundation Models on University-Level Physics Problem Solving
AI 导读
研究团队推出 Physics 基准，包含 1034 道来自博士资格考试、覆盖六大学物理领域的开放式题目，均配有解答与独立核验的参考答案。他们评估了 53 个闭源与开源模型，在不含图表的测试题上，高准确率集中在少数 thinking 模型，最强开源模型已接近闭源水平，但多数开源模型解题数仍不足一半。对 Qwen3.5-9B 的分析显示其在物理知识应用和图表理解上存在失败，误读图表影响了多个子问题。
正文
Abstract:We introduce Physics, a benchmark for evaluating foundation models on advanced physics problem solving. It contains 1034 annotated, open-ended problems from PhD-qualifying exams across six core physics domains. Solving these problems requires applying domain-specific knowledge to physical setups and carrying out multi-step mathematical reasoning. Each problem includes a worked solution and independently checked reference answers. An LLM-based grading protocol evaluates individual answers and complete multi-part problems, supporting numeric, symbolic, and text responses. We evaluate 53 proprietary and open-weight models. On test problems without figures, high accuracy is concentrated in a small group of thinking models, with the strongest open-weight models approaching proprietary performance. Most open-weight models still solve fewer than half of these problems. Our analysis of Qwen3.5-9B reveals failures in applying physics knowledge and interpreting figures, with misread figures affecting several subquestions.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2503.21821 [cs.AI] |
| (or arXiv:2503.21821v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2503.21821 arXiv-issued DOI via DataCite |
|
| Journal reference: | Findings of ACL 2025 |
Submission history
From: Kaiyue Feng [view email]
[v1]
Wed, 26 Mar 2025 06:21:56 UTC (1,815 KB)
[v2]
Mon, 12 Oct 2026 18:17:30 UTC (364 KB)
来源：arXiv：cs.AI · arxiv.org