arXiv：cs.LG· Ruiqi Zhang, Jiahao Wang, Mingxuan Li, Haichen Luo, Chaoting Wang, Guoyu Mou, Keyu Lai, Hanchao Lv, Jiaxu Wang, Yibo Zheng, Aijun Yang, Xiaohua Wang·· 3 小时前AI 评分38
SimuVerity：面向工程级 Simulink 模型生成的智能体基准测试
SimuVerity: Benchmarking Agents for Engineering-Grade Simulink Model Generation
AI 导读
SimuVerity 是一个包含 101 个文本到可执行 Simulink 模型生成任务的基准，覆盖十个工程领域，用于评估智能体是否满足工程需求而非仅能编译或模仿参考模型。研究评估了六个智能体系统，最佳系统总分仅 42.86，表明结构相似度无法反映工程性能，部分高分模型仍存在严重的视觉布局混乱问题。
正文
Authors:Ruiqi Zhang, Jiahao Wang, Mingxuan Li, Haichen Luo, Chaoting Wang, Guoyu Mou, Keyu Lai, Hanchao Lv, Jiaxu Wang, Yibo Zheng, Aijun Yang, Xiaohua Wang
Abstract:Existing Simulink benchmarks mainly evaluate whether generated models compile, execute, or resemble a reference model. These criteria do not establish whether a model satisfies its engineering requirements. We introduce SimuVerity, a benchmark of 101 text-to-executable Simulink model-generation tasks across ten engineering domains. For each task, executable-system profiles ground the engineering specification and four families of native simulation scenarios. A hierarchical evaluator first checks artifact delivery, native executability, and engineering qualification, then scores qualified models across six dimensions covering accuracy, output quality, mechanistic fidelity, control and causal integrity, operating-domain robustness, and dynamic response. We evaluate six agent systems with SimuVerity. The best system achieves an overall score of only 42.86. The results show that structural similarity is a poor proxy for engineering performance: capability bottlenecks arise both in producing qualified implementations and in satisfying multidimensional requirements after qualification. Meanwhile, some high-scoring models still exhibit severe visual-layout disorder. SimuVerity provides a systematic basis for assessing agents' engineering capabilities and diagnosing failures in executable Simulink model generation.
| Comments: | 26 pages, 12 figures. Code and data are available at this https URL |
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.02304 [cs.SE] |
| (or arXiv:2610.02304v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2610.02304 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiahao Wang [view email]
[v1]
Thu, 1 Oct 2026 17:58:03 UTC (4,188 KB)
来源：arXiv：cs.LG · arxiv.org