arXiv：cs.CL· Andrew Perrault (Relantic, Inc., Columbus, OH, USA), Anmol Takiar (Relantic, Inc., Columbus, OH, USA), Dominik Winecki (Relantic, Inc., Columbus, OH, USA), Arnab Nandi (Relantic, Inc., Columbus, OH, USA)·· 16 小时前
RealityBench：用 Blender 模拟环境评测视觉语言模型记录人类体力工作的能力
RealityBench: Simulated Environments to Evaluate a Vision Language Model's Ability to Document Human Physical Work
AI 导读
RealityBench 在 Blender 中构建覆盖 11 种环境的模拟物理工作场景，用于评测视觉语言模型能否判断画面适合作工作凭证，并理解现场任务状态。对 26 个视觉语言模型的测试显示，顶级模型在答案以文字出现在画面中时基本能完成记录任务，但在读取测量值和判断小零件、空间关系上表现薄弱；最佳模型的可接受性判断仅相当于单个非作者标注者，成本与人工通过相当。
正文
Abstract:Augmenting worker capabilities in the physical world through guidance has been a decades-long goal for AI agents. One key supporting capability is to understand what the worker is doing, seeing through the human's camera or smart glasses, and assist them in documentation. Despite advances in vision language models, their capabilities in these settings remain uncertain and are challenging to benchmark because most physical work data is proprietary, and involves too much variability to study in a principled manner. Unlike structured settings such as kitchens and warehouses that have predictable lighting, geometry, and objects, field-facing tasks such as rooftop solar installation and vehicle repair are unconstrained and high variance. Here, we model real-world physical work in Blender across eleven environments, incorporating variants in both image acceptability and documentation. The acceptability task asks whether a particular view would be suitable proof-of-work and documentation asks whether an agent can understand the task-relevant state of the environment from that view. Across 26 vision-language models, we find that top models largely solve documentation when the answer is text in frame, but are weak at reading measurements and at judging small parts and spatial relations, such as whether a tape measure leans. Acceptability is harder to pin down because human raters disagree with each other. Measured against a scorer that agrees with the raters' majority about as often as a single rater does, the best models cannot be distinguished from a single non-author rater, at a cost within the range of a human pass. In the three environments where we can trace acceptability errors, most wrong rejections come from misreading the scene rather than the requirement. On the three documentation questions for which we have real images, model rankings on rendered frames transfer to real images.
| Comments: | 28 pages, 3 figures, 14 tables. Data: this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.14397 [cs.CV] |
| (or arXiv:2610.14397v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14397 arXiv-issued DOI via DataCite |
Submission history
From: Andrew Perrault [view email]
[v1]
Fri, 9 Oct 2026 23:41:20 UTC (4,422 KB)
[v2]
Wed, 14 Oct 2026 02:49:35 UTC (4,422 KB)
来源：arXiv：cs.CL · arxiv.org