arXiv：cs.AI· Zhenyu Zhao, Nanshan Jia, Jihyeon Je, Yifu Tang, Alvin Chan, Michael Spedden, Michael V. Palleschi, Sui Huang, Jingshen Wang, Zeyu Zheng·· 6 小时前AI 评分34
3D-DefectBench：面向细粒度 3D 生成缺陷的 VLM 评估流水线受控析因研究
3D-DefectBench: A Controlled Factorial Study of Vision-Language Model Evaluation Pipelines for Fine-Grained 3D Generation Defects
AI 导读
研究者提出 3D-DefectBench，一个针对 3D 生成评估流水线的大规模基准，用九个覆盖几何、纹理与提示词遵循的细粒度二分类缺陷替代纯整体评分，并保留可选的人工严重度标注。
正文
Abstract:Automated evaluation is essential for scaling generative 3D systems, where exhaustive human review is costly and slow. Yet the reliability of an automated judge depends on the full evaluation pipeline, including the vision-language model (VLM), asset rendering, visual evidence, task specification, and human reference labels. We introduce 3D-DefectBench, a large-scale benchmark for rigorous evaluation-pipeline analysis. It complements holistic ratings and pairwise preferences with nine fine-grained binary defects spanning geometry, texture, and prompt adherence, with optional human severity annotations. Using a balanced factorial design, we vary the VLM, camera protocol, visual input, and prompt schema across 84 inference designs, and validate the resulting conclusions on a broader set of frontier models. Model choice is the dominant source of variation in agreement with human labels, while other pipeline factors also influence agreement, interact with the model, and can alter the best configuration. A compact six-view RGB protocol performs comparably to denser view sets and configurations augmented with depth or normal channels, making it a strong cost-effective default. Under this fixed design, the best of 12 VLMs still trail trained human labelers, and texture agreement drops sharply from expert-agreement to noisier silver labels. Severity annotations further show that binary judges recover most defects humans flag as severe. These results highlight the importance of evaluating automated judges as complete pipelines and calibrating them across human reference regimes.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Graphics (cs.GR) |
| Cite as: | arXiv:2607.10826 [cs.CV] |
| (or arXiv:2607.10826v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2607.10826 arXiv-issued DOI via DataCite |
Submission history
From: Zhenyu Zhao [view email]
[v1]
Sun, 12 Jul 2026 16:41:24 UTC (11,928 KB)
[v2]
Tue, 6 Oct 2026 05:25:52 UTC (14,150 KB)
来源：arXiv：cs.AI · arxiv.org