arXiv：cs.CL· Wenqiang Zhu, Xinping Lei, Zhiyi Lai, Zhengshuo Gong, Tianzhuang He, Songwei Yu, Weihao Li, Jingxuan Xu, Zihang Xu, Gang Wu, Zhongqiu Wang, Yuhui Bai, Jie Li, Zhao Liu, Junhao Fan, Yan Wu, Yanan Wu, Jiaheng Liu, Quan Gu, Ariel Deng·· 16 小时前
ANCHOR：评估仓库级智能体编码中的自然约束层级与顺序
ANCHOR: Assessing Naturalistic Constraint Hierarchies and Ordering in Repository-Grounded Agentic Coding
AI 导读
研究者提出 ANCHOR 基准，用于评估仓库级编码智能体的"工程约束遵循"能力，覆盖八种语言组的自然多语言仓库任务，包含 10 类 78 种叶类型的约束分类体系。
正文
Authors:Wenqiang Zhu, Xinping Lei, Zhiyi Lai, Zhengshuo Gong, Tianzhuang He, Songwei Yu, Weihao Li, Jingxuan Xu, Zihang Xu, Gang Wu, Zhongqiu Wang, Yuhui Bai, Jie Li, Zhao Liu, Junhao Fan, Yan Wu, Yanan Wu, Jiaheng Liu, Quan Gu, Ariel Deng
Abstract:For repository-grounded coding agents, a functionally correct patch is not necessarily an acceptable one: developers also constrain how the task must be carried out-modification scope, workflow order, validation, tool use, and reporting. This capability, which we call engineering constraint following, remains largely invisible to current evaluations. We introduce ANCHOR (Assessing Naturalistic Constraint Hierarchies and Ordering in Repository-Grounded Agentic Coding), a benchmark built on naturalistic, multilingual repository tasks spanning eight language groups. ANCHOR combines (i) a hierarchical taxonomy of 10 categories and 78 leaf types distilled from real developer interactions, (ii) an automated pipeline that instantiates task-grounded atomic constraints and composes them into typed AND/THEN/PRIORITY relations, and (iii) a trajectory-grounded evaluation framework that checks both the code diff and the execution trajectory with rule-based and LLM judges. Across 19 agent configurations spanning 2 scaffolds and 10 foundation models on 358 tasks, the strongest measured system, Claude Opus 4.8 under Cline, reaches 70.1% atomic compliance and 63.7% relation compliance, while 19.0% of tasks pass every scored checklist item under the strict all-item FCCR criterion. A five-engineer audit of constraints from 50 instances reports 94% feasibility, 94% repository consistency, and 96% realism. In the primary paired set (14 configurations), the mean signed change in task completion after adding constraints is +0.7 percentage points; individual changes vary in both directions. These results distinguish functional success from constraint compliance, with failures concentrated in workflow ordering, tool-use discipline, and reporting, and substantial interactions between models and scaffolds.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.16859 [cs.CL] |
| (or arXiv:2610.16859v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.16859 arXiv-issued DOI via DataCite |
Submission history
From: Ken Deng [view email]
[v1]
Mon, 12 Oct 2026 14:28:38 UTC (2,446 KB)
[v2]
Wed, 14 Oct 2026 06:22:34 UTC (13,509 KB)
来源：arXiv：cs.CL · arxiv.org