arXiv：cs.CL· Yihan Li, Hanyi Zhang, Xiaoxi Jiang, Man Guo·· 3 小时前AI 评分40
Goldsmith：基于 gold-loss 引导的定义优化与智能体标注框架
Goldsmith: Gold-Loss-Guided Definition Optimization with an Agentic Annotation Harness
AI 导读
Goldsmith 是一个智能体流水线，将专家标注的小规模 gold set 转化为可复用的结构化标注定义，并把该定义当作可训练的文本对象来优化。候选定义在相同 gold 样本上运行并以可执行结构化损失打分，LLM 编辑器将高损失失败转为文本梯度修订，仅在损失下降时接受。
正文
Abstract:Many annotation projects begin before experts have a stable guideline or enough labels to train a task-specific model. We present Goldsmith, an agentic pipeline that turns a small gold set---expert-annotated calibration examples representing the intended task boundaries---into a reusable structured annotation definition. Goldsmith treats this definition as a trainable textual object. Candidate definitions are run on the same gold examples and scored with an executable structured loss, while the output schema, formatting, retrieval, repair, judging, and human review remain in an external harness. A large language model (LLM) editor converts the highest-loss failures into textual-gradient revisions, which are accepted only when the measured loss decreases. In prompt-optimization comparisons, Goldsmith improves over direct rewriting, OPRO, APE, and PromptBreeder under matched evaluation protocols. The resulting definition also improves downstream annotation when combined with retrieval, score-based routing, and human review across typed span, pair-level relation, and fixed-trigger event-argument tasks. These results show that scarce expert supervision can support both task-definition learning and scalable annotation.
| Comments: | 20 pages, 4 figures, 11 tables. Accepted to the main conference of EMNLP 2026 |
| Subjects: | Computation and Language (cs.CL) |
| ACM classes: | I.2.7 |
| Cite as: | arXiv:2610.09489 [cs.CL] |
| (or arXiv:2610.09489v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.09489 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yihan Li [view email]
[v1]
Wed, 7 Oct 2026 05:45:25 UTC (548 KB)
来源：arXiv：cs.CL · arxiv.org