arXiv：cs.LG· Etienne Casanova, Rafal Kocielnik, R. Michael Alvarez·· 5 小时前AI 评分57
研究提出 DSF 指标：LLM 内化先验影响标注任务表现，提示词纠错能力有限
On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance
AI 导读
ICML 2026 Oral 论文提出 Definition-Specific Familiarity（DSF）指标，在九个 LLM 和六个毒性数据集上发现 DSF 可预测零样本标注表现（partial r=+0.41）。
正文
Abstract:Large Language Models (LLMs) are increasingly used for zero-shot annotation and LLM-as-a-judge tasks, yet their reliability hinges on how model-internalized priors interact with user-provided instructions. We investigate three dimensions of this interaction: (1) how an LLM's familiarity with data and task definitions relates to performance, (2) whether additional information in prompts can correct zero-shot errors ("decision stickiness"), and (3) model susceptibility to misaligned task definitions. We introduce Definition-Specific Familiarity (DSF), which measures alignment between a model's elicited concept and the target definition. Across nine LLMs and six toxicity datasets (five primary datasets plus an additional robustness dataset), DSF predicts annotation performance after controlling for dataset identity (partial $r=+0.41$). This association remains positive across all prompting conditions tested. In contrast, three common text-memorization metrics show no positive association. We show that prompting has limited corrective power: only 34.8% of zero-shot errors are corrected by additional instructions or examples, with high-confidence errors especially persistent. Misaligned definitions systematically shift predictions without reducing reported confidence, making confidence unreliable for detecting definition-policy mismatch. Together, these findings establish definition alignment as a practical model-selection criterion and show that better prompting alone cannot substitute for validating model-policy fit.
| Comments: | Updated based on camera-ready from ICML 2026 (Oral & Spotlight); PMLR vol. 306. 9 pages, 5 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Machine Learning (cs.LG); Machine Learning (stat.ML) |
| MSC classes: | 68T50 |
| ACM classes: | I.2.7; I.2.6; K.4.0 |
| Cite as: | arXiv:2606.00467 [cs.CL] |
| (or arXiv:2606.00467v2 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.00467 arXiv-issued DOI via DataCite |
|
| Journal reference: | Proceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026 |
Submission history
From: Rafal Kocielnik [view email]
[v1]
Sat, 30 May 2026 01:21:14 UTC (842 KB)
[v2]
Fri, 2 Oct 2026 06:19:21 UTC (538 KB)
来源：arXiv：cs.LG · arxiv.org