arXiv：cs.AI· Mikhail Aleksandrov, Nail Bashirov, Maksim Ivanov, Andrew Tsukanov, Timofey Karyagin, Alexander Miasnikov, Gleb Molodtsov, Valentin Samokhin, Aleksandr Beznosikov·· 22 小时前
并非所有漂移都是机制：追踪 LLM 中的提示词适配
Not All Drift Is a Mechanism: Tracing Prompt Adaptation in LLMs
AI 导读
研究对比了离散提示词优化、prompt tuning 与 prefix tuning 对稠密模型和 MoE 语言模型内部计算的影响：优化文本主要改变回答构建，输入级提示词将影响写入内容表征，逐层 prefix 则通过注意力维持直接控制。
正文
Abstract:Prompt-based adaptation offers several ways to specialize a frozen language model, ranging from natural-language prompt optimization to learned continuous prompts. These methods are usually compared through downstream performance, although similar outputs need not arise from similar internal computations. We study how discrete prompt optimization, prompt tuning, and prefix tuning alter the processing of dense and mixture-of-experts language models. Our analysis shows that the location of the adaptation predicts how task information is carried to the output: optimized text primarily changes response construction, input-level prompts write their influence into content representations, and layer-wise prefixes maintain direct control through attention. In MoE models, these differences further induce distinct expert-routing patterns. At the same time, common representation-level measurements respond strongly to generic context changes and do not reliably distinguish useful adaptations from matched placebos. We argue that understanding prompt adaptation requires separating representation drift from the part of that drift that is aligned with, and causally used by, the model's task computation. This perspective provides practical guidance for selecting, evaluating, and interpreting prompt adaptation methods.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.18412 [cs.LG] |
| (or arXiv:2610.18412v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.18412 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Gleb Molodtsov Mr [view email]
[v1]
Tue, 13 Oct 2026 13:52:52 UTC (562 KB)
来源：arXiv：cs.AI · arxiv.org