arXiv：cs.AI· Dayan Pan, Jingyuan Wang, Xie Yu·· 6 小时前AI 评分36
DyPAM：面向大语言模型参数高效微调的动态位置注意力调制
Dynamic Positional Attention Modulation for Parameter-Efficient Fine-Tuning of Large Language Models
AI 导读
研究者提出 PEFT 方法 DyPAM（Dynamic Positional Attention Modulation），直接作用于 query 和 key 表示，通过输入条件化的维度级调制结合头级、层级结构调制，在不修改预训练主干的前提下适配 RoPE 引起的维度相关位置结构。在多个主干模型的数学与常识推理基准上，DyPAM 持续优于现有强 PEFT 基线，论文已被 KDD 2026 接收。
正文
Abstract:Parameter-efficient fine-tuning (PEFT) has become a standard approach for adapting large language models to downstream tasks. However, most existing PEFT methods rely on uniform and static adaptations, without accounting for the structured heterogeneity of attention across dimensions, heads, layers, and input tokens. In practice, attention representations exhibit non-uniform behavior, and positional encoding mechanisms such as rotary positional embeddings (RoPE) induce dimension-dependent positional structure, making uniform adaptation suboptimal. In this work, we propose DyPAM (Dynamic Positional Attention Modulation), a PEFT method that adapts how positional information contributes to attention by operating directly on the query and key representations. DyPAM combines input-conditioned, dimension-wise modulation with head-wise and layer-wise structural modulation, performing fine-grained adaptation of positional attention aligned with the RoPE-induced structure without modifying the pretrained backbone. Extensive experiments on mathematical and commonsense reasoning benchmarks across multiple backbone models demonstrate that DyPAM consistently outperforms existing strong PEFT baselines.
| Comments: | Accepted by KDD 2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.07848 [cs.CL] |
| (or arXiv:2610.07848v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.07848 arXiv-issued DOI via DataCite (pending registration) |
|
| Journal reference: | Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD '26), 2026 |
| Related DOI: | https://doi.org/10.1145/3770855.3817911
DOI(s) linking to related resources |
Submission history
From: Dayan Pan [view email]
[v1]
Tue, 6 Oct 2026 06:55:38 UTC (2,491 KB)
来源：arXiv：cs.AI · arxiv.org