arXiv：cs.LG· Yanjun Chen, Jinghan Wang, Xiaoyu Shen, Wenjie Li, Wei Zhang·· 4 小时前AI 评分34
动作塑形：策略只吸收其能表达的东西
Action Shaping: Policies Absorb What They Can Express
AI 导读
论文提出"动作塑形"概念：在动作通道上加偏移并在部署时移除的做法此前没有理论保证，作者证明可训练策略能吸收其输出层可精确复现的偏移，且移除后回报不变。最小实例是一个零初始化线性头加可学习门控，门控会自行先升后降，在 20 个任务上移除该头几乎无损失；非线性头参数更多却无法被吸收。偏移幅度还可在移除前预示损失大小。
正文
Abstract:Reward shaping has a theorem: a potential-based term can be removed without changing the optimal policy. The same practice on the action channel, an offset added in training and dropped at deployment, has no theorem. Nothing cancels an action offset, so the correction is kept at deployment or removed without a guarantee. We call it action shaping and state its principle. A trainable policy absorbs an offset its own output layer can reproduce exactly, which is what we mean by express; what is absorbed can be removed with the return intact. Its minimal instance is a zero-initialized linear head behind a learnable gate, added to an actor that trains through a learned action-value function, with no penalty or schedule. The gate rises and then falls on its own, for deterministic and stochastic actors alike, and on 20 tasks removing the head costs almost nothing. The condition is exact reproduction, not capacity: a nonlinear head with more parameters is not absorbed, and in a paired control, one linear path added to a nonlinear base head restores absorption. Exact reproduction gives the loss a flat direction that gradient noise drifts along, and the offset's amplitude indicates, before removal, what dropping the head will cost. Action shaping thus gains the counterpart of the shaping theorem, a condition for absorption, together with the mechanism behind it and a diagnostic that reads it. Policies absorb what they can express, and only that.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2609.32752 [cs.AI] |
| (or arXiv:2609.32752v2 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2609.32752 arXiv-issued DOI via DataCite |
Submission history
From: Yanjun Chen [view email]
[v1]
Sat, 26 Sep 2026 16:18:55 UTC (781 KB)
[v2]
Tue, 6 Oct 2026 15:32:07 UTC (777 KB)
来源：arXiv：cs.LG · arxiv.org