arXiv：cs.LG· Amine Andam, Jamal Bentahar, Mustapha Hedabou·· 4 小时前
正则化鲁棒强化学习的统一约束视角
A Unified and Constrained View of Regularization-Based Robust Reinforcement Learning
AI 导读
该论文为基于正则化的鲁棒强化学习提出统一理论框架，推导出名义策略与最坏情况策略性能差距的新上界，每个上界由现有正则化目标加上两者间的 KL 散度惩罚构成，解释了 KL 惩罚提升鲁棒性的原因。基于这些上界，作者将鲁棒训练建模为约束优化问题，指出现有方法对应固定 Lagrange 乘子的特例，并让乘子与策略联合更新以自动调节正则化权重。多个连续控制任务的对抗评测验证了理论分析。
正文
Abstract:Regularization-based methods have become a standard approach for training Deep Reinforcement Learning policies against adversarial input perturbations. In this paper, we unify these methods by deriving new upper bounds on the performance gap between the nominal and worst-case policies. Each upper bound is expressed as an existing regularization objective plus a KL-divergence penalty between the nominal and worst-case policies, which further explains why adding a KL penalty improves robustness in practice. Building on these bounds, we formulate robust training as a constrained optimization problem, showing that existing methods correspond to the special case of a fixed Lagrange multiplier. We instead update the multiplier jointly with the policy to automatically tune the regularization weight. Finally, we conduct extensive adversarial evaluations across several continuous control tasks to validate our theoretical analysis.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2609.13050 [cs.LG] |
| (or arXiv:2609.13050v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.13050 arXiv-issued DOI via DataCite |
Submission history
From: Amine Andam [view email]
[v1]
Fri, 11 Sep 2026 16:45:36 UTC (324 KB)
[v2]
Wed, 14 Oct 2026 10:47:25 UTC (376 KB)
来源：arXiv：cs.LG · arxiv.org