arXiv：cs.LG· Julia Barth, Armin Sommer, Kunvar Thaman, Jannik Schilling·· 3 小时前
目标去哪了？目标泛化失败中策略对齐的三种不同失效形式
What Happens to the Goal? Distinct Failures of Policy Alignment in Goal Misgeneralization
AI 导读
研究在三个 Procgen 环境和五个机器人任务中发现，目标误泛化存在三种策略-目标错位形式：功能性目标替换、策略捷径，以及预期目标与代理目标线索之间的线索竞争。当目标信息能到达决策环节时，引导可将预期目标达成率从 16.0% 提升至 95.3%；当目标只到达策略而未到达决策时，仅靠新的下游连接可将达成率从 31.6% 恢复至 84.3%；目标完全惰性时两种方法均无效。
正文
Abstract:Goal misgeneralization is a common failure mode in which an agent behaves competently out of distribution but does not pursue the intended goal. However, we still have a weak understanding of what happens to the intended goal in misgeneralizing agents. Across three Procgen environments and five robot tasks, we find three distinct forms of policy-goal misalignment: (1) functional target substitution, where behavior follows an alternative goal, (2) a policy shortcut, where behavior follows a learned action strategy, and (3) cue competition between intended and proxy goal cues. We distinguish these regimes by tracing where goal information remains available internally, how strongly it influences the policy, and what role it plays in action selection, and find that the goal is lost at different depths. Where the goal reaches the decision, as in cue competition, steering redirects the policy to the goal's location, raising intended-goal reach from 16.0% to 95.3%. Where it reaches the policy but not the decision, only new downstream wiring recovers goal reach, from 31.6% to 84.3% in the environment with the action shortcut. Where it is inert, neither works. Our results show that goal misgeneralization is not a single failure of goal representation, but distinct failures of policy alignment at different depths, tightly connected to which recovery works.
| Comments: | 10 pages main text, 120 pages total. Code: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Robotics (cs.RO) |
| Cite as: | arXiv:2610.19731 [cs.LG] |
| (or arXiv:2610.19731v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19731 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Julia Barth [view email]
[v1]
Wed, 14 Oct 2026 07:52:24 UTC (8,556 KB)
来源：arXiv：cs.LG · arxiv.org