arXiv：cs.AI· Shawheen Naderi, Fatemeh Akherati, Yubei Chen·· 4 小时前
可视化 PPO 智能体价值函数中的世界模型
Visualizing the World Model in an Agent's Value Function
AI 导读
研究通过对 Crafter 环境中 Proximal Policy Optimization 智能体的价值函数做反事实干预，发现其虽仅以优化奖励为目标，却学到了钻石何时可采集、水和食物影响健康、库存相关启发式等概念。作者还讨论了这些可视化带来的安全启示，以及当前强化学习智能体阻碍构建真正高级机器智能的一些缺陷。
正文
Abstract:Could model-free reinforcement learning algorithms actually be learning a model of the environment? In this work, we apply counterfactual interventions which hold everything about a state fixed, besides a targeted change, to explore if the value function of a Proximal Policy Optimization agent forms a world model of the Crafter environment. Despite only being trained to optimize reward, we discover that the value function learns concepts such as the rules governing when a diamond is collectible, water and food having an impact on health, heuristics regarding inventory, and more. Finally, we discuss the safety implications of our visualizations and some shortcomings of current reinforcement learning agents that will prevent us from building truly advanced machine intelligence.
| Comments: | 20 pages, 15 figues, 3 tables |
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.14186 [cs.AI] |
| (or arXiv:2610.14186v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14186 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shawheen Naderi [view email]
[v1]
Fri, 9 Oct 2026 18:01:55 UTC (11,537 KB)
来源：arXiv：cs.AI · arxiv.org