arXiv：cs.LG· Abhisek Keshari·· 5 小时前AI 评分40
ProcGen 泛化差距衡量的是什么？动作规则、残余熵与缺失的随机基线
What Does a ProcGen Generalization Gap Measure? Action Rules, Residual Entropy, and the Missing Random Floor
AI 导读
研究指出强化学习的泛化差距应参照随机策略在同一评测环境下的随机基线来解读。在 8 个 ProcGen 环境用 PPO 以 8M 步预算训练时，miner 中采样策略在留出关卡得分为基线的 5.1 倍，而 argmax 策略每次运行都低于基线。审计 12 个 ProcGen 代码库发现 9 个在评测时采样动作却未显式声明选择，作者建议报告差距时说明动作规则、随机种子并给出两个关卡集上的基线。
正文
Abstract:A generalization gap in reinforcement learning, return on training levels minus return on held-out levels, is usually reported without a reference point. We argue that it should be read against a measured random floor: the return of a uniform-random policy on the same levels under the same evaluation harness. On eight ProcGen environments with PPO at a compute-limited budget (8M steps, 16 parallel environments; three games extended to 25M), the floor changes what standard numbers mean. The test-time action rule decides which policy is measured: in miner, the sampled policy scores 5.1x the floor on held-out levels while its argmax scores below it in every run, and greedy evaluation places two environments significantly below the floor. Used as a convergence diagnostic, raw policy entropy flags six of eight environments, but 32-66% of that entropy lies on actions with identical effects; against the floor, five of eight sampled policies are clearly above it on held-out levels and heist's is not distinguishable from it. An audit of twelve ProcGen codebases finds that nine sample test-time actions with no explicit choice at the evaluation call site. We recommend that every reported gap state its action rule, seed its evaluation and specify its tests before analysis, and report the floor on both level sets.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2609.32532 [cs.LG] |
| (or arXiv:2609.32532v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2609.32532 arXiv-issued DOI via DataCite |
Submission history
From: Abhisek Keshari [view email]
[v1]
Sat, 26 Sep 2026 12:14:35 UTC (50 KB)
[v2]
Fri, 2 Oct 2026 16:40:58 UTC (70 KB)
来源：arXiv：cs.LG · arxiv.org