arXiv：cs.AI· Youting Wang, Yuan Tang, Bowen Liu, Xuan Liu, Dingyan Shang·· 22 小时前
LLM 奖励设计为何失败：稀疏结构化强化学习的诊断驱动迭代优化
When LLM Reward Design Fails: Diagnostic-Driven Refinement for Sparse Structured RL
AI 导读
针对稀疏结构化强化学习任务，LLM 生成奖励函数应视为调试而非一次性生成，研究提出诊断驱动的迭代优化方法，用训练诊断和失败模式分类指导奖励函数修订。在 MiniGrid 的 DoorKey-8x8 上成功率从无 shaping 的 2.3% 提升至 97.6%，KeyCorridor 从一次性生成的 31.2% 提升至 86.7%。
正文
Abstract:For sparse, structured reinforcement-learning tasks with semantic reward-function interfaces, LLM-generated reward shaping is better framed as debugging than one-shot generation. We study PPO-trained agents using MiniGrid as core evaluation and MuJoCo reaching/locomotion as boundary stress tests. Our qualitative audit finds two recurring one-shot failure modes -- reward flooding and semantic/API misunderstanding -- and a rarer, less reliably distinguished weak-shaping case. We propose diagnostic-driven iterative refinement, using training diagnostics and a failure-mode taxonomy to guide reward-function revision. On sparse structured tasks with diagnosable reward failures, refinement improves DoorKey-8x8 from 2.3% success without shaping to 97.6%, and KeyCorridor from 31.2% one-shot to 86.7% with high seed-to-seed variance. Controls show these gains are not from generic retrying or extra training alone: metrics-only re-prompting causes large drops, while a static-vocabulary control recovers much of the gap. Thus taxonomy prompting is a major mechanism, with only partially isolated incremental evidence for dynamic labels. In prompt-matched comparisons across Haiku 4.5, Opus 5, and DeepSeek V4 Pro, effects depend on task and model; none of the six contrasts survives family-wise correction. A four-call, episode-budget-matched population control is also competitive. Component-removal and sensitivity tests reveal calibration limits. In continuous control, success-based diagnostics can misfire in dense-reward locomotion; return-trend feedback removes one false-positive mechanism without robust locomotion gains. The method is bounded to sparse structured tasks with reliable structured interfaces under PPO training.
| Comments: | Published in Transactions on Machine Learning Research (TMLR), 2026. 32 pages. Final published version: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Information Retrieval (cs.IR) |
| Cite as: | arXiv:2605.28918 [cs.LG] |
| (or arXiv:2605.28918v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.28918 arXiv-issued DOI via DataCite |
|
| Journal reference: | Transactions on Machine Learning Research, 2026 |
Submission history
From: Youting Wang [view email]
[v1]
Wed, 27 May 2026 17:57:43 UTC (2,840 KB)
[v2]
Sat, 10 Oct 2026 05:41:37 UTC (2,802 KB)
来源：arXiv：cs.AI · arxiv.org