arXiv：cs.LG· Debajoy Mukherjee, Jeremy Carleton, Sushil Vemuri, Srinivas Shakkottai, Dileep Kalathil·· 11 小时前
TreeHCA：面向多轮 LLM 智能体的后见引导探索与信用分配
TreeHCA: Hindsight-Guided Exploration and Credit Assignment for Multi-Turn LLM Agents
AI 导读
TreeHCA 通过后见引导探索与后见信用分配相结合，利用参考答案似然变化重访中间历史并采样替代续写，构建推理与工具交互树，并推导出逆后见重要性修正与归一化树回溯，将终局结果传播为轮级优势。在七个问答 benchmark 和三个 Qwen 主干模型上，TreeHCA 的聚合 exact match 超过所有基线，验证了配对探索与信用分配对多轮 LLM 智能体训练的有效性。代码已公开。
正文
Abstract:Outcome-supervised group-relative policy optimization (GRPO) assigns the same advantage to every token in an LLM agent trajectory, leaving useful and unhelpful decisions undifferentiated. Hindsight credit assignment (HCA) can refine this feedback, but its estimates rely on trajectories containing informative alternatives at consequential histories. We introduce TreeHCA, which couples hindsight-guided exploration with hindsight credit assignment. TreeHCA uses changes in reference-answer likelihood to revisit intermediate histories and sample alternative continuations, forming a tree of reasoning and tool interactions. Because this exploration changes the distribution of sampled continuations, we derive the corresponding inverse-hindsight importance correction, including an exact finite-pool identity. We use this structure in a normalized tree backup that propagates terminal outcomes into turn-level advantages. This shared hindsight structure is TreeHCA's central design principle, linking informative exploration with hindsight-guided credit assignment. Across seven question-answering benchmarks and three Qwen backbones, TreeHCA outperforms all baselines in aggregate exact match, validating the effectiveness of paired exploration and credit assignment for multi-turn LLM agent training. Our code is available at this http URL.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.12828 [cs.LG] |
| (or arXiv:2610.12828v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12828 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Debajoy Mukherjee [view email]
[v1]
Fri, 9 Oct 2026 00:23:39 UTC (294 KB)
来源：arXiv：cs.LG · arxiv.org