HINT-SD：面向长时程智能体的定向事后自蒸馏框架

View PDF HTML (experimental)

Abstract:Training long-horizon LLM agents with reinforcement learning is challenging because sparse outcome rewards reveal whether a task succeeds, but not which intermediate actions caused the outcome or how they should be corrected. Recent methods alleviate this issue by generating rewards or textual hints from turn-level action-output signals, or by using feedback-conditioned self-distillation. However, generating feedback at every turn is inefficient when many intermediate turns are already successful or neutral, and applying feedback at a fixed or misaligned turn often fails to supervise the actions that contributed to the failure. To bridge this gap, we propose HINT-SD, a targeted self-distillation framework that uses full-trajectory hindsight to select failure-relevant actions and applies feedback-conditioned distillation only to targeted action spans. Experiments on BFCL v3 and AppWorld show that our method outperforms the dense per-turn feedback baseline by up to 13.60 percentage points on average while achieving a 2.26$\times$ reduction in time per training step, suggesting that selecting where to distill is key to effective and efficient long-horizon agent training.

Comments:	EMNLP Findings 2026. Code : this https URL
Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Cite as:	arXiv:2605.17873 [cs.LG]
	(or arXiv:2605.17873v3 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2605.17873 arXiv-issued DOI via DataCite

Submission history

From: Woongyeong Yeo [view email]
[v1] Mon, 18 May 2026 05:34:03 UTC (79 KB)
[v2] Thu, 27 Aug 2026 08:46:37 UTC (134 KB)
[v3] Fri, 2 Oct 2026 15:21:59 UTC (134 KB)