arXiv：cs.AI· Ziang Ye, Wentao Shi, Lei Chen, Zhengzhou Cai, Yu Wang, Zhiyuan Yao, Zihan Wang, Xierui Song, Fuli Feng, Yixin Cao·· 11 小时前
ReTRAIL：让长时程智能体回溯环境状态并复用探索经验
Travel Back To Move Forward: Retrospective Exploration and Evolving for Long Horizon Agents
AI 导读
研究者提出 ReTRAIL 框架，让 LLM 智能体在长时程任务中把环境回退到更早状态，同时保留已有的有效进展并复用被放弃探索中学到的知识。该框架用 State-Anchored GRPO 比较回溯状态下的不同后续路径，并以分支推进奖励引导反思。
正文
Abstract:Large language model agents increasingly operate in long-horizon environments where early mistakes can trigger cascading failures and trap agents in dead-end trajectories. Agents must therefore be able to proactively recover from such states and turn their exploratory experience into guidance for subsequent decisions. We introduce ReTRAIL, a framework that enables agents to rewind the environment to earlier states while preserving valid progress and carrying forward knowledge learned from abandoned exploration. The environment state moves backward while the agent's knowledge continues to move forward. ReTRAIL trains a shared policy with two complementary signals: State-Anchored GRPO compares alternative continuations from revisited states to improve execution, while a branch-advancement reward guides reflection through the progress of subsequent exploration. Experiments across four interactive benchmark MineSweeper, Sokoban, ALFWorld and ScienceWorld show that simply adding a rewind primitive provides limited benefit and can even degrade performance. In contrast, ReTRAIL enables agents to rewind from unproductive states and use insights from abandoned branches to guide subsequent exploration. These results demonstrate the value of coupling environment rewinding with experience reuse within an ongoing task for long-horizon task solving. The code is available at this https URL.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.13058 [cs.AI] |
| (or arXiv:2610.13058v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.13058 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ziang Ye [view email]
[v1]
Fri, 9 Oct 2026 06:11:39 UTC (1,156 KB)
来源：arXiv：cs.AI · arxiv.org