RARM：面向机器人操作强化学习的置信度门控进度奖励模型

View PDF HTML (experimental)

Abstract:Reinforcement learning for robot manipulation is often bottlenecked by reward design, especially in long-horizon tasks: sparse success rewards provide weak supervision, while hand-crafted dense rewards are tedious to design and generalize poorly across tasks. Progress-based reward models offer a promising alternative by estimating how far an observation has advanced toward task completion, but existing approaches often require task-specific demonstrations or progress labels, and can assign high rewards to visually plausible but physically incorrect states. We introduce the Reference-Anchored Reward Model (RARM), a lightweight visual comparator that converts a single successful demonstration into a dense, progress-aware reward. RARM is trained once on general-purpose videos with a contrastive temporal objective, requiring no robot-specific data, task-specific reward labels, or per-task reward engineering. At deployment, RARM matches rollout clips to reference clips and rewards only confident forward progress, suppressing uncertain matches that may otherwise produce false-positive rewards. Across 9 simulated manipulation tasks from LIBERO and MetaWorld and 4 real-world tasks, RARM achieves the best overall success rates in subsequent RL training, with particularly large gains on long-horizon tasks such as cloth folding, where unreliable progress estimates are especially harmful.

Subjects:	Robotics (cs.RO); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2606.22027 [cs.RO]
	(or arXiv:2606.22027v5 [cs.RO] for this version)
	https://doi.org/10.48550/arXiv.2606.22027 arXiv-issued DOI via DataCite

Submission history

From: Pengzhi Yang [view email]
[v1] Sat, 20 Jun 2026 13:03:21 UTC (16,178 KB)
[v2] Wed, 24 Jun 2026 05:16:35 UTC (16,178 KB)
[v3] Mon, 24 Aug 2026 04:09:24 UTC (16,181 KB)
[v4] Tue, 25 Aug 2026 07:44:37 UTC (16,181 KB)
[v5] Mon, 12 Oct 2026 07:47:18 UTC (13,680 KB)