HuggingFace Daily Papers·· 2 天前
Orchestra：让机器人按多模态指令完成长时程操作
Let the Mind Conceive, Let the Hand Achieve: Following Multimodal Instructions for Long-Horizon Robotic Manipulation
AI 导读
机器人智能体框架 Orchestra 将任务规划与执行管理从动作生成中分离，可把完整多模态指令拆解为有序子任务并跟踪执行进度。它在九个 RMBench 任务上平均成功率达 68.22%，比 Mem-0 高 26.22 个百分点；在新基准 RMBench-Compose 上平均成功长度（ASL）为 7.35 个子任务，是 HarnessVLA 的 2.46 倍。
正文
Abstract:A robot following a multi-step instruction needs to act on the right objects in the right order, even as the scene changes. It must also maintain awareness of task progress and determine when to advance. To address these challenges, we introduce Orchestra, an agent harness that separates task planning and execution management from action generation. Orchestra decomposes a complete multimodal instruction into ordered subtasks, tracks completed steps, and updates the locations of relevant objects and destinations during execution. It maintains a shared context containing the active subtask, spatial grounding, and recent observations. An action policy uses this context to generate actions, while a separately trained progress evaluator estimates when to advance to the next subtask. To evaluate whether robots trained on atomic instructions and short trajectories can execute longer subtask sequences, we introduce a benchmark named RMBench-Compose. Orchestra achieves a mean success rate of $68.22\%$ across nine RMBench tasks, outperforming Mem-0 by $26.22$ percentage points. On RMBench-Compose, it achieves an Average Success Length (ASL) of $7.35$ subtasks, $2.46$ times that of HarnessVLA. Across four real-world multi-step tasks, it improves mean ASL by $25.0\%$ over Orchestra using the base $\pi_{0.5}$ policy. These results show that Orchestra helps robots carry out longer sequences of subtasks while preserving the requested subtask order and acting on the specified objects. Our code and project page are available at this https URL.
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.18352 [cs.RO] |
| (or arXiv:2610.18352v1 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2610.18352 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Youpeng Wen [view email]
[v1]
Tue, 13 Oct 2026 13:20:59 UTC (16,845 KB)
来源：HuggingFace Daily Papers · arxiv.org