arXiv：cs.AI· Lonnie Tankersley, Noorbakhsh Amiri Golilarz, Soroush Korivand·· 22 小时前
基于硬件感知 LLM-VLM 集成的具身零样本任务规划，实现有物理依据的机器人操作
Embodied Zero-Shot Task Planning via Hardware-Aware LLM-VLM Integration for Grounded Robotic Manipulation
AI 导读
研究者提出一种硬件感知的 LLM-VLM 操作框架，在 NVIDIA Isaac Sim 中对模拟 Franka Research 3 机器人实现零样本任务规划。
正文
Abstract:Large language models can translate natural-language instructions into plausible manipulation plans, but robotic execution requires physically grounded decisions that respect robot embodiment, workspace geometry, reachability, collision constraints, and grasp feasibility. This paper presents a hardware-aware large language model-vision-language model (LLM-VLM) manipulation framework for a simulated Franka Research 3 robot in NVIDIA Isaac Sim. The framework separates symbolic reasoning from metric execution: an API-hosted LLM planner receives a structured scene summary and generates only symbolic task information, including the object to manipulate, target relation, approach preference, and objects to avoid. A local Qwen3-VL 8B VLM verifier supports semantic confirmation of ambiguous objects and open-container surrogates. Both the LLM planner and the VLM verifier are prohibited from generating robot coordinates, grasp poses, keepout dimensions, gripper commands, trajectory waypoints, or success labels. Instead, these decisions are handled by deterministic grounding and execution modules that use the robot model, scene geometry, MoveIt 2 collision checking, inverse kinematics, grasp validation, release verification, and final visual assessment. Execution feedback is returned only as symbolic context for replanning. The framework is evaluated through simulated digital-twin trials involving pickup, occluded-source access, keepout reasoning, relation placement, and sequential manipulation tasks.
| Comments: | 6 pages, 5 figures, 1 table. Accepted and presented at the 25th IEEE International Conference on Machine Learning and Applications (ICMLA 2026), Oct 5-7, 2026 |
| Subjects: | Robotics (cs.RO); Artificial Intelligence (cs.AI) |
| ACM classes: | I.2.9; I.2.10; I.2.7 |
| Cite as: | arXiv:2610.14106 [cs.RO] |
| (or arXiv:2610.14106v1 [cs.RO] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14106 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Soroush Korivand Dr. [view email]
[v1]
Thu, 8 Oct 2026 22:59:11 UTC (15,156 KB)
来源：arXiv：cs.AI · arxiv.org