arXiv：cs.CL· Yi Wang, Tianzi Wang, Tong Lei, Dahan Wang, Peter Bell·· 4 小时前
如何设计文本接口提升 LLM 音视频问答推理能力
Designing Textual Interfaces To Enhance Llm Reasoning On Audio-Visual Question Answering
AI 导读
研究比较了三种文本接口设计对音视频问答任务中提示词优化的影响：单一 OmniLLM 直接推理、captioner-reasoner 系统和由主 Agent 编排子 Agent OmniLLM 的智能体架构。基于 Qwen3-Omni 和 Gemini 的实验显示，直接提示词优化带来依赖模型的增益，而 captioner-reasoner 与智能体系统虽提供更可解释的优化接口，却更难优化。
正文
Abstract:Automatic prompt optimization (APO) has shown strong effectiveness for text-based LLMs and LLM systems, but its application to audio-visual understanding systems requires suitable textual interfaces through which multimodal reasoning can be exposed to text-based optimizers. We investigate how the design of textual interfaces affects prompt optimization for multimodal reasoning, on the audio-visual question answering task. We compare three interface designs, including a direct inference system consisting of a single omni-modal large language model (OmniLLM), a captioner-reasoner system, and an agentic architecture in which a primary agent orchestrates subagent OmniLLMs. Experiments with Qwen3-Omni and Gemini show that direct prompt optimization provides model-dependent gains; the captioner-reasoner system and the agentic system expose more interpretable optimization interfaces but are harder to optimize, where error analysis of cases affected by optimization is conducted. Our study highlights textual interface design as an important factor in making prompt optimization effective for omni-modal reasoning systems. Our implementation details are specified in GitHub repo.
| Comments: | Submitted to IEEE ICASSP 2027 |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Audio and Speech Processing (eess.AS) |
| Cite as: | arXiv:2610.20085 [cs.AI] |
| (or arXiv:2610.20085v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20085 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yi Wang [view email]
[v1]
Wed, 14 Oct 2026 12:23:59 UTC (1,883 KB)
来源：arXiv：cs.CL · arxiv.org