arXiv：cs.CL· Yanshu Li, Ruixiang Tang, Junyi Jessy Li, Elias Stengel-Eskin·· 16 小时前
ActICL：面向智能体多模态上下文学习的后训练方法
ActICL: Post-Training for Agentic Multimodal In-Context Learning
AI 导读
研究提出 ActICL，通过工具增强推理训练 MLLM 智能体自主获取和使用 ICL 上下文，在迭代推理中分析信息缺口并有针对性地检索、过滤候选示例。在七个基准和三个 MLLM 上，ActICL 平均性能较标准相似度检索提升 11.6 个百分点，较最强先前基线提升 4.4 个百分点。
正文
Abstract:Multimodal in-context learning (ICL) is an important capability of multimodal large language models (MLLMs). Its performance is closely tied to the demonstrations it uses, making demonstration retrieval a core problem for ICL. Common methods use fixed scoring methods to retrieve all the demonstrations at once, which may introduce redundancy and noise. Context-aware retrieval methods have emerged to address this problem by allowing incremental retrieval from a growing context. However, none of them lets the MLLM learn how to judge what new demonstrations it needs and then carry out progressive retrieval on its own. To this end, we introduce ActICL, which trains an MLLM agent to acquire and use an ICL context via tool-augmented reasoning. An ActICL agent analyzes its own information gaps during iterative reasoning and tool-use actions. Based on these gaps, it performs targeted retrieval to acquire a set of initial candidates. It can then use the existing context to further filter the candidates and refine the context, or it can iteratively perform a new retrieval. When the agent judges that the context is sufficient for the task, or when the resource limit is reached with the user request met, it sends the acquired context to an answering model, either the agent itself or another MLLM. We use a two-stage post-training procedure that jointly optimizes the agent's acquisition and answering. With an adaptive number of demonstrations, ActICL improves average performance by 11.6 percentage points over standard similarity-based retrieval across seven benchmarks and three MLLMs, and by 4.4 points compared to the strongest prior baseline. The contexts acquired by ActICL agents can also be used effectively by frontier models. When GPT-5.6-Sol performs 4-shot ICL, ActICL's contexts improve its performance by 3.5 points over standard retrieval, averaged over three benchmarks.
| Comments: | Project page: this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19475 [cs.CV] |
| (or arXiv:2610.19475v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19475 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yanshu Li [view email]
[v1]
Wed, 14 Oct 2026 02:54:40 UTC (2,614 KB)
来源：arXiv：cs.CL · arxiv.org