arXiv：cs.LG· Tom\'a\v{s} \v{C}\'i\v{z}ek, Matej Straka, Illia Volkov, Mat\v{e}j Kripner, Ervin Maci\'c, Jiri Matas, Klara Janouskova, Martin Schmid·· 4 小时前
SAM3RL：用强化学习控制记忆库实现视觉目标跟踪
SAM3RL: Memory Control via Reinforcement Learning for Visual Object Tracking
AI 导读
SAM3RL 为 SAM 3 跟踪器加装轻量记忆控制器，通过强化学习决定哪些帧进入记忆库，在 12 项跟踪与视频目标分割基准中的 10 项取得新的 SOTA，性能超过所有手工规则且无运行时开销。同一方法应用于 SAM 2 得到 SAM2RL，在 12 项基准中的 9 项优于手工规则，显示该方法可跨模型代际通用。
正文
Abstract:Segment Anything Model 3 (SAM 3) is a foundation model for image and video segmentation that sets the state of the art in visual object tracking. Its tracker, inherited from SAM 2, stores information from past frames in a memory bank, managed by a simple First-In-First-Out rule, allowing temporal consistency in complex videos. Several recent methods improve SAM 2 with hand-crafted memory update rules, each targeting a specific failure mode, such as distractors, occlusions, or object motion. However, their gains transfer only partially to the stronger SAM 3 backbone. We instead present a more general, data-driven approach that frames memory control as a sequential decision-making task. SAM3RL is an extension of the SAM 3 tracker with a lightweight memory controller, trained by reinforcement learning, that decides which frames enter the memory bank in order to maximize tracking performance. SAM3RL sets a new state of the art on 10 of 12 tracking and video object segmentation benchmarks, outperforming all hand-crafted rules without any runtime overhead. The same approach applied to SAM 2 gives SAM2RL, which outperforms hand-crafted rules on 9 of the 12 benchmarks, showing the generality of our method across model generations.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.19931 [cs.CV] |
| (or arXiv:2610.19931v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19931 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tomáš Čížek [view email]
[v1]
Wed, 14 Oct 2026 10:17:47 UTC (2,643 KB)
来源：arXiv：cs.LG · arxiv.org