arXiv：cs.AI· Tingrui Huang, Ai Bo, Junzhe Zhang·· 18 小时前
CauCRL：用反事实 rollout 解决离线对比目标条件强化学习中的混淆问题
Counterfactual Rollouts for Offline Contrastive Goal-Conditioned Reinforcement Learning
AI 导读
针对离线示范中存在隐藏信息导致对比强化学习（CRL）学到虚假动作—结果关联的问题，研究者提出 Causal Contrastive Reinforcement Learning（CauCRL），通过从正负离线示范中学习悲观的反事实转移，并在反事实 rollout 上训练 critic，在混淆的目标达成任务上显著提升成功率。
正文
Abstract:Contrastive reinforcement learning (CRL) learns goal-conditioned values by associating state--action pairs with the states subsequently reached, assuming that no unobserved variable jointly affects the recorded action and its outcome. When offline demonstrations come from a demonstrator with information hidden from the learner, this assumption fails and the critic absorbs spurious action--outcome associations. We formalize this setting with confounded Markov decision processes and introduce counterfactual transitions that separate what the data identify from what they do not; minimizing over all compatible transitions yields a worst-case occupancy that lower-bounds the true one. Causal Contrastive Reinforcement Learning (CauCRL) learns a pessimistic counterfactual transition from positive and negative offline demonstrations and trains the critic on counterfactual rollouts, substantially improving success rates over standard CRL on confounded goal-reaching tasks.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.14953 [cs.LG] |
| (or arXiv:2610.14953v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.14953 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tingrui Huang [view email]
[v1]
Sat, 10 Oct 2026 14:25:32 UTC (648 KB)
来源：arXiv：cs.AI · arxiv.org