arXiv：cs.LG· Eric O. Korman·· 3 小时前
逆推理模型：仅用答案数据做高效后训练与验证
Inverse Reasoning Models for Efficient Post-Training and Verification from Answer-Only Data
AI 导读
研究者提出逆推理 LLM 的自蒸馏方法，仅用 prompt-answer 数据即可后训练推理模型，将 CoT 视为潜变量，逆模型一次训练后可离线复用。在 Qwen3-0.6B 上，GRPO 训练饱和后经一轮离线蒸馏，困难提示 pass@1 约翻倍，Countdown 上各难度指标均优于拒绝采样自蒸馏。
正文
Abstract:Post-training reasoning LLMs often requires making a tradeoff between costly supervision and compute inefficiency: supervised finetuning typically requires reasoning traces, which is data inefficient since in many applications one only has prompt-answer tuples and lacks the groundtruth chain-of-thought (CoT) sequence. On the other hand, reinforcement learning (with verifiable rewards, for example) can operate without CoT data but it is much more compute-inefficient since it requires multiple autoregressive completions per optimization step, none of which are guaranteed to give a non-zero policy gradient. In this work we explore a middleground, a self-distillation technique that uses answer-only data, treating CoT as latent variables. This relies on training a partner ``inverse reasoning LLM" that generates the original's reasoning traces conditioned on the answer. The inverse model is trained once and reused across incoming batches of answer-only data, with all generation happening offline and up front, making the approach particularly suited to continual post-training. Using Qwen3-0.6B on the Countdown and GSM8K tasks, we show that after GRPO training has saturated, a single offline distillation round from the inverse model roughly doubles pass@1 on hard prompts and, on Countdown, outperforms rejection-sampling self-distillation on every hard-tail metric at every mix ratio (with smaller gains on GSM8K); rejection sampling by construction has no signal on problems the model never solves on its own. Continued GRPO on the fresh data retains an edge at hard pass@1 but trails distillation at pass@8 and pass@32 while regenerating rollouts at every step. As an added bonus we see that the inverse model has a second application as an accurate verifier of correctness. When used for best-of-$n$ selection at test time, it raises selection accuracy on Countdown from 0.753 (majority voting) to 0.969.
| Comments: | Published at the COLM 2026 Workshop on Efficient Reasoning |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.18878 [cs.LG] |
| (or arXiv:2610.18878v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.18878 arXiv-issued DOI via DataCite |
Submission history
From: Eric Korman [view email]
[v1]
Thu, 1 Oct 2026 01:56:34 UTC (1,340 KB)
来源：arXiv：cs.LG · arxiv.org