arXiv：cs.AI· Ximing Lu, Aviral Kumar, Jan Kautz, Yejin Choi, David Acuna·· 11 小时前
MagicBeanStalk：用递归数据合成逃离 RLVR 零优势陷阱
MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis
AI 导读
MagicBeanStalk 通过自顶向下分解与自底向上递归合成，将种子问题扩展为策略与预算校准的新训练题，破解约 88% 问题在每次 rollout 中全对或全错的零优势陷阱。
正文
Abstract:Data-centric scaling of RLVR has focused on collecting or generating verifiable training problems. However, corpus size alone does not determine whether the dataset provides a useful learning signal for a target policy. To guide the synthesis of effective RLVR data, we experimentally characterize dataset effectiveness as a function of its problems, the target policy, and the training rollout budget. Under this characterization, we show that a large open-source math corpus pooled from widely used RLVR datasets falls almost entirely into a zero-advantage trap: for a fixed target policy and rollout budget, roughly 88% of its problems are solved in every rollout or failed in every rollout. Filtering concentrates what remains, but it can only select, discarding the hard problems. Effective synthetic problems should therefore require the reasoning operations and compositions required to solve those hard problems. Yet their difficulty must be calibrated, so that some rollouts succeed, some fail, and at least one correct solution fits within the training budget. Prompting a teacher to vary hard problems directly does not achieve this. Guided by these empirical principles, we introduce MagicBeanStalk, a top-down decomposition and bottom-up recursive data-synthesis algorithm that grows seed problems into families of substantively new, policy- and budget-calibrated training problems while preserving the seeds' reasoning blueprints. Across 8 math benchmarks, continual RLVR on MagicBeanStalk problems improves gpt-oss-120b from 61.2 to 69.8 (+8.6) and DeepSeek v4Flash from 72.4 to 77.9 (+5.4) on a 16K-token budget. Training on the full 78K source corpus improves by only +1.1 and its best filtered real data subset by +5.8. In finance, we synthesize over 14K verifiable problems from only 340 solutions and improve over base by 6.9 and 2.4 points on SecQue and FinanceBench, respectively.
| Subjects: | Artificial Intelligence (cs.AI); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.12659 [cs.AI] |
| (or arXiv:2610.12659v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12659 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: David Acuna [view email]
[v1]
Thu, 8 Oct 2026 19:25:48 UTC (1,554 KB)
来源：arXiv：cs.AI · arxiv.org