arXiv：cs.AI· Tomer Galanti, Josh Susskind, Eran Malach·· 16 小时前
更少比特，更多尝试：内存约束下如何恢复模型能力
Fewer Bits, More Tries: Recovering Capability Under Memory Constraints
AI 导读
论文提出用“验证式重试”在内存受限的本地部署中恢复模型能力：让更小或更压缩的模型并行生成多个候选，再用公开或模型自生成的测试筛选输出。在四个代码生成基准上，7B Qwen 模型八次并行尝试在 MBPP+ 上基本追平 32B 模型、其余三个基准上超过它，同时模型内存约为其 1/4、速度最高快 6 倍且能耗更低（Nvidia GPU 实测）。
正文
Abstract:Large language models are generally more capable than smaller ones, but their memory requirements can make local deployment difficult. We ask whether part of this capability gap can instead be recovered by running a smaller or more compressed model multiple times and selecting among its outputs using public or model-generated tests. Single-user local inference can leave hardware underutilized, allowing parallel candidate generation to trade additional test-time computation for accuracy at modest latency cost. Across four code-generation benchmarks, verified retry recovers substantial accuracy lost to model size or compression. A 7B Qwen model with eight parallel attempts essentially matches a 32B model on MBPP+ and exceeds it on the other three benchmarks, while using about $4\times$ less model memory, running up to $6\times$ faster, and using less energy, as measured on Nvidia GPU. We observe similar tradeoff across model sizes, compression levels and hardware choices, although exact benefit depends on the hardware profile and problem setting. These results show that model capacity can be traded for verified test-time computation when correct solutions remain reachable within a modest retry budget.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.15372 [cs.AI] |
| (or arXiv:2610.15372v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2610.15372 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tomer Galanti [view email]
[v1]
Sun, 11 Oct 2026 05:52:48 UTC (292 KB)
来源：arXiv：cs.AI · arxiv.org