arXiv：cs.CL· Zixuan Liu, Gustavo Didier, Fangzheng Wu, Brian Summa, Zizhan Zheng·· 6 小时前
Weighted Ensemble：面向语言模型稀有事件估计的免训练采样框架
Weighted Ensemble Sampling for Rare-Event Estimation in Language Models
AI 导读
Weighted Ensemble（WE）是一个免训练的稀有事件估计框架，仅从原始语言模型采样，通过周期性将采样预算重新分配给更可能到达目标稀有事件的前缀，改进直接蒙特卡洛采样。该方法利用 LLM 内部表示作为前缀级信号来指导预算重分配，无需额外学习提议分布，并提供了理论分析。在覆盖小规模与现代大规模 LLM 的安全相关任务实验中，WE 提升了稀有事件概率估计的效率与可靠性。
正文
Abstract:Rare but severe failures of large language models can become practically significant at deployment scale. Estimating their probabilities with direct Monte Carlo sampling is highly inefficient, while proposal-based methods typically require auxiliary learning of an additional proposal distribution. We introduce \textbf{Weighted Ensemble (WE)}, a training-free framework for rare-event estimation in language models. WE samples exclusively from the original language model and improves upon direct Monte Carlo by periodically reallocating the sampling budget toward prefixes that appear more likely to reach the target rare event. To guide this reallocation without learning a separate proposal distribution, we exploit internal LLM representations as prefix-level signals and empirically validate their effectiveness. We provide a theoretical analysis of the proposed algorithm. Experiments on safety-related tasks spanning both small-scale and modern large-scale LLMs demonstrate that WE improves the efficiency and reliability of rare-event probability estimation.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2610.19202 [cs.CL] |
| (or arXiv:2610.19202v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19202 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zixuan Liu [view email]
[v1]
Tue, 13 Oct 2026 21:11:48 UTC (111 KB)
来源：arXiv：cs.CL · arxiv.org