arXiv：cs.CL· Hare Krishna·· 11 小时前
掩码文本重建中的语料采样分歧预测
Forecasting Corpus-Sampling Disagreement in Masked Text Reconstruction
AI 导读
研究提出一种基于邻近 token 对计数的平滑预测器，在完整 50,257-token GPT-2 词表上推导出两个独立语料训练出的预测器之间预期分歧的一阶公式。
正文
Abstract:If a second text corpus were collected from the same population, how much would a predictor's outputs change? We study this question for masked-text reconstruction using a smoothed predictor built from nearby token-pair counts and evaluated over the full 50,257-token GPT-2 vocabulary. Treating each source document as one observation, we derive a first-order formula for the expected disagreement between predictors fitted to two independent corpora. The formula accounts for variable document length, random context counts, and correlations across token offsets, and it can be evaluated from a separate reference corpus without access to either target corpus. On TinyStories, WikiText-103, and CNN/DailyMail, growing each corpus from 512 to 2,048 chunks of 64 tokens lowers disagreement by only 5--40\%, far short of the 75\% implied by $1/n$ scaling. The forecast tracks this slow decline, which we trace to smoothing of rarely seen contexts. Forecast accuracy depends less on how these correlations are modeled than on the size of the reference corpus: small references systematically underpredict disagreement. A source-split jackknife that uses only reference data reduces this calibration error by 57--59\%. After this correction, a simpler smoothed-multinomial approximation is most accurate when the reference corpus is smaller than the target corpus, and a nonlinear source bootstrap is most accurate when it is larger. Finally, small masked diffusion language models trained on the same corpora show that the count-based forecasts partly identify which inputs are most sensitive to the corpus, but miss the neural models' much faster decline in disagreement.
| Comments: | 12 pages + Appendices |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG); Machine Learning (stat.ML) |
| Cite as: | arXiv:2610.12987 [cs.CL] |
| (or arXiv:2610.12987v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.12987 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hare Krishna [view email]
[v1]
Fri, 9 Oct 2026 04:40:47 UTC (154 KB)
来源：arXiv：cs.CL · arxiv.org