arXiv：cs.LG· Jiacheng Li, Jianchao Tan, Kefeng Zhang, Yuchen Xie, Xunliang Cai·· 3 小时前
CAST：面向 LLM 预训练的历史自适应谱裁剪方法
CAST: History-Adaptive Spectral Clipping for LLM Pretraining
AI 导读
CAST（Clipping with Adaptive Scales and Thresholds）是软谱裁剪（SSC）的历史自适应扩展，用 AdamW 的历史二阶矩设定行、列缩放，并用裁剪输出谱范数估计值的指数移动平均设定各矩阵阈值，在 AdamW 更新前恢复原始坐标。
正文
Abstract:Soft spectral clipping (SSC) reduces gradient singular values, but its effect depends on the gradient scale and clipping threshold. We introduce CAST (Clipping with Adaptive Scales and Thresholds), a history-adaptive extension of SSC. AdamW's previous second moment sets row and column scales. An exponential moving average of the clipped output's estimated spectral norm sets each matrix's threshold. CAST restores the original coordinates before the AdamW update. We train a mixture-of-experts (MoE) model with 19.08B total parameters for 83.89B tokens. CAST achieves lower training and mean validation losses than nine gradient-processing baselines, with the fewest relative gradient spikes. It reaches the best baseline's final smoothed training loss in 7.2\% fewer steps.
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.19761 [cs.LG] |
| (or arXiv:2610.19761v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19761 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiacheng Li [view email]
[v1]
Wed, 14 Oct 2026 08:15:43 UTC (905 KB)
来源：arXiv：cs.LG · arxiv.org