arXiv：cs.LG· Mingquan Feng, Yixin Huang, Yifan Fu, Shaobo Wang, Junchi Yan·· 7 小时前AI 评分34
KO：受动力学启发的神经优化器，用 PDE 模拟方法优化神经网络
KO: Kinetics-inspired Neural Optimizer with PDE Simulation Approaches
AI 导读
研究者提出即插即用优化模块 KO（Kinetics-inspired Optimizer），将参数动力学建模为粒子系统，通过离散 Boltzmann 传输方程引入随机相互作用，以提升参数多样性并缓解权重凝聚。理论分析证明 KO 在保持收敛保证的同时可证明地增加参数多样性。在 CIFAR-10/100、ImageNet 图像分类和大规模语言模型预训练中，KO 以可忽略的额外计算成本持续提升准确率。
正文
Abstract:The design of effective optimization algorithms for neural networks remains a fundamental challenge, and most existing methods rely on heuristic extensions of gradient-based updates. We introduce KO (Kinetics-inspired Optimizer), a plug-and-play optimization module grounded in kinetic theory and partial differential equations. KO models parameter dynamics as a particle system, augmenting standard gradient updates with stochastic interactions induced by a discretization of the Boltzmann transport equation. This mechanism naturally promotes parameter diversity and mitigates weight condensation, the tendency of parameters to collapse into low-dimensional subspaces, a phenomenon closely associated with degraded generalization. We provide both a rigorous theoretical analysis and a physical interpretation, showing that KO provably increases parameter diversity while preserving convergence guarantees. Extensive experiments on image classification benchmarks (CIFAR-10/100, ImageNet) and large-scale language model pretraining demonstrate that KO consistently improves accuracy over competitive baselines with negligible additional computational cost.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2505.14777 [cs.LG] |
| (or arXiv:2505.14777v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2505.14777 arXiv-issued DOI via DataCite |
Submission history
From: Yixin Huang [view email]
[v1]
Tue, 20 May 2025 18:00:01 UTC (6,225 KB)
[v2]
Tue, 6 Oct 2026 05:39:39 UTC (6,291 KB)
来源：arXiv：cs.LG · arxiv.org