arXiv：cs.LG· Leif Sieben, Yoel Zimmermann·· 4 小时前
预训练真能让化学语言模型学会化学吗？
Does Pretraining Teach Chemical Language Models Chemistry?
AI 导读
研究系统考察了一系列化学语言模型（CLM）在 64 个分子性质预测数据集上的预训练效果，发现预训练带来的下游增益并不随规模扩展，性能饱和且低于梯度提升基线。在评估的各类预训练方式中，只有基于传统物理化学描述符的监督预训练能稳定提升下游表现，部分预训练方式甚至使模型不如随机初始化的编码器。
正文
Abstract:Chemical language models (CLMs) are pretrained on large corpora of molecules with the aim of learning representations that transfer to diverse downstream tasks. Here, we systematically study the effects of pretraining using a series of such CLMs across 64 molecular property prediction datasets. We find that the downstream gains of pretraining in CLMs do not scale and performance saturates below gradient-boosting baselines. Across the regimes evaluated, only supervised pretraining on traditional physico-chemical descriptors consistently improves downstream performance for the CLMs tested whereas some forms of pretraining result in models worse than a randomly initialized encoder. While models can resolve molecular structure well, they do so no better than a randomly initialized model, suggesting that, at best, CLMs learn descriptor-like properties in addition to low-level token statistics during pretraining, but surprisingly little chemistry beyond what classical molecular features already capture.
| Subjects: | Machine Learning (cs.LG); Computation and Language (cs.CL); Chemical Physics (physics.chem-ph) |
| Cite as: | arXiv:2610.18651 [cs.LG] |
| (or arXiv:2610.18651v2 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.18651 arXiv-issued DOI via DataCite |
Submission history
From: Yoel Zimmermann [view email]
[v1]
Tue, 13 Oct 2026 16:14:35 UTC (555 KB)
[v2]
Wed, 14 Oct 2026 17:58:29 UTC (555 KB)
来源：arXiv：cs.LG · arxiv.org