arXiv：cs.LG· Alberto Mart\'inez, Natalia Flechas Manrique, Lloren\c{c} Espinosa-Portal\'es, Luc Andrea·· 3 小时前
ProtModernBERT：面向低算力场景的高效蛋白质语言建模
ProtModernBERT: Compute-Efficient Protein Language Modeling
AI 导读
ProtModernBERT 是一款现代化 BERT 式蛋白质语言模型，在预训练 FLOPs 减少最多 80-90% 的情况下，性能匹配或接近 Amplify-120M 与 ProtBert，并缩小了与 ESM2 的差距。研究通过基于聚类的验证协议和随机初始化模型对比，表明现代架构组件提供的归纳偏置在低算力场景下比参数量更关键。
正文
Abstract:Large protein language models provide a powerful framework for learning sequence-function relationships and guiding experimental design. However, architectural improvements are underexplored, especially in low-compute regimes of relevance for resource-constrained research laboratories. We show that the training cost for encoder-only transformer architectures can be reduced by leveraging modern architectural refinements, offering an encouraging performance / compute-efficiency trade-off. We introduce ProtModernBERT, a modernized BERT-style model pre-trained on amino acid sequences, and compare it with ProtBert, Amplify, and ESM2 across a set of supervised structural tasks -encompassing secondary structure prediction, disorder prediction, and subcellular localization- and ProteinGym's zero-shot functional benchmark. We find that ProtModernBERT matches or closely tracks the performance of Amplify-120M and ProtBert with up to 80-90\% reduction in pre-training FLOPs, and narrows the gap to ESM2 when controlling for pre-training. Our analysis using a cluster-based validation protocol for hyperparameter selection and randomly initialized models disentangles the contributions of architecture and data, showing that modern components provide stronger inductive biases that are more consequential than parameter count in low-compute regimes.
| Comments: | 12 pages, 5 figures |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Quantitative Methods (q-bio.QM) |
| Cite as: | arXiv:2610.19738 [cs.LG] |
| (or arXiv:2610.19738v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19738 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Llorenç Espinosa-Portalés [view email]
[v1]
Wed, 14 Oct 2026 07:59:49 UTC (2,188 KB)
来源：arXiv：cs.LG · arxiv.org