用4B参数与1T token缩小 point-in-time 语言模型性能差距

View PDF HTML (experimental)

Abstract:Large language models trained on unrestricted internet corpora inevitably embed information from the future, introducing lookahead bias that compromises the validity of backtests and causal inference in finance and the social sciences. Point-in-time language models--trained exclusively on text available up to each calendar date--eliminate this leakage by construction, but existing efforts typically produce models that lag substantially behind their unconstrained counterparts. We show that this performance gap can be substantially narrowed through scale. Training decoder-only transformers with up to 4 billion parameters on 1 trillion chronologically filtered tokens from FineWeb, we construct a sequence of monthly model checkpoints spanning 2013-2024. Across a range of common-sense reasoning and language understanding benchmarks, our models approach the performance of leading open-weight models of comparable size (e.g., Gemma-3-4B and LLaMA-7B) trained on temporally unrestricted data, although a performance gap remains on several tasks. Instruction fine-tuning via LoRA further improves downstream usability. We release the complete pipeline--including dataset construction, training infrastructure, and evaluation code--to enable reproducible point-in-time language modeling and to support research applications that require strict temporal validity.

Subjects:	Computation and Language (cs.CL); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2607.11889 [cs.CL]
	(or arXiv:2607.11889v3 [cs.CL] for this version)
	https://doi.org/10.48550/arXiv.2607.11889 arXiv-issued DOI via DataCite

Submission history

From: Johannes Schwab [view email]
[v1] Fri, 24 Apr 2026 17:00:53 UTC (180 KB)
[v2] Fri, 17 Jul 2026 09:32:58 UTC (101 KB)
[v3] Mon, 12 Oct 2026 12:26:01 UTC (106 KB)