arXiv：cs.CL· LLM-jp, :, Chaoran Liu, Daisuke Kawahara, Hideyuki Tachibana, Hirokazu Kiyomaru, Jun Suzuki, Koshi Eguchi, Kouta Nakayama, Qianying Liu, Rio Yokota, Ryo Takahashi, Sadao Kurohashi, Sosuke Hosokawa, Taishi Nakamura, Takashi Kodama, Yun-Ang Wu, Yusuke Miyao, Yusuke Oda·· 16 小时前
LLM-jp-4 技术报告：8B 稠密与 32B-A3B MoE 的日本主权大模型
LLM-jp-4 Technical Report
AI 导读
LLM-jp-4 是一组日本主权大语言模型，包含 8B 稠密模型和 32B-A3B MoE 模型，均从零开始基于约 12 万亿 tokens 训练。团队为训练数据、外部模型与软件设定了明确的筛选标准并公开开发流程，以支持独立验证与复现。评测显示其较此前 LLM-jp 模型大幅提升，与同规模开放权重模型相比具备竞争力；本版本未包含 33B 稠密模型结果。
正文
Authors:LLM-jp: Chaoran Liu, Daisuke Kawahara, Hideyuki Tachibana, Hirokazu Kiyomaru, Jun Suzuki, Koshi Eguchi, Kouta Nakayama, Qianying Liu, Rio Yokota, Ryo Takahashi, Sadao Kurohashi, Sosuke Hosokawa, Taishi Nakamura, Takashi Kodama, Yun-Ang Wu, Yusuke Miyao, Yusuke Oda
Abstract:This paper presents LLM-jp-4, a family of Japanese sovereign large language models (LLMs) developed with an emphasis on transparency, reproducibility, and practical usability. LLM-jp-4 comprises an 8B dense model and a 32B-A3B mixture-of-experts model, both trained from scratch on approximately 12 trillion tokens. To support independent verification and reproduction, we establish explicit criteria for selecting training data, external models, and software, and make the development process publicly available. Evaluation results demonstrate substantial improvements over previous LLM-jp models and competitive performance against open-weight models of similar scale. By documenting the development principles, training procedures, and empirical findings of LLM-jp-4, we aim to provide a reference for future sovereign AI development.
| Comments: | Results for the LLM-jp-4 33B dense model are not included in this version and will be added in a future revision |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2610.19331 [cs.CL] |
| (or arXiv:2610.19331v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.19331 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hirokazu Kiyomaru [view email]
[v1]
Wed, 14 Oct 2026 00:16:21 UTC (986 KB)
来源：arXiv：cs.CL · arxiv.org