xAI 开源 314B 参数 MoE 模型 Grok-1 权重与架构

Open Release of Grok-1 We are releasing the weights and architecture of our 314 billion parameter Mixture-of-Experts model Grok-1. Mar 17, 2024

AI 导读

xAI 开源 Grok-1 的基础模型权重和网络架构，采用 Apache 2.0 许可。Grok-1 是 314B 参数的 Mixture-of-Experts 模型，每个 token 激活 25% 的权重，由 xAI 基于 JAX 和 Rust 的自研训练栈从零训练，预训练于 2023 年 10 月完成，未针对对话等具体应用做微调。使用说明见 github.com/xai-org/grok。

推荐理由

官方公告明确了 Grok-1 的参数规模、MoE 激活比例和 Apache 2.0 许可，读者可据此评估其开源使用条件。

正文 · 原文

We are releasing the base model weights and network architecture of Grok-1, our large language model. Grok-1 is a 314 billion parameter Mixture-of-Experts model trained from scratch by xAI.

This is the raw base model checkpoint from the Grok-1 pre-training phase, which concluded in October 2023. This means that the model is not fine-tuned for any specific application, such as dialogue.

We are releasing the weights and the architecture under the Apache 2.0 license.

To get started with using the model, follow the instructions at github.com/xai-org/grok.

Model Details

Base model trained on a large amount of text data, not fine-tuned for any particular task.
314B parameter Mixture-of-Experts model with 25% of the weights active on a given token.
Trained from scratch by xAI using a custom training stack on top of JAX and Rust in October 2023.