Elon Musk’s synthetic intelligence firm, xAI, has launched the weights and structure of their 314 billion parameter Mixture-of-Experts mannequin, Grok-1, below the Apache 2.0 license.
This follows a pledge by Musk final Monday to make Grok freely accessible to the general public. As somebody who has been carefully following developments within the AI area, I’ve to say this can be a large step ahead by way of openness and accessibility.
What is Grok?
Grok is a big language mannequin with 314 billion parameters, making it the most important open-source mannequin at present accessible. For context, that’s greater than double the dimensions of OpenAI’s GPT-3, which was thought of a breakthrough when it was launched in 2020.
In a chart shared by X person Andrew Kean Gao, you may see how large Grok’s dimension is in comparison with its rivals.
But Grok isn’t simply larger, it additionally makes use of a Mixture-of-Experts (MoE) structure, permitting it to strategically activate subsets of its parameters for various duties. This theoretically makes it extra environment friendly and adaptable in comparison with a standard dense mannequin.
Here’s a abstract of Grok-1 launch
- 314B parameter Mixture-of-Experts mannequin with 25% of the weights energetic on a given token
- The base mannequin is skilled on a considerable amount of textual content information, not fine-tuned for any specific process.
- 8 specialists (2 energetic)
- 86 billion parameters
- Apache 2.0 license
- Paper: https://x.ai/weblog/grok-os
- Code: https://github.com/xai-org/grok-1
For extra updates, comply with Grok on X.
How to put in Grok
The directions on loading and operating Grok-1 are defined on this GitHub repository. Clone the repo to your native.
