Key Points
- Elon Musk shared a message on his X account, saying that the second variant of his AI chatbot, Grok-1.5 become available next week.
- Grok 2 should exceed current AI on all metrics and is now in training, according to Musk.
Elon Musk has just announced that Grok 2, the second variant of his first AI chatbot called Grok-1.5, is currently in the works and will become available next week.
He also explained that Grok 2 should exceed current AI on all metrics.
https://twitter.com/elonmusk/status/1773655245769330757
According to the official announcement, Grok-1.5 comes with improved reasoning capabilities and a context length of 128,000 tokens.
Introducing Grok-1.5 – capabilities and reasoning
Grok-1.5 is the latest model capable of advanced reasoning and understanding long context. It will be available on the X platform to early testers and existing Grok users in the coming days.
The official notes recently shared the model weights and network architecture of Grok-1, which provided a glimpse into the progress made by xAI until November last year.
However, Musk’s team has since made further improvements in their latest model, Grok-1.5, which now has enhanced reasoning and problem-solving capabilities.
The official notes reveal that one of the most important enhanced that Grok-1.5 will see involves coding performance and math-related tasks:
“One of the most notable improvements in Grok-1.5 is its performance in coding and math-related tasks. In our tests, Grok-1.5 achieved a 50.6% score on the MATH benchmark and a 90% score on the GSM8K benchmark, two math benchmarks covering a wide range of grade school to high school competition problems. Additionally, it scored 74.1% on the HumanEval benchmark, which evaluates code generation and problem-solving abilities.”
Long-context understanding
Another important feature that Grok-1.5 will get is the capability to process long context of up to 128k tokens within the context window.
The AI-powered language model will have an increased memory capacity of up to 16 times the previous context length. This allows it to utilize information from substantially longer documents.
As its context window expands, it can handle longer and more complex prompts while still maintaining its instruction-following capability. In the Needle In A Haystack (NIAH) evaluation, Grok-1.5 demonstrated powerful retrieval capabilities for embedded text within contexts of up to 128K tokens in length, achieving perfect retrieval results.
Grok-1.5 Infra
Grok-1.5 has been developed using a specialized distributed training framework that is based on JAX, Rust, and Kubernetes. With this training stack, the team can easily prototype new ideas and train complex models on a large scale.
One of the biggest challenges of training large language models (LLMs) on a compute cluster is ensuring maximum reliability and uptime of the training job.
However, the custom training orchestrator is designed to automatically detect and remove problematic nodes from the training job, thus ensuring the job runs smoothly without any interruptions.
Official notes also reveal that the team optimized checkpointing, data loading, and training job restarts in order to minimize downtime in case of failure.
Musk’s company, xAI, has launched an AI chatbot named Grok in 2023. xAI was launched in July 2023 with a mission to advance the collective understanding of the universe through AI.