Musk: The next-generation large model Grok 3 will require 100,000 NVIDIA H100 chips
Elon Musk expects that the next generation large-scale model Grok 3 will require 100,000 NVIDIA H100 GPUs, with potential power consumption equivalent to that of a small city. Musk pointed out that the current development of AI technology faces two major challenges: a shortage of high-end GPUs and enormous power demand. Nevertheless, advancements in computing and storage technology will make it possible to train even larger language models in the coming years. Musk predicts that an artificial intelligence smarter than the most intelligent human will be born in the next one to two years
Tesla CEO Elon Musk recently made a bold prediction about the development of Artificial General Intelligence (AGI) during an interview.
He believes:
AGI may achieve breakthroughs surpassing human intelligence in the next 2 years, but this process will require a large number of GPUs and power consumption.
According to Musk, his AI company xAI is currently training the second-generation large language model Grok 2, with the next training phase expected to be completed in May. The training of Grok 2 has already consumed about 20,000 NVIDIA H100 GPUs. The development of the advanced version Grok 3 in the future may require up to 100,000 NVIDIA H100 GPUs.
Musk pointed out:
The current development of AI technology faces two major challenges: one is the shortage of high-end GPUs, such as the NVIDIA H100, acquiring 100,000 GPUs quickly is not an easy task; the other is the huge power demand. A single NVIDIA H100 GPU consumes about 700 watts of power when working at full load, so 100,000 such GPUs will consume up to 70 megawatts of power. Considering the requirements of servers and cooling systems, the power consumption of a data center equipped with 100,000 NVIDIA H100 processors will be approximately 100 megawatts, equivalent to the power consumption of a small city.
These two major limiting factors highlight the challenges faced in expanding AI technology to meet the growing computational demands.
Nevertheless, advancements in computing and storage technology will make it possible to train even larger language models in the coming years. The Blackwell B200 GPU platform showcased by NVIDIA at the technology summit GTC 2024, designed to support large language models scalable to tens of trillions of parameters, heralds a crucial step in the development of AGI.
Musk predicts:
An artificial intelligence smarter than the smartest human will emerge in the next one to two years. If AGI is defined as intelligence surpassing the smartest human, I believe it is highly likely to be achieved next year or within two years