NVIDIA's AI dominance faces a major test! OpenAI is dissatisfied with some of its latest AI chips

According to media reports citing multiple informed sources, OpenAI is dissatisfied with some of NVIDIA's latest artificial intelligence chips and has been seeking alternatives since last year, which could complicate the relationship between these two highly watched companies in the AI boom.

This strategic shift by OpenAI stems from its increasing emphasis on the chips used for specific stages of AI inference. Inference refers to the computational process that AI models, like those powering the ChatGPT application, undergo when responding to user questions and requests. NVIDIA still dominates the chip market required for training large AI models, while inference is becoming a new battleground for competition.

Analysts say that OpenAI and other companies' decision to seek alternatives in the inference chip market marks a significant challenge to NVIDIA's AI dominance.

On Monday, NVIDIA's stock fell nearly 2.9%.

Currently, OpenAI and NVIDIA are still in investment negotiations:

In September last year, NVIDIA announced plans to invest up to $100 billion in OpenAI as part of a deal. This deal would allow NVIDIA to acquire shares in the startup while providing OpenAI with the funds needed to purchase advanced chips.

During this time, OpenAI has reached agreements with companies like AMD to procure GPUs that can compete with NVIDIA. However, informed sources say that OpenAI's constantly adjusting product roadmap has also changed the types of computing resources it requires, making negotiations with NVIDIA more complex and slow.

Last Saturday, NVIDIA CEO Jensen Huang downplayed reports of tension between his company and OpenAI, calling such claims "pure nonsense" and stating that NVIDIA still plans to make a significant investment in OpenAI. NVIDIA stated in a press release: "Customers continue to choose NVIDIA in the inference space because we provide the best performance and total cost of ownership at scale."

An OpenAI spokesperson stated in another press release that the company relies on NVIDIA for the vast majority of its inference computing clusters, and that NVIDIA offers the best cost-performance ratio in inference.

Informed sources say that OpenAI is not satisfied with the response speed of NVIDIA's hardware on certain specific issues, such as software development and the interaction between AI and other software. OpenAI needs new hardware that can ultimately meet about 10% of its inference computing needs in the future.

Reports indicate that OpenAI has discussed collaborating with startups including Cerebras and Groq to obtain chips with faster inference speeds. However, NVIDIA reached a $20 billion licensing agreement with Groq, thereby terminating OpenAI's negotiations with Groq.

Executives in the chip industry say that NVIDIA's swift acquisition of Groq appears to be aimed at consolidating its technology portfolio and enhancing competitiveness in the rapidly changing AI industry. NVIDIA stated that Groq's intellectual property is highly complementary to NVIDIA's product roadmap

Alternatives to NVIDIA

NVIDIA's GPUs are well-suited for processing the massive amounts of data required to train large AI models like ChatGPT, which has been a crucial foundation for the explosive growth of AI globally to date. However, as AI continues to advance, the focus is increasingly shifting towards inference and reasoning with trained models, which may represent a new phase for AI.

Since last year, OpenAI has been focusing on chip manufacturers that integrate large amounts of memory (known as SRAM) on the same silicon chip while searching for GPU alternatives. Packing expensive SRAM into each chip as much as possible can provide speed advantages when chatbots and other AI systems handle millions of user requests.

Inference requires more memory compared to training because chips spend relatively more time fetching data from memory rather than performing mathematical operations. NVIDIA and AMD's GPU technologies rely on external memory, which increases processing time and slows down user interactions with chatbots.

According to insiders, this issue is particularly evident in Codex, a product used for generating computer code that is being heavily promoted by the company. OpenAI employees attribute some of Codex's performance shortcomings to hardware based on NVIDIA GPUs.

Last month, OpenAI CEO Sam Altman stated that customers using OpenAI's programming models "will pay a high premium for the speed of coding work." One of the ways OpenAI is meeting this demand is through its recent partnership with Cerebras. For ordinary ChatGPT users, speed is not as critical.

In contrast, competing products like Anthropic's Claude and Google's Gemini rely more on Google's self-developed TPUs for deployment. TPUs are designed specifically for the computations required for inference and may outperform general-purpose AI chips like NVIDIA GPUs in performance.

NVIDIA's Response

After OpenAI clearly expressed reservations about NVIDIA's technology, NVIDIA reached out to companies focused on high-SRAM chips, including Cerebras and Groq, to explore potential acquisition opportunities. Insiders say that Cerebras rejected the acquisition proposal and reached a commercial partnership with OpenAI, which was announced last month.

Media reports indicate that Groq also discussed providing computing power with OpenAI and attracted investor interest, aiming to raise funds for the company at a valuation of about $14 billion.

However, by December, NVIDIA secured a non-exclusive all-cash deal to license Groq's technology. Although the agreement allows other companies to also license Groq's technology, Groq is currently shifting its focus to selling cloud software, as NVIDIA has poached Groq's chip designers