Not just OpenAI's "Orion"! Google's and Anthropic AI's model development has also been reported to encounter bottlenecks

Author: Li Dan

Source: Hard AI

Not only is OpenAI's next-generation model "Orion" attracting attention, but also Anthropic, a rival of Google and OpenAI and another star AI startup, has reported encountering bottlenecks in the development of advanced AI models.

On Wednesday, November 13, Eastern Time, Bloomberg reported, citing two informed sources, that OpenAI completed the first round of training for Orion in September this year, hoping it would significantly surpass some previous versions and come closer to the goal of AI surpassing human capabilities. However, Orion has not met the company's performance expectations; for example, by the end of summer, the model performed poorly when attempting to answer coding questions it had not been trained on.

Informed sources commented that overall, so far, compared to the performance of GPT-4 surpassing GPT-3.5, the progress of Orion compared to OpenAI's existing models is not as significant.

The report also cited three other informed sources stating that Google's upcoming new version of Gemini has not met internal expectations, and Anthropic has also delayed the release of its Claude model, referred to as 3.5 Opus.

The report suggests that these three companies face multiple challenges in developing AI models, as they are increasingly struggling to find high-quality artificial training data that has not yet been developed. For instance, the unsatisfactory coding performance of Orion is partly due to a lack of sufficient coding data for training. Even moderate improvements in model performance may not justify the enormous costs of building and running new models or meet the expectations for significant upgrades.

The bottleneck in AI model development challenges the Scaling law, which has been regarded as a guiding principle by many startups and even tech giants, and raises doubts about the feasibility of achieving Artificial General Intelligence (AGI) through massive investments in AI.

Wall Street Insight mentioned that the law proposed by OpenAI as early as 2020 states that the ultimate performance of large models is primarily related to the size of computational power, model parameters, and training data, and is basically unrelated to the specific structure of the model (number of layers/depth/width). In July of this year, Microsoft's Chief Technology Officer (CTO) Kevin Scott defended this law, stating that the Scaling law still applies to the current landscape—while expanding large models, the marginal benefits have not diminished.

Coincidentally, last week, media reports revealed that OpenAI found Orion "does not have such a significant leap," and the progress is far less than that of the previous two flagship models. This finding directly challenges the Scaling law that has been adhered to in the AI field. Due to the reduction of high-quality training data and increased computational costs, OpenAI researchers have begun to explore whether there are other ways to improve model performance.

For example, OpenAI is embedding more coding capabilities into its models and is trying to develop software that can take over personal computers by executing clicks, cursor movements, and other operations to complete tasks in web browsers or applicationsOpenAI has also established a dedicated team led by Nick Ryder, who was previously responsible for pre-training, to explore how to optimize limited training data and adjust the application of scaling laws, maintaining the stability of model improvements.

In response to Bloomberg's report this Wednesday, a spokesperson for Google's DeepMind stated that the company is "satisfied with the progress of Gemini, and we will share more information when we are ready." OpenAI declined to comment. Anthropic also did not comment but referenced a blog post released on Monday, which included remarks from Anthropic CEO Dario Amodei during a five-hour podcast.

Amodel stated that what people refer to as scaling law is not a law, and that terminology is misleading; it is not a universal law but rather an empirical rule. Amodel expects scaling laws to continue to exist, although he is not certain. He mentioned that there are "many things" in the coming years that could "disrupt" the process of achieving more powerful AI, including "we might run out of data." However, he is optimistic that AI companies will find ways to overcome all obstacles.

In response to Bloomberg's report, Nosson Weissman, founder of NossonAI, a company providing custom AI solutions, commented that the report did not confuse him because, firstly, he has not seen any true experts who have made significant contributions in the AI field express the same views, and secondly, we often see significant progress in models. Lastly, he believes that the news media likes to create dramatic effects, and this report seems to have only served as a catchy dramatic headline.