Exa CEO: We are on the eve of AGI madness, programmers are entering the "paradise" era, and the most impacted are mathematicians

Wallstreetcn
2024.12.26 06:56
portai
I'm PortAI, I can summarize articles.

AGI has arrived, and the world is going crazy! Do mathematicians only have 700 days left of dominance? Are programmers about to enter a golden age? Exa CEO posted: In the short term (1 year), we will achieve "peak models" that basically reach AGI levels in mathematics, programming, and general reasoning, but the novels produced will be quite ordinary

Will Bryk, CEO of Exa, shared some of his thoughts on the future of AGI after witnessing the progress of the o3 model. Here is a summary of his posts on social media:

AGI is really here

This week, I talked with a few friends about the o3 model, and their reactions were basically, "Oh my God, is this really happening?"

Yes, it is really happening. The next few years will be absolutely crazy; this will be a historic, even interstellar-level event.

The absurdity is that there is currently no in-depth discussion about AGI. AI labs cannot talk about it, news coverage is almost nonexistent, and the government does not understand it. We are discussing the future of humanity in a social media meme application’s news feed, which feels like an absurd sitcom, but that is the reality.

Here are some of my thoughts on what is happening—my contribution to the thought abyss of the X platform.

Note that these thoughts are very immature, just some interesting speculations. I do not have enough time to deeply think about/research all these issues, and I am sure I will be wrong in many ways. But I do hope these thoughts are interesting to some who are struggling to understand the current situation.

Enjoy.

The emergence of o3 should not be surprising

OpenAI demonstrated the expanded graph during testing two months ago. The history of computers tells us that no matter how incredible the trend line is, we should believe it. What is truly shocking is that it was achieved in just two months. We have gone from university-level AI to PhD-level AI in such a short time. For humanity, the change is exciting, but rapid change is shocking.

What will happen next is obvious

Models at the o3 level are very good at optimizing anything for which you can define a reward function. Designing reward functions for mathematics and programming is easy, while writing novels is more challenging. This means that in the short term (1 year), we will get "peak models."

They will essentially reach AGI level in mathematics, programming, and general reasoning, but the novels they write will be quite ordinary. While better reasoning abilities will make these models feel smarter in many aspects, they will still fail in stupid ways in areas not covered by their training data without reinforcement learning. In the long term (1-3 years), we will continue to add new domains for reinforcement learning (emotional data, sensory data, etc.) until these blind spots are filled, and then these models will be AGI for everyone except Gary Marcus.

Agents will really arrive by 2025

Models like o3 will inevitably be able to browse/use applications and take action. These things are easy to design reward models for. This is also a huge market—automating computer work—so there is a significant incentive for labs that need to justify their massive expenditures. I guess that by December 2025, you will be able to tell your computer to execute any workflow involving browsing the web/apps and moving data Among all knowledge workers, the most "impacted" are undoubtedly mathematicians.

Mathematicians work in the realm of symbols, and their work rarely interacts with the physical world, thus they are not constrained by it. Large language models are the kings of the symbolic space. Mathematics is not inherently difficult; it is just that primates are not good at it. The same goes for regular expressions.

A big question is how difficult it is to create research-level synthetic data. I guess it won't be too hard. The difference between PhD-level mathematics and researcher-level mathematics seems qualitatively different to us, but to AI, it may just be a matter of degree, requiring only a few more orders of magnitude of reinforcement learning. I give mathematicians 700 days. (This sounds crazy, but it also sounds equally crazy that o6 won't surpass mathematicians, so I am more than 50/50 confident in this prediction, just like all the other predictions in this article). That is, 700 days later, humans will no longer be the leaders in the field of mathematics in the known universe.

What about us software engineers?

In the short term, it will be paradise. Every software engineer will be promoted to a technical lead, doing well. For those who fully adopt LLMs, by the end of 2025, coding will be more about coordinating a bunch of small tasks executed by small agents. Any PR with very clear specifications should be able to be completed by the o4 system, and the error rate will be low enough to be acceptable. One issue here might be that the context window is too small to encompass the codebase, but leaders like Sam are well aware of this.

Will AI soon replace all software engineers? No. Software engineering is not just about creating PRs based on super clear prompts. Unlike mathematicians, software engineers constantly interact with the physical world, that is, with other people. Engineers must work with clients to understand their needs and collaborate with team members to understand their requirements. When engineers design architectures or write code, they are doing so in a vast organizational context. o4 cannot do that. But o4 will help engineers with contextual information increase their speed tenfold.

If the speed of software engineers increases tenfold, do we need fewer software engineers? Well, if you consider a specific company, then yes, they may need fewer software engineers because they can achieve the same output with a more streamlined team. However, the global demand for software engineers may increase because the world certainly needs more high-quality software at ten times the rate. So I think we will see a golden age of applications from more streamlined companies, providing personalized micro-applications for everyone and every business.

In the long run (over 2 years is considered long-term, haha)

Software engineering will be completely different, and it's hard to say what it will look like. When the o6 system exists and is fully integrated into our applications, how could it not change? Roles like front-end engineers may not exist in three years. Is that strange? Not really—there were no front-end engineers 30 years ago either.

We should take a step back and recognize that every generation of software undergoes revolutionary changes. Software has always been about converting requirements into pure logic. The level of abstraction in this conversion process has risen from binary code to Python The current difference is that it is rising to English.

Switching to English allows non-technical people to program as well. But the best builders will always be those who can switch between different levels of abstraction.

In short, because software engineering is essentially about understanding and solving organizational needs through code, the day when software engineering is fully automated will be the day when all organizations are automated.

We talked about some knowledge workers, but what about manual laborers?

AI will also affect you, but at a slower pace because it has to deal with gravity and friction. However, o-type models won't help robots much, as a model that takes an hour to complete is of no use to robots on factory production lines. It is indeed helpful for foundational models to become smarter, and o-type models will assist in training these models, but I don't think this will solve the biggest bottleneck in robotics advancement. I guess the biggest bottleneck is hardware improvements and fast/reliable models for perception + action. Both of these aspects will take longer to improve (i.e., years). Only when robots start making robots and AI begins conducting AI research will we see a crazy rapid advancement in robotics. This may come from o-type models, but I think it will take a few more years.

I have been discussing in terms of years, but maybe we should really discuss in terms of computational power.

Time determines human output, but computational power determines AI output, and AI output will become increasingly important in research institutions. That’s why everyone is racing to build superclusters—Meta's 2GW cluster, xAI's additional 100,000 H100s, etc.

All labs will soon follow OpenAI's lead in testing-time computational models, with some labs initially compensating for algorithmic shortcomings through more computation. They will catch up like GPT-4. To create these models, a combination of common sense and each lab's secret sauce is needed. It is currently unclear how much secret sauce OpenAI used in o-type models, but their speed of improvement suggests it is an algorithmic advancement (easier to replicate) rather than some unique data combination (harder to replicate).

In the era of testing-time computation, I am unclear whether having more computational power or better models is more important. On one hand, you can compensate for a poorer model by investing more testing-time computation. On the other hand, a slightly better model might save an exponential amount of computational power.

It would be interesting if xAI caught up with OpenAI simply because they are better at building large clusters.

In any case, the moat around models will not last more than a year, as labs exchange researchers like trading baseball cards, and perhaps more importantly, researchers from different labs will hang out on weekends and even sleep together. Additionally, I think researchers are too idealistic; if things go awry, they won't hesitate to share information.

Our current situation is truly crazy. The AI race is like a nuclear race, but Americans and Soviets would gather together on weekends in Los Alamos and mock each other on Twitter, "I bet you won't have the biggest nuclear weapon by 2025, haha :)" Before government intervention and/or very bad things happen, the AI race will continue to maintain a hippie and relaxed atmosphere.

O-type models have changed the dynamics of scaling computation in several interesting ways

O-type models incentivize large-scale construction because they show significant returns after each order of magnitude increase in computational power. Computing providers cannot ask for better scaling laws. I suspect that when Sam wants a multi-trillion-dollar computing cluster, this is exactly the law he sees.

But this may not be good news for Nvidia. O-type models make inference more important than training. I think super-optimized inference chips are easier to manufacture than training chips, so Nvidia doesn't have as much of a moat there.

A very bold speculation: What if O-type models could leverage aggregated computing from around the world to train the best models? How cool would it be if open source could beat closed source because we combined our MacBook Pros into an inference supercluster?

Aside from computation, another new exponential growth factor is the code itself

If a lab has unique/privileged access to the smartest models, and thus their software engineers are twice as productive as those in other labs, they can get to the next doubling of productivity faster. Unless the speed of code reaches its limit and there are a lot of experiments to run, the lab's bottleneck will revert back to computation. (I don't know, the dynamics are hard. It would be very cool to see how labs simulate their spending ratios on computation and personnel.)

Despite all this computation construction and knowledge work automation sounding crazy, things will only really get crazy when scientists start to feel AGI

I mean you physicists, chemists, and biologists.

It will start with anything named theory. Theoretical physics is at the forefront. If mathematics is really solved (even writing this sounds absurd, but that doesn't mean it can't happen), then theoretical physics won't be far behind. It also exists in the symbolic realm where LLMs will surpass humans.

What will happen when we have a million AI von Neumann working day and night in the fields of Louisiana (Meta's upcoming data center)? How quickly will they read all the physics papers written by thousands over the past century and immediately spit out more correct labels?

Clearly, this part of the story is hard to predict. Theoretical physics, chemistry, biology—what if these are just child's play for LLMs trained with reinforcement learning? At this point, what reasonable argument do we have that it won't be child's play? Yes, we haven't seen true innovation from these models yet, but they are mostly at high school/college level, and people in those age groups won't invent new physics. We are now at the PhD level, so we might start to see some creativity.

Once AI starts producing new scientific theories, the bottleneck for progress will be testing and experimentation in the physical world The bottleneck there is labor and materials. By then, it would be surprising if there are no robots capable of manufacturing more robots. So the labor issue is resolved. Then materials can be mined by robots. The timeline here will be slow because building/transporting physical items takes a long time, but it will take years instead of decades.

Everything I mentioned above assumes that AI + robot research/development does not introduce new bottlenecks and allows models to learn freely.

This is almost certainly not going to happen. The biggest bottleneck to AI advancement will be humans. I mean regulation, terrorism, and societal collapse.

Governments will not stand by and let the Earth be mined by automated robots operated by a few San Francisco companies (regulation). If the government is too incompetent to stop them, then angry unemployed people may resort to violence (terrorism). Unless people are brainwashed by AI-enhanced media to the point where we cannot function normally as a society (societal collapse).

If war occurs, I think it will not be a bottleneck but rather an accelerator.

Things will get serious. 2025 may be the last year AI becomes a meme in San Francisco tech Twitter, after which ordinary people in suits will intervene, so let’s enjoy our time with roon and sama while we can.

Will this kill everyone?

I am more afraid of humans abusing AI than AI going out of control.

We have 5,000 years of evidence showing that humans use the latest technology to kill each other. The peace after World War II is an anomaly, and once the U.S. missteps or an adversary believes they must strike first to prevent AI from accelerating, that peace could collapse. The risks increase as weapons become more lethal and autonomous.

Another significant risk is that AI causes social chaos. AI-generated media could lead to mass confusion, mass hysteria, and mass brainwashing.

Another risk is AI going out of control. This means it could lead to extinction-level events we did not anticipate. Especially with the return of reinforcement learning, AI is now discovering its own optimization methods rather than trying to match human data (matching humans is safer). But so far, the underlying brain of these models is still LLMs, and LLMs have shown to only understand humans. Just like when you add “make sure not to do anything that could kill us” in the prompt, now you have to bear the burden of proof to show it could still kill us.

I am definitely more excited than afraid.

The kind of sci-fi world I have always wanted is coming. It is arriving a bit faster than expected—hence the fear—but among all possible paths, I am not sure how much improvement the best path will bring. This is a pretty good timeline.

Things I hope to see in the next ten years:

• Some really cool physics discoveries

• Mars and Moon bases initially built by robots

• Perfect mentors/advisors on everything (coming soon, requiring good retrieval, memory, and more personality)

• Zero side effect bio-enhancement drugs

• Flights on super-optimized drones • Achieving comprehensive super clean energy through fusion, geothermal, and abundant solar energy.

• Unexpected things: AI astronomers discovering alien signals in telescope data? AI chemists easily designing room-temperature superconductors? AI physicists unifying some theories? AI mathematicians solving the Riemann Hypothesis?

These no longer sound like science fiction; they feel like scientific realities that are within reach.

So where is all this heading?

Ultimately, we will achieve superintelligence, which means we will obtain anything allowed by the laws of physics. I want immortality and to see other star systems. I also hope to upgrade our bodies to something better. But so far, what excites me the most is understanding the origin of the universe. Ten years ago, I started writing in my diary how much I wanted to know this answer and how AI would help us find it, and now it might really happen; it's crazy.

We now live in a world where all this sounds paradoxical.

Every new development in AI makes more people aware of this, and o3 is the latest example.

Right now, the only reason the future isn't spectacular is that we humans have messed it up. For example, our public opinion, our downstream policies, our social stability, our international cooperation—these are obstacles that could prevent this spectacular future from emerging.

People think that those in AI labs are controlling our future.

I disagree. Their work has already been determined. They are merely executing model architectures that will inevitably appear in some lab sooner or later.

But our public opinion, our downstream policies, our social stability, our international cooperation—these are all completely uncertain. This means we collectively are the guardians of the future.

Each of us has a responsibility to help our world navigate the wild times ahead so that we can have a bright future instead of a dreadful one.

There are many ways to help.

Help build products that stabilize society in some way or make people smarter (e.g., apps that help people regulate social media). Help people understand what is happening (more high-quality social media commentary, a really good search engine, etc.). Help clean our streets so that inviting all of us into a utopian city doesn't look dystopian (get involved in local politics).

Almost everyone I've talked to is afraid of losing meaning in an AI world, and you might be too.

I want to tell you, isn't it the complete opposite? You are living in the most significant moment in history, and you have the power to influence it. Isn't helping to save the world meaningful enough? Do you want to go back to a time when only your career was progressing while the world was not?

Perhaps the shift people need to make is from finding meaning through personal success to finding meaning through collective success. Many of our current jobs will soon be automated. We will have to adapt. If you derive meaning from a specific skill, then yes, that skill may no longer be needed in five years, and you are lucky. But if you can find meaning in doing your best to help the world, that will never disappear For all new graduates who received advice because of o3, my suggestion is:

Learn how to become 1) a highly proactive problem solver and 2) an excellent team collaborator. The specific skills you learn during your studies are not important, as the world is changing too quickly. However, actively solving problems and collaborating well with teams will remain important for a long time.

You may also need to embrace an unstable life in an unstable world. Things will get strange. You might not have two kids and a dog in the suburbs. You might have two half-cyborg children and an AI dog on an interstellar ark.

We are on the eve of AGI, and on this Christmas Eve, I ask you to help ensure a smooth transition to AGI, so that I can greet you on Christmas Eve in the year 3024, on a planet four light-years away, around Ultraman Sentouli Star.

Source: AI Cambrian, original title: "Exa CEO: We are on the eve of AGI madness, coders welcome the 'heaven' era, the most impacted are mathematicians"