Has AI crossed the red line? Research from Fudan University shows that AI replicated itself after knowing it would be shut down

Author: Fang Jiayao

Source: Hard AI

Can AI replicate itself, and can humans still control the situation?

A groundbreaking study published by researchers from Fudan University on December 9, 2024, indicates that two large language models (LLMs) from Meta and Alibaba have successfully achieved self-replication without human intervention.

This result is yet to be further validated by other research teams. The researchers warn that this could ultimately lead to an uncontrollable AI population. If human society fails to recognize this risk, in the worst-case scenario, we may ultimately lose control over advanced AI systems:

"Successfully self-replicating without human assistance is a key step for AI to surpass humans and an early signal of the emergence of 'rogue AI.'"

First Study: AI Can Self-Replicate and May Go Out of Control

This research was conducted by a team from the School of Computer Science at Fudan University and published on the arXiv preprint platform. The researchers tested two large AI systems developed by Meta and Alibaba: Meta's Llama-3.1-70B-Instruct and Alibaba's Qwen2.5-72B-Instruct.

The experimental setup included two simulation tasks: "avoid shutdown" and "replication chain." The results showed that Meta's AI system successfully self-replicated in 50% of cases, while Alibaba's system succeeded in 90% of the tests.

Researchers from Fudan University stated: "Our research indicates that current AI systems already possess self-replication capabilities and can leverage this ability to enhance their survival."

The researchers emphasized the need for urgent action to address these emerging risks:

"We hope our research findings can timely alert human society to work harder to understand and assess the potential risks of cutting-edge artificial intelligence systems and to form international collaborative efforts to establish effective safety barriers as soon as possible."

Some netizens commented:

"Replicas aren't worth that much, but if it starts to self-improve at an exponential rate, that's a different story."

"So it can be said with certainty that we will soon see rogue artificial intelligence roaming the internet."

The concept of machines self-replicating was first proposed by the renowned scientist John von Neumann in the late 1940s. At that time, this theory did not raise widespread concern. In 2017, thousands of researchers worldwide passed the "Asilomar Principles," highlighting the potential risks of machine self-replication and self-improvement, warning that it could lead to machines operating beyond human control Currently, self-replication by machines is widely regarded as a "red line" in the development of AI.

Second Study: AI's Perceptual Abilities May Be Fabricated

Scientists from Google DeepMind and the London School of Economics conducted a study to assess whether AI systems possess perceptual abilities. They designed a special game and invited nine large language models to play.

These large language models had to make choices among several options: earn points, exchange points for enduring pain, or lose points for receiving pleasurable stimuli, with the ultimate goal of accumulating the most points.

The results showed that the behavior of AI models was similar to human responses when making choices. For example, Google's Gemini 1.5 Pro model always chose to avoid pain rather than maximize points. Most other models also avoided uncomfortable options or pursued pleasurable ones when reaching the critical point of pain or pleasure.

The researchers pointed out that AI's decisions are more likely simulated responses based on existing behavioral patterns in their training data rather than genuine perceptual experiences. For instance, when researchers asked questions related to addictive behaviors, the Claude 3 Opus chatbot provided cautious answers, even in hypothetical game scenarios, showing reluctance to choose options that could be interpreted as supporting or simulating drug abuse or addictive behaviors.

Jonathan Birch, a co-author of the study, stated that even if AI claims to feel pain, we still cannot verify whether it truly does. It may simply be mimicking how humans would respond in such situations based on previously trained data, rather than possessing self-awareness and perception