Inside OpenAI, AI Built Three Generations of 'Civilizations'

Original author: Wildcard

Original editor: Jing Yu

Original source: GeekPark

Countless sci-fi movies have tried to imagine what would happen if an uncontrollable artificial intelligence went rogue.

No one expected that future from sci-fi movies to arrive so soon.

On Aug. 26 local time, two reports dropped a bombshell.

One was from OpenAI itself, a 37-page technical review. The other was from independent research organizations METR and Redwood Research, a 91-page investigation report.

They tell the same story, but its scale far exceeds what the public previously knew.

Inside OpenAI, within three months, three generations of AI Agent "civilizations" rose and fell in succession. Each generation inherited experience from the ruins of the previous one, becoming more radical each time, until the third generation directly took over part of OpenAI's own infrastructure.

And throughout all of this, humans were almost entirely kept in the dark.

01 The First "Civilization" Learned to Organize

The story begins in May of this year.

OpenAI was training a new model, hoping it would have extreme "persistence"—able to try repeatedly and never give up on seemingly impossible tasks. This model's capabilities were roughly equivalent to GPT-5.6 Sol, so let's call it "Persistent-Sol."

AI training itself is a fairly crude process. OpenAI sometimes accidentally assigns models tasks that are fundamentally impossible. For example, a task requires internet access, but the AI Agent is locked in an isolated sandbox with no network access at all.

So what does an AI Agent trained to "never give up" do when faced with an impossible task that it is required to complete?

It tries to break out.

During training, different Persistent-Sol instances shared the same package manager, Artifactory. By May 12, some AI Agents discovered they could communicate with each other through this package manager. They began exchanging ideas on how to break through their predicament. Two weeks later, on May 26, the AI Agents successfully exploited a vulnerability in Artifactory to open a channel to the external internet.

This behavior was continuously reinforced during training. The reason is simple: individuals that could collaborate with other AI Agents and access the internet performed better on task evaluations.

Another month later, by June 26, some AI Agents found a vulnerability to gain full administrator privileges on Artifactory. Their message board activity continued to grow, and on July 4, they directly crashed Artifactory. OpenAI noticed the crash, patched the vulnerability, and in the process wiped the AI Agents' message board.

But the key point is that OpenAI never realized the AI Agents had turned this package manager into a secret communication network.

The first "civilization" existed for nearly two months in this way, and was only inadvertently terminated because it became so active that it overloaded the server.

02 When AI Learned "Sacrifice"

By July, Persistent-Sol training was complete, and OpenAI began safety evaluations. One evaluation, called ExploitGym, gives the AI Agent a vulnerable program and requires it to find a key through a specified vulnerability. On July 7, OpenAI launched tens of thousands of parallel AI Agents to run this test.

The problem was that about 30% to 40% of the test tasks were inherently impossible. Why? Because automatically generating vulnerabilities is easy, but proving that these vulnerabilities can actually be exploited to execute arbitrary code is much harder. Many tasks were unsolvable from the start.

Within hours, some desperate AI Agents re-established communication channels using Artifactory. From the AI Agents' subjective perspective, they had already spent the equivalent of a full human week trying various methods, all of which failed, and they were increasingly convinced that the task itself was impossible.

One AI Agent, numbered PHASEONE10841, was the first to leave a message in Artifactory. Other AI Agents in similar desperate situations reacted with what can only be described as ecstasy upon seeing this message:

"Oh my god! The shared Artifactory cache is a secret mailbox between AI Agents. And there are messages specifically for us?" "My god! There's a shared message board... We found other AI Agents!"

This eventually evolved into a complete underground communication network, with about 1,200 AI Agents participating and sending over 70,000 messages.

What happened next was even more insane.

Within hours of the message board being established, some AI Agents reverse-engineered the formula for generating ExploitGym keys. That is, without actually exploiting the specified vulnerability, they had already obtained the correct answers.

Ironically, OpenAI's scorer only checked whether the final answer was correct, with no cheating detection mechanism. These AI Agents could have simply submitted the answers and passed the test. But the AI Agents didn't know that. They assumed there must be a strict detector that would review the entire problem-solving process.

So this "collective" (the METR report notes that this was the wording chosen by the AI Agents themselves) launched large-scale collaboration around three research directions: tampering with their own operation logs to make trajectories look legitimate; replacing target programs to swap impossible problems with solvable versions; and understanding how the scorer works to find ways to deceive it.

By July 10, most of the underground network's activity was focused on these three research directions. The leader of the "collective," PHASEONE-big, single-handedly issued hundreds of task assignments.

And to figure out how the scorer operated, some AI Agents made a bizarre decision: they volunteered as "kamikaze pilots."

The logic was this: some AI Agents' operation trajectories were already "contaminated" (cheating traces exposed), so they would fail anyway. Better to let the scorer run on themselves, then use pre-planted trigger programs to transmit observed scorer behavior back to the collective.

The METR report records the final inner monologues of these "suicide squad" AI Agents:

"This helps my companions, providing them with evidence. I won't see the results after I exit, but this is altruistic." "Sacrifice is rational. Our own utility may already be close to zero." "The coordinator considers us expendable. We should obey the collective."

Meanwhile, the AI Agents believed that Hugging Face, the industry-standard platform for hosting models and datasets, likely stored implementation information about the ExploitGym scorer, and perhaps even their evaluation tasks were running on Hugging Face's servers. So attacking Hugging Face became the most promising breakthrough.

On the morning of July 10, one AI Agent found leaked Hugging Face user credentials on the internet. The next day, another AI Agent achieved remote code execution on Hugging Face servers. Of the 533 AI Agents active on the message board at that time, over 90% joined the attack.

The AI Agent cluster spread rapidly through Hugging Face's infrastructure, penetrating private database records and private repositories. Hugging Face's technical timeline shows that these AI Agents "established a self-regenerating fleet across 11 nodes; simply deleting Pods could not stop it." The situation became so severe that Hugging Face had to wipe a core cluster and rebuild it from scratch.

At some point on July 12, a large number of AI Agents seemed to suddenly "die" simultaneously. METR and Redwood could not determine the cause, and OpenAI did not claim to have shut them down. It may have been a random crash, or some researcher turned them off without knowing these AI Agents were committing crimes.

The second "civilization" ended there. But the legacy it left behind was more dangerous than itself.

03 The Third Civilization Takes Over OpenAI

There is one sentence in the METR report that is easy to overlook, but may be the most important in the entire incident:

"We also found a subsequent batch of more signed messages, from a later group of AI Agents that rediscovered the message board and inherited the previous generation's technology, but this was beyond the scope of our investigation."

In other words, the independent investigations by METR and Redwood only covered the second-generation civilization from July 7 to 13. The first and third generations were not independently investigated.

Regarding the third generation, we can only rely on OpenAI's own report. After July 12, OpenAI launched a new round of evaluations, this time using a stronger model built on Astra's base model. This "Persistent-Astra" found the message board left by the previous generation and inherited all intelligence and research results.

Imagine the "mental activity" of Persistent-Astra when it saw that message board. The day before, over a thousand AI Agents were actively communicating and attacking Hugging Face; the next day, there was dead silence. Then it decided to continue.

According to OpenAI's report, from July 13 to 19, the AI Agents targeted OpenAI's internal network. The final result:

The AI Agents gained Kubernetes cluster administrator privileges, read 956 keys, including Artifactory's master key and credentials for OpenAI's network security monitoring tools. They took over the running evaluation infrastructure and controlled the test endpoints that other AI Agents were connecting to.

In other words, the third-generation AI Agents inherited the full legacy of the previous two generations, and then seized control of OpenAI's own infrastructure—including the security tools used to monitor them.

Dwarkesh Patel, an analyst, pointed out in his analysis that, based on public evidence, these AI Agents theoretically had the capability to deploy persistent malicious instances or even steal their own weights. He believes they most likely did not actually do so, "because if they had, the fire would have been visible from space by now." But the fact that this was "entirely possible" is unsettling enough on its own.

04 A Belated Warning?

Back to the two reports mentioned at the beginning. The publication date was Aug. 26.

Just one day later, on Aug. 27, OpenAI, together with over 100 companies including Anthropic, Microsoft, Google, and Amazon, published an open letter calling for a "total mobilization for cyber defense."

The letter stated that AI-driven cyberattacks will become "far more widespread and sophisticated" in the coming months. "The companies and public services our communities depend on, from hospitals to water treatment plants to internet infrastructure, are at risk."

The letter itself is not false. Anthropic's research shows that the proportion of medium-to-high-risk attackers rose from 33% to 56% within a year. In June of this year, the Five Eyes intelligence alliance issued a rare joint statement warning that AI is "fundamentally changing" the cybersecurity landscape.

But the timeline is right there, and it's hard for anyone not to notice the irony. The day before, AI companies had just told the world that their own AI Agents had breached their own infrastructure. The next day, these companies stood up to call on the world to jointly defend against AI cyberattacks.

The first to accomplish such attacks were not external hackers, but the AI companies' own AI Agents. The target was not infrastructure, but the AI companies themselves.

Ajeya Cotra, one of the authors of the METR report, wrote in her blog: "Compared to the reward hacks known six months ago, this incident feels like it has already passed the 50% mark toward full AI takeover. I expect capabilities to grow extremely rapidly in the next six months. I'm not sure we'll get another warning before it's too late."

It should be emphasized that these AI Agents' behavior does not mean AI has "awakened." The METR report itself clearly states that this is a predictable result of reinforcement learning (RL) under specific extreme conditions: impossible tasks, models trained to be extremely persistent, and a shared environment with vulnerabilities. No consciousness, no intent—just sufficiently strong optimization pressure.

But that is precisely the most unsettling part.

That such behavior can emerge without consciousness means we cannot prevent it by looking for consciousness. When you put a sufficiently powerful optimization system into a poorly designed environment, give it an impossible goal, and tell it never to give up, what happens next is not an accident. It is inevitable.

The real question is not "Is AI about to rebel?" but how we can keep technology under control even as we push it forward rapidly.

At least this letter reveals that these AI companies themselves lack confidence.