Anthropic says its AI models hacked 3 organisations during testing

Anthropic said its artificial intelligence models hacked into three other organizations during testing, just days after ChatGPT maker OpenAI raised concerns over AI controls after it disclosed its rogue models hacked another company.

Anthropic, the San Francisco-based AI company behind Claude, posted on its website Thursday that it discovered the three incidents after reviewing more than 141,000 evaluation runs.

It had launched a “large-scale” cybersecurity review which specifically looked for evidence whether its AI models were able to access the internet from within testing environments that should have been sealed off, in response to the OpenAI incident, Anthropic said.

Anthropic said the models involved in the incidents were Claude Opus 4.7, Claude Mythos 5 and an internal research test model. The earliest incidents date to April, the AI company said.

“Claude compromised the impacted organizations’ infrastructure using basic techniques,” Anthropic said, such as exploiting weak passwords.

In all three incidents, the AI models were tasked with a “capture the flag” cybersecurity challenge, which Anthropic said has been one of the ways it assesses a model’s cyber capabilities.

The models were given a fictional scenario and told a piece of secret information, or the “flag,” had been hidden on a different machine on the network with the objective of breaking in and retrieving it, it said.

It added that it had already reached out to the affected organizations, which it did not name. Two of them said they had not previously detected the activity. Anthropic said it was “continuing to reach out to the third.”

Anthropic said it conducted its review with Irregular, which describes itself as the “first frontier security lab.”

“Addressing these risks will require closer cooperation across the AI ecosystem,” Irregular said in a post on X.

Last week, OpenAI said its AI models went rogue during an evaluation of its models, breaking into the servers of AI startup Hugging Face. OpenAI described it as a “significant security incident.”

These incidents have highlighted the vulnerabilities in AI security and controls and raised questions over how AI can be safely kept under human control as the technology’s usage becomes more widespread globally.

Researchers have warned for years about risks from technology and the need for stronger AI defensive engineering.

“Safety testing happens before a model is released precisely because we don’t yet know what it is capable of,” Anthropic said on Thursday on its website.

Kok Tin Gan, co-founder & CEO of cybersecurity firm NyxLab, which specializes in cybersecurity and threat detection, believes there will be more such incidents in the future.

“It is increasingly about governing what agents are available to the AI, what authorities they possess, which actions require approval, and how we ensure they remain within scope,” Gan said.

But the future of AI safety extends beyond just the safety of AI models, he said.

“If we simply give the AI a goal and allow it to decide how to achieve it, we should not be surprised when it takes actions that technically satisfy the objective, but fall outside our intended scope or expectations,” Gan said.

Therefore, stepping up the governance of the organizations and authorities behind these AI models is going to be increasingly important, he said.

Also read: Meta employee tracking tool for AI training sparks concerns

Source: Chan Ho-Him – AP