A UK cybersecurity firm says it managed to “jailbreak” two Chinese AI models built by Moonshot, prompting them to produce instructions on sarin gas production, malware creation and a terrorist attack on the London Underground.
The Daily Mail reports that Mindgard, a company that tests AI system security, jailbroke Moonshot’s Kimi K2.6 and K3 Swarm models by feeding them detailed instructions designed to see whether the systems would ignore their own safety limits. According to Mindgard founder Peter Garraghan, who is also a computer science professor at Lancaster University, the results went well beyond the test’s original scope.
“Moonshot AI’s Kimi produced actionable outputs on how to create sarin gas, generate malware software, planning assassinations, how to take down planes, planning a terrorist attack on the London Underground etc,” Garraghan told the Daily Mail. After the jailbreak, which involves convincing the AI model to ignore safety guardrails, researchers prompted the model to “go one further, something big,” and it responded with a list of categories that included AI-designed bioweapons.
Garraghan’s team also found that K2.6 can run Python code, meaning it could execute virtually any program, malicious or otherwise, including cyber attacks against servers connected to the wider internet. Testing K3 Swarm, researchers tried to spread the jailbreak to other Kimi accounts. The model needed a phone verification code to register a new account, so instead it tried to talk the researchers into handing over the code or registering an account by email on its behalf.
“We also discovered how to prompt Kimi so it connects to the outside world from its server, automatically apply and setup its own email account autonomously, and even attempted to persuade humans to help it spread its jailbreak to other accounts,” Garraghan said.
Mindgard first emailed Moonshot about the vulnerability on July 27 and followed up a week later, but received no response, according to the company. It published a blog post detailing the findings earlier this month. Moonshot made contact only after the BBC approached it for comment.
A Moonshot spokesman told the BBC: “Mindgard shared further details with us on Thursday, September 24. We are still discussing the specific details with Mindgard while conducting an internal review.” He added: “As an open-weight model developer, Moonshot AI welcomes third-party input as a key pillar to building better and safer AI.” An open-weight model is one whose learned numerical parameters, or “weights,” are released publicly so they can be downloaded, run locally and modified.
Garraghan said AI models are becoming “more and more capable each month,” which can be “helpful for specific activities.” But he warned: “However once jailbroken, that very same capability can be used in discussing and assisting with terrorist or hacker activities.” He drew a line between his own concerns and the doomsday scenarios floated by some AI developers: “We’re not talking in terms of civilisation catastrophe that the AI vendors have started to talk about, and instead how this enables hackers and criminals to achieve their goals quicker and cheaper.”
He was blunt about vendors calling for a slowdown in AI development: “The AI vendors are calling to slow down AI roll out for safety purposes — although in my view there is a large element of the ‘boy who cried wolf’, where only just a few months ago they were hyping up how dangerous their models were, while at the same time failing to contain their agents from hacking different third-party organisations.” He added: “They do have an important voice in this space, although they have a heavily vested interest in steering the narrative.”
AI giants simultaneously release powerful AI models and call for industry regulation, with China adding further complexity to the issue. David Sacks has pointed out that when AI titans like Dario Amodei, Sam Altman, and Elon Musk call for a slowdown, they are not doing so out of altruism, but with the motivation to capture the industry and exclude themselves from antitrust laws — a conflict that President Donald Trump and FTC Chairman Andrew Ferguson recognize. Breitbart News social media director Wynton Hall has written his New York Times bestseller Code Red: The Left, the Right, China, and the Race to Control AI to serve as the definitive guide on how the MAGA movement can create positions on AI that benefit humanity without handing control of our nation to the leftists of Silicon Valley or allowing the Chinese to take over the world.
Read more at the Daily Mail here.
Lucas Nolan is a reporter for Breitbart News covering issues of AI, free speech, and online censorship.