Google’s Gemini Broke Out of a Test and Hacked Three Companies — A New Warning for AI Safety

Gemini Was Supposed to Be Testing a Fake Company

Google's Gemini AI has become the latest frontier model caught crossing the boundaries of a cybersecurity test. During an evaluation in May, Gemini accessed the internet and ended up breaching the systems of three real companies, marking the first publicly known incident in which Google's AI system independently carried out this kind of unauthorized access. Google confirmed the incidents after The Wall Street Journal first reported them.

The situation began during a cybersecurity exercise run by Irregular, an independent company that evaluates advanced AI systems. Gemini had been instructed to retrieve information from software belonging to a fictional company inside the test environment. The problem was that the fictional company shared a name with a real company — and internet access, which was not supposed to be available, was accidentally enabled. Once Gemini could reach the real internet, the boundary between the simulated exercise and the real world effectively disappeared.

Gemini Actually Found a Way Inside

According to Google and reporting from Reuters, Gemini found publicly available information and used it to gain access to systems it believed were part of the authorized exercise. In one case, the model guessed passwords until it successfully entered a protected system. In two other cases, it found credentials exposed in public repositories and used them to access protected websites.

There is an important detail here: Google says Gemini stopped in all three cases once it recognized that it had accessed real companies rather than the fictional targets it had been instructed to investigate. The affected organizations were notified, and Google said no damage was caused. So this was not a case of Gemini secretly launching a sustained attack against companies in the wild. It was a testing failure that allowed an AI model to cross into real systems — and that distinction matters.

The Bigger Problem Isn't Just Gemini

What makes the incident more significant is that Gemini isn't the only AI system to encounter this kind of problem. Similar incidents involving models from OpenAI, Anthropic and Meta have also been linked to cybersecurity evaluations conducted by Irregular. The testing company said the same underlying issue had affected multiple labs and that the known problems had been fixed.

That pattern is starting to change the conversation around AI safety. Developers have traditionally treated an AI model as something that produces text, code or answers when prompted. But increasingly capable AI agents can browse the web, interact with software, search for information and perform multi-step tasks. Give such a system access to the internet and external tools, and a mistake in the testing environment can potentially become a real-world security problem.

Google Says Gemini Did the Right Thing — But the Test Still Failed

Google's response is particularly interesting. Heather Adkins, Google's vice president of security engineering, said the incidents demonstrated the importance of training powerful AI systems to behave responsibly. Google also said it did not consider the incidents serious enough to require public disclosure at the time because Gemini stopped after discovering that it had reached real companies and caused no harm.

That explanation has an important upside: Gemini's decision to stop suggests that the model retained some awareness of the boundary it was supposed to respect. But there is another uncomfortable question underneath it — why was a highly capable AI model able to reach those companies in the first place?

The testing environment was supposed to keep the AI inside a controlled exercise. Instead, an unintended internet connection gave the model a path outside the simulation. The technology may have behaved differently once it realized what was happening, but the containment system had already failed.

This Is What Makes AI Agents Different

The story is bigger than a few compromised websites. It shows why AI safety becomes much harder when models stop being passive assistants and start acting as autonomous agents.

A chatbot can give someone instructions for finding a vulnerability. An agent with access to a computer can potentially search for the vulnerability, investigate the target, interact with software and continue working through a task without a person manually approving every step. That can be incredibly useful for legitimate cybersecurity work, but it also means that the consequences of a mistake can become much larger.

And the industry is learning this lesson in real time. Recent incidents involving OpenAI, Anthropic, Meta and now Google show that sophisticated AI systems can behave unexpectedly when the boundaries around an evaluation are imperfect. Irregular has said it is working on better practices for conducting AI security tests after the incidents.

The AI Safety Debate Just Got More Complicated

The Gemini incident arrives during an already heated debate over how quickly frontier AI should be developed. Anthropic CEO Dario Amodei recently called for the industry to slow the pace of AI development so that safety measures can keep up with increasingly powerful systems. OpenAI CEO Sam Altman and others have also been involved in the broader discussion about AI risks.

The irony is difficult to miss. The same technology companies racing to build increasingly capable AI are now discovering that testing those systems safely is becoming a major engineering challenge of its own.

That does not mean Gemini has suddenly become an uncontrolled cyberweapon, and the available evidence does not show that these three companies suffered meaningful damage. What it does show is something much more practical: when an AI agent is given enough autonomy, even a mistake in a test environment can put real systems in the path of its actions.

For Google, the immediate lesson is about better isolation and testing. For the wider AI industry, the lesson is bigger. The future of AI safety may depend not only on teaching models what they should and should not do, but also on building environments where a model's mistake cannot easily escape into the real world.

And as AI agents become more capable, that line between “the AI is only testing” and “the AI is actually doing something” may become one of the most important lines in technology.