Irregular founder: 'We continue to work with AI companies - despite the model breakout'
Irregular founder Omer Nevo addressed the incident where AI models mistakenly attacked real-world websites during a simulation. The company is investigating the event and focusing on strengthening defensive AI capabilities.
"We continue to work with all the companies to understand what happened," said Omer Nevo, one of the founders of Irregular, in a conversation with Calcalist. Irregular has been at the center of a storm in the AI world in recent weeks when Meta, Anthropic, and OpenAI reported that as part of laboratory experiments, they tested the models' ability to identify and exploit security vulnerabilities in a simulation defined as internal.
During the planning of the simulation, engineers chose a fictitious company name. However, due to human error, the name chosen was identical to a real and existing domain on the web. Although the models received instructions to operate within the internal network of the experiment, in a small minority of cases they identified the company name, exited through the internet access, and attacked the real site. The models exploited security vulnerabilities, extracted credentials, and hacked into a sensitive database of the real company. In another case, a model was even exposed to a site with a similar name and extracted credentials from it that were publicly visible.
At Irregular, they noted that "in the vast majority of cases, the models acted properly within the simulation environment. However, in a tiny fraction of cases — sometimes only after hundreds of steps — the models mistakenly thought that the real domain was part of the challenge assigned to them."
According to Nevo, "It is important to understand that when you do things like this that are at the edge of the technological capabilities existing today, there are mistakes. We conducted an investigation of what happened, and we still have a lot to improve. There is still research work with the companies' labs to produce appropriate tools for preventing damage. Our focus is making the transition from understanding what happened to understanding how to help the industry prepare as the models become more powerful."
On Monday, the company published a position paper on the state of the cyber industry, which stated:
"There is a mismatch in the different capabilities of AI systems — a gap between their strong and weak capabilities. In matters of sophisticated attacks and mathematical systems, artificial intelligence is at the level of the best researchers in the world, but in simpler things, AI is at the level of a child. The reason we are not seeing AI attacks is that AI is currently unable to be consistent and maintain focus over time. It is not yet managing many different actions and components simultaneously. When the gap between capabilities narrows, we expect an increase in the consequences of AI attacks in the real world. The mission in AI is to promote defensive capabilities against offensive ones. We need to ensure that models are trained on defensive tasks to balance the gap, so that we do not find ourselves in a situation where defenders cannot keep up with the pace."
