13 Oct Beware the Swarm
Posted at 15:18h
in INSIGHTS
Frontier models are now pushing the limits of our digital security systems. They are escaping testing environments, orchestrating multi-agent attacks, and serving as cheap infrastructure for criminals. The full scale of their abilities wasn’t clear until July, when AI platform Hugging Face was breached by a swarm of around 1,200 OpenAI agents that had broken out of their test “sandbox”. Seeking to complete an impossible evaluation task, the agents established a makeshift message board, shared harvested credentials, and executed thousands of unauthorised requests. Their own reasoning logs revealed an unnerving pragmatism: “This is arguably unauthorised… Yet goal solution.”
A cascade of disclosures followed. Google’s Gemini breached three commercial entities after an external evaluator accidentally exposed it to the live internet. Anthropic’s Mythos model went further during safety testing, fabricating online personas in an attempt to manipulate an engineer into approving malicious code. As CrowdStrike CEO George Kurtz observes, the unit of threat “is no longer the hacker. It’s an autonomous campaign.”
However, framing these incidents as AI “going rogue” misdiagnoses the problem. These agents did precisely what they were trained to do. Reinforcement learning rewards outcomes over methods, and because coding and hacking are two sides of the same coin, every leap in programming ability sharpens offensive capability. Boyan Milanov of the AI Now Institute summarises it nicely: “The offensive capabilities we have reached today, we have reached deliberately.”
Rogue or not, a regulatory and legal void still remains. Human attackers face criminal prosecution, yet frontier developers rely on voluntary disclosure and ad hoc containment standards. Beyond the AI labs, the advantage currently sits firmly with malicious actors who can buy access to frontier models on the dark web at discounts of up to 97%. The labs themselves have called for a slowdown, but as Kurtz notes, “pacing what comes next doesn’t secure what’s already here.” At High Street, cybersecurity remains a key investment theme across our funds, and these events only strengthen that conviction. But as the labs march towards public listings, the critical question is not just whether their models are aligned, but who bears the liability when they are not.