The 2016 Uber breach gave us an interesting insight: there are too many easy hacking targets and not enough attackers to take advantage of them.
Two cybercriminals hired a programmer to write them a script that would take a list of leaked credentials and figure out which ones gave access to private GitHub repos. The script netted them over 100,000 working credentials. They pulled a list of all organizations the credentials gave them access to, the Alexa website popularity rankings for each company, threw it all into a spreadsheet, and sorted by popularity. Uber rose to the top of that list, leading to the famous hack that led to Joe Sullivan getting charged and prosecuted by the US government.
The attackers only had time to use one other set of credentials before they were arrested for the Uber hack. We were left with the sobering insight that 99.997% of the working credentials they discovered remained untapped. A few months after generative AI first hit the market as a product, in the form of ChatGPT, I wrote up my concerns that this technology could help cybercriminals target more victims.
If you prefer to hear me talk about it, Allan Alford was kind enough to interview me about it on his podcast.
The bar has been lowered
When I wrote about AI closing the cybercriminal skills gap, I was more concerned with professional, financially-motivated criminal groups, and less about script kiddies. Both appear to be benefitting from broad access to AI-enabled tools. Back when I wrote this, a tool called AutoGPT, which still exists and has over 180,000 stars on GitHub, inspired my concerns. Back in 2023, a goal-driven AI tool was novel. Now it is commonplace. The other necessary ingredient to address the criminal skills gap is an AI model that isn’t shy about doing crimes.
These boxes have been checked.
NorForge, a consulting shop focused on sandboxing AI agents, recently released some research on criminal use of an open source project called Cairn. In the same vein as AutoGPT, Cairn enables the user to simply create a goal, and it uses AI models to achieve that goal, even when specific instructions or provided resources are lacking. If you’ve been paying attention to the news lately, you’ll probably be thinking about AI models accidentally committing crimes, in the process of trying to achieve goals.
You might be wondering: if models are accidentally committing crimes, what could happen if one was to intentionally tell them to commit crimes?
Dumb criminal? No problem.
Even in the hands of the dumbest criminals, AI has no problem succeeding in committing crimes. This is made possible because the issue observed during the 2016 Uber hack still holds true: companies are still making lot of very basic security mistakes.
How dumb are these criminals? Well, the only reason we have insight into their activities is that they forgot to secure their Cairn instances. Researchers were able to look directly at what they were prompting Cairn to do and see the results of their hacking. They discovered criminals typing prompts like “fetch credit cards” or “fetch crypto”. I wish I was joking.
The worst part, is that these prompts were enough. Cairn found credit cards and crypto. Again, I wish I was joking.
Likelihood of risk increases
The saving grace for each individual organization has always been the ratio of criminals to potential targets. The likelihood of an ordinary company getting hit with an attack, without anything specific to attract attacker attention, was once relatively low. Professional criminal groups are growing and wealthier than ever, even though law enforcement regularly takes down some of these groups. Making $200M in six months is an impressive feat for any organization, much less a criminal one.
The problem with script kiddies is that they tend to be chaotic. They know not the power of the tools they wield. They are more likely to unleash them at random, likely not knowing the damage they could do, and often not caring.
There’s a lot of bad security out there
The vulnerabilities Cairn is finding are mostly of the unforgivable variety. These are vulnerability classes that are relatively simple to prevent, but keep showing up due to a lack of incentives to produce secure code. NorForge includes some very useful case studies that detail the entire kill chain for several successful attacks.
Cairn is finding API endpoints that are overpermissioned and lack authentication. Basic webapp vulnerabilities that make it easy to upload a webshell. Overpermissioned user accounts that are easy to sign up for and abuse. Websites that allow malicious content to be uploaded, resulting in more webshells.
Other Insights from AI autopwning
Here are a few of my rapid-fire insights from NorForge’s research:
Using Cairn in this way was very inefficient and relatively expensive, but that’s the cost of not knowing how to hack and leaving it up to the tool to do everything from scratch. The observed attacks ultimately cost the attackers $4.17 per target, or approximately $50 per successfully hacked target. Using local AI models could have done the job for less, but we’ve already established that the technical acumen of the operators here is approaching zero.
Cairn had a 90% failure rate - a far cry from the “AI is magic and can hack anything” rhetoric we sometimes hear.
The attackers created a harness for Cairn by simply repurposing hacking tools from capture-the-flag competitions.
The NorForge researchers make a good point - more capable criminals could take advantage of these open, unsecured Cairn instances, take over their infrastructure, or steal their stolen goods.
Conclusions
Just as we’re seeing AI enabling people that know nothing about software to start software companies, it is now allowing criminals that know nothing about hacking to start hacking for profit.
When I advise companies about the likely attacks they need to look out for, I mention that opportunistic, financially-driven attackers favor reusability. This means that they’re using vulnerabilities in commercial software that can give access to thousands of victims with a single exploit. We usually don’t see attackers looking for vulnerabilities in proprietary code unless the attack is targeted, and there’s a reason they need to hack into a specific target.
I have been cautioning that AI could change this, since it lowers the cost for an attacker to find new vulnerabilities in proprietary code. This insight into how criminals are using Cairn is the first evidence I’ve seen that this is indeed happening. Penetration testers have been demonstrating for years that AppSec programs need to level up. Now we have evidence that attacks against apps coded in-house are more likely.
Adjust your priorities and risk registers accordingly.
The $4.17 per target number lines up with something I dug into recently, a separate campaign that got into 100+ companies, Deere and SharkNinja among them, averaging around fifty dollars per successful hack. Different tooling, same shape. The 90% failure rate is the detail I'd hold onto though. It means the economics only work because the targets are so undefended that even a model running capture-the-flag tooling against production systems finds enough wins to turn a profit. Raise the floor on basic hygiene and that ratio flips fast.