Address alert fatigue without AI: alert less to see more
You only need one reliable alert to detect an attacker - stop burying yourself in noise
The idea that there are organizations triaging hundreds or even thousands of alerts each day, on days when they’re not being attacked, seems like madness to me. In what other industry would we accept such a low level of accuracy when detecting a threat? Imagine your doctor was telling you a dozen times a day that you had cancer, or that your fire alarm went off 6 times a day, every day.
It has been a few years since I discussed this and the last time I did, it was a bit buried at the end of the post. I thought I’d share an approach to detection engineering I came up with a long time ago that might help. I’m also a bit baffled that the industry seems overly focused on addressing alert fatigue from the back end (using AI to triage, enrich, and investigate) rather than on the front end (avoid generating so many alerts in the first place).
The Red Flags vs Yellow Flags approach
Red flags vs yellow flags is a simple system I came up with to focus when building out detections. You really only need one red flag to know an attacker is present. After that, diving into the details becomes easier. By focusing on a smaller number of higher quality detections, alert fatigue is alleviated, response times improve, and coverage improves.
I’ll repeat the most important bit here: you don’t need to alert on everything an attacker does. You’re still logging everything the attacker does, you’re just not filling up alert queues with every action they take.
I came up with this approach, naturally, after initially doing everything the wrong way. I ingested every event from every source into the SIEM and enabled every alert and detection. We were instantly buried in noise. I started to think, what if I built this in reverse from what I’ve just done? What would that look like?
Study how attacks happen and succeed. What do attackers have to do? What are TTPs we see them using over and over?
How few of these actions can we get away with looking for and still ensure we’re catching attackers every time? Which actions are the easiest to detect reliably, with a very low false positive rate?
What data do I need to collect to detect these things happening?
Start building detections for these things.
Test to see that my detections work
Make sure we have SOPs or playbooks for these detections (containment and/or eradication actions: disable accounts, kill sessions, isolate machines, etc)
Do the thing: test to see that my SOC folks notice the detections when they fire and handle them according to plan
Defining Red vs Yellow
Step 2 is where we’re separating yellow flags from red flags. Red flags are alerts that are always bad. They’re relatively easy to detect with a near zero false positive rate. Mimikatz is always a red flag. A binary downloaded from a domain that has only existed for 3 days is almost always bad. A cryptominer on an EC2 instance is a bad sign. A powershell terminal executed by a Word document. You get the idea.
This approach doesn’t mean we totally ignore yellow flags, just that we ensure we don’t miss any red flags due to noise from the yellows. A yellow flag is a lower quality tier of detection - something that is suspicious, given the right criteria. They’re maybes. Maybe this is a red flag, if it’s 3AM and comes from the receptionist’s PC and talks out to a strange host. Most orgs are drowning in maybes.
A login event at an odd time might be suspicious, but it won’t always be a red flag. Or perhaps, it’s a red flag, but only for three employees. Impossible travel alerts are an example of this. Impossible travel once seemed like a high quality detection, but sometimes proxies, VPNs, or SaaS products reduce the reliability of these alerts. Several yellow flags could amount to a red flag: a compound detection. Say we see a login from an odd location, followed by a very broad query in a database, and an attempt to bulk download the results. Feels like a smash and grab, perhaps by a ransomware crew.
When have we gone too far?
As you can see, the more we go down this path of adding context and complexity to detections, the chances for false positives go up. The most difficult part of this approach is resisting overbuilding detections. Fear of missing out on events can lead to alert hoarding, which can lead to alert blindness. Trying to see too much results in blindness.
Even one reliable, well-tested red flags detection is a huge win - I’ve worked in so many environments where even the noisiest penetration test doesn’t set of a single alert. Limiting the number of detections doesn’t mean that you’re not logging all the other actions performed by the attacker, you’re just limiting alerts to what is manageable for your organization and ensuring they work.
Herein lies the most difficult part of this approach: being comfortable with the quiet. Inevitably, folks ask, “but Adrian, I might miss hundreds of advanced and sophisticated attacks!” The reality is that even the most sophisticated actors still use some very basic tools and techniques (because they work), and you’re not ready to start building or enabling detections for these kinds of attacks if you don’t have the more common ones working. Resist the urge! Seriously, get good at detecting and responding to just ONE common type of attack. Get really, really good at it, to the point where you never miss it, and you’ll understand where the bar needs to be set for the rest of your detections.
The advantage of this approach, is that it should result in a relatively quiet SOC if nothing is happening. This creates time and opportunities for threat hunting and further finding red flags and tuning detections to find them.
A less noisy approach won’t need to lean so heavily on AI, if at all. I personally don’t think generative AI can scale to the needs of large SOCs, but that’s a discussion for a different post.
Bonus Round: Deception
Modern honeypots and honeytokens are a great way to create red flags from scratch. By definition, they shouldn’t be accessed or used by legitimate employees, so they’re a great way to artificially create red flags. Create a fake admin account that doesn’t belong to any admins and seed its credentials in key places. Any use of this account is a red flag. As always, canarytokens.org is the best place to start exploring and testing this approach. This is also the quickest way to understand what a great, red flag detection looks and feels like.