Cybersecurity - A Market for Lemonade

What else are you going to do with all these cyber lemons?

George Akerlof’s seminal economics paper, “The Market for ‘Lemons’: Quality Uncertainty and the Market Mechanism” is based on how information asymmetry between buyers and sellers can hurt the buyer. These markets can become more common when it is difficult for the buyer to determine the quality of something, because it:

1. requires expert knowledge

2. requires special tools, or

3. requires access the buyer doesn’t have

The cybersecurity market takes this to an extreme: sometimes, even the sellers aren’t aware of the quality of their product. Ross Haleliuk said it best in his book, Cyber for Builders.

“The quality of what is bought and sold is not known.”

When evaluating, buying, and using cybersecurity products, it can be difficult to determine how well a product works. It isn’t uncommon to discover security products that weren’t even functional after being deployed. Another challenge is determining how effective the product is.

No one is (or should be) satisfied with simply blocking EICAR, for example. If an anti-virus product is only good at blocking commodity malware but no self-respecting cybercriminal will ever use commodity malware, this product isn’t useful for much more than a compliance checkbox.

FireEye, it had what networks crave

Back in 2012, I was building a security program for a large enterprise. I spent the better part of a month trying to figure out what FireEye’s product did. This was before FireEye went public and their only product was the NX appliance. Every time a salesperson pitched me the product, their description brought me to the same conclusion: “so, this is intrusion detection? This is an IDS/IPS?”

The salesperson quickly pushed back on the description. “Oh no, I was told in no uncertain terms that our product is NOT an IDS or an IPS device.” This was unsurprising, as Gartner had declared IDS dead as far back as 2003 (it is still commonly used today). We’d loop back to the beginning of the conversation and they would explain it to me all over again.

After chatting with a third sales rep at FireEye, it became obvious that their own employees (or at least their sales teams), didn’t seem to understand what the product did.

The fourth time was the charm. I finally got an engineer on the line and was told that the FireEye NX appliance was designed to catch malware on the wire, but only really bad malware. It would ignore the commodity stuff. Long story short, my org bought a heavily discounted NX appliance, despite my recommendation to pass on it.

We pulled it out of the box, racked it, plugged it into a SPAN port (no chance were we going to put it in-line) and powered it on. The first challenge was how to determine whether or not it was working.

How do you test a product that only detects ‘really bad’ malware? What even is ‘really bad’? How does it determine bad from not-so-bad malware? FireEye had a solution: a custom PDF that, like EICAR, was guaranteed to trigger an alert every time. That solved the problem of functional testing. Would it be effective though? Would it actually detect malware?

It did not. FireEye generated roughly one false positive per month. In the same time, we were hit with a few malware infections per month. Malware got past the FireEye appliances by shipping as a JAR file that contained an obfuscated WinPE that it would reassemble on the desktop, after it got past our watchful FireEye’s network surveillance. This was one of many malware delivery evasions that worked over the years to bypass security products.

The product didn’t just lack value, it had negative value. I calculated that the cost of the product itself, plus the time it wasted every time we had to respond to a false positive and investigate it, put FireEye’s value over six digits in the red.

Another problem was that, due to the cost of the appliance, we could only afford to purchase one, and put it in our headquarters, where we had the least problem with malware infections. Most of our issues were at sales offices in the field, that often had no more than 5 employees per office. The whole idea for the device was fundamentally flawed.

This would become even more obvious years later as we saw the company struggle with heavy churn (customers not renewing contracts). I was an industry analyst at this point and referred to this event as FireEye Buyers’ Remorse. This was unsurprising, given my experience with the product, and my theory that both the seller and buyer didn’t seem to understand what the product did. The NSS Labs/FireEye battle a few years later further proved this theory.

The FireEye story is a long-winded way to point out that the Cybersecurity market produces a lot of lemons. The average buyer doesn’t have the security talent in house necessary to simulate a real attack and determine if there’s any value in buying these kinds of products. They can maybe afford to pay for a penetration test once a year, but unless they’re paying for the best in the business, they’re probably going to get someone a few years out of school operating mostly automated tools.

Security products aren’t all about preventing attacks, however. We have security operations products, GRC products, application security products, vulnerability management products, and more. Most suffer from similar information asymmetry issues that hark back to The Market for Lemons.

It is not practical for most companies to properly evaluate and test security products before buying them.

When life gives you lemons

Cybersecurity products also have this nagging tendency to create entirely new problems. A large portion of the market pumps out what I think of as busywork generators. Right out of the box, these products will consume logs, scan for vulnerabilities, detect potential attacks, and identify compliance gaps. Overnight, security analysts can be buried in hundreds of thousands or even millions of tasks.

You’ve probably heard the terms ‘alert fatigue’ or ‘vulnerability overload’. This problem, again, has its roots in information asymmetry. The vendors say, “this stuff is bad”, and many buyers aren’t in a position to refute or challenge it. Meanwhile, security products fail to stop threats, detect attacks, or alert practitioners when products are misconfigured. If you’ve been a practitioner for even a year or two, you likely have examples you can cite. Here are a few of mine:

1. When I first used a SIEM in 2004, I was floored to discover that there was no mechanism to tell me when devices suddenly stopped sending logs. I had to build it myself. Again, in 2012, I found nothing had changed. SIEMs still didn’t perform this basic, fundamental task.

2. I once discovered that a client’s DAST scanner, which was scanning 14 websites by domain, had 13 misspelled domain names (they were missing the ‘m’ on dot com). 13 of the 14 domains it was scanning didn’t exist, and the tool didn’t tell them that. Since 1 of the 14 was getting scanned, they assumed everything was fine.

3. These days, security tools represent significant security risks and attack surface - one out of ten vulnerabilities in CISA’s known exploited vulnerabilities list belong to a security vendor.

Visibility is important in cybersecurity - no security leader wants to be in a situation where they’re blind to an attack, a vulnerability, or a gap in their compliance program. Buyers ask the vendor to show them everything, and their staff drown in the results. I’ve observed an unwillingness to reduce this noise level.

Why? A sort of hoarding or FOMO effect exists with security leaders. Is it more defensible to enable all the alerts and say “we missed it because we don’t have enough people in the SOC”? Or is it more difficult to disable the noisiest alerts and risk missing something?

You can bet the market noticed this problem and was eager to sell a solution.

This market makes lemonade

Initially, the solution was to click the export-to-CSV button and make sense of the data in Excel. However, once vendors noticed buyers struggling, we started to see entire market segments spring up to solve the problems that other security products created.

Pause for a moment to consider a scenario:

1. You bought a vulnerability scanner for $75k. You run your first scan. The results overwhelm your team.

2. You ask for more headcount and hired more folks to address the workload. Maybe this worked for a while, maybe not.

3. Web 2.0 happens. Your company builds fancy webapps and maybe even offers SaaS to customers.

4. You buy a web scanning tool for $35k. You run your first scan. The results overwhelm your team

5. The cloud is invented and your company adopts it, but doesn’t get rid of the existing datacenter.

6. You buy a cloud scanning tool for $50k. You run your first scan. The results overwhelm your team.

7. By the way, you’re not sure if ANY of these tools have gaps. You’re not sure if they’re finding all the critical vulnerabilities, because testing security tool efficacy is hard. Also, you don’t have time, because you’re too busy chasing all the busywork these tools are generating.

8. A product category emerges and offers an enticing pitch: what if we took all those scans, combined them, and prioritized the findings, making your staff’s job easier? This product costs $125k.

Now we’re making lemonade. The buyer isn’t even sure the scanners are producing value but are buying another layer of products to fix the problems created by the first layer. I want to be clear - the lemonade makers aren’t the lemons, they’re just spotted a market opportunity: “hey, that’s a lot of lemons you’ve got there. You might as well make some lemonade, right?

Some examples of lemonade makers:

• The risk-based vulnerability management (RBVM) market, which is the example from the scenario above. These tools don’t include a vulnerability scanner, so you’ve got to purchase them in addition to vulnerability scanning tools.

• Security analytics, SOAR, UBEA, and others were add-ons to the classic SIEM, offering to do what the original SIEM vendors promised 20 years ago: correlate data, extract insights.

• But wait, why did we need a SIEM in the first place? Probably because we didn’t want to have to log into a dozen different security products to check for alerts, correlating timestamps across devices manually. We’re potentially three or more levels deep on this one, especially if you have a data lake, ETL products, an MSSP/MDR to handle SOC monitoring in the off-hours…

• Products like FireEye’s NX (’Breach Detection Systems’) positioned themselves as complementary to IDS and IPS devices as well as endpoint security products.

• You can buy a license for an expensive TPRM spreadsheet and then an AI-powered GRC product to automate filling it out.

• Interestingly, there’s some analysis out from At-Bay showing that you’re more likely to file an email-incident-related cyber insurance claim if you have an email security appliance vs just using the security built into your email platform. Dig a little deeper and you’ll find that At-Bay is also getting into the lemonade business.

• I’ll stop now, but feel free to mention more in the comments, or challenge any of the examples I’ve come up with here.

Hope you’re thirsty

The problems created by core security products are almost always very obvious process issues (e.g. too much data to process, analyze, or action). This is great for second tier products, because it’s very easy for them to demonstrate and measure value in terms of time saved versus the old product. The problem is that the buyer doesn’t know if the core products were doing a good job to begin with.

“Wait, I didn’t even want lemons in the first place, did I?”

I can’t fault these second-tier lemonade vendors, because they’re responding to real issues that customers have. It can be an uphill battle for them as well - it’s not easy for buyers to hear that they need to buy an additional product to make their existing tools work properly. Buyers will often defer the decision until the problem becomes super painful.

Why doesn’t this secondary market simply replace the core vendor and solve the core problems, then?

There might not be a universal reason for this, but I have a few thoughts:

1. rebuilding the core product is difficult and expensive (e.g. building 200,000 vulnerability checks from scratch)

2. the solution to the core problem isn’t obvious

3. the industry’s collective sunk cost is too great: the solution requires a completely different approach, requiring the vendor to reeducate the market and go against common practices, certifications, standards, and regulations

The final theory is a really tough one. It has been observed that security leaders are swayed towards the choices that are safest for their personal careers. When the choice is going against the grain versus doing the safe thing, the data tells us the latter is going to be the more common path.

Conclusion

There’s no Ozempic for cyber yet, so we’re faced with a familiar dilemma. We can keep downing lemonade, getting nowhere with our goals, or start doing the hard (diet and exercise) work necessary to determine what works and ditch what doesn’t. The latter is tough, as there are little to no incentives to challenge the status quo, even when it is clearly broken.

In consumer markets, information asymmetry tends to solve itself when value is obvious. If a particular model or brand of laptop breaks a lot, it will have lots of bad reviews online. A laptop breaking is an outcome that can’t be ignored.

Cybersecurity isn’t like this. It is more comparable to the health supplement market. Do I feel amazing today because I got an extra hour of sleep, or because I started making drinks with this green powder? Most consumers don’t have the time or patience to scientifically test every new nutrition product or trend they try out, so if they think it makes them feel better, they’ll keep buying it or doing it.

At least with nutritional supplements, there will be folks that will put them to the test, or send products to a lab to determine if they’re even safe to consume. Cybersecurity doesn’t really have an equivalent, unfortunately.

This is a critical problem in cybersecurity. When we don’t know how well our tools or controls are working, we’re at a distinct disadvantage as defenders. As Charlie Miller once put it:

Buyers should be pushing back on products that create more problems than they solve. However, without a clear path to better products, practitioners continue on, buying what their peers bought that didn’t get them fired. Ironically, a big part of the problem is a lack of risk appetite for better security products.