The best defense against false negatives

We hear a lot of questions about detection coverage. Organizations want to know what we can detect, they want to know if we can help them sort out what they can’t detect, and they want to know how we identify and mitigate the risks associated with missed detections or false negatives. 

The short answer is defense-in-depth. The slightly longer answer is that we perform investigations across a wide array of detective, preventive, and other products that operate in disparate technology domains but ultimately constitute most of the places that adversaries target. It’s a layered and intentionally redundant approach, and in this blog we’re going to explain the underlying security processes and principles we deploy to limit false negatives and benefit from continuous coverage improvement.

It’s all about coverage

Conversations about what you can or can’t detect lead invariably back to the same place: MITRE ATT&CK®. ATT&CK is the best tool we have for enumerating all the known techniques available to an adversary, but it’s not without problems. It can give you a useful taxonomy for describing coverage, but it doesn’t tell you whether you’re actually prepared to detect and respond to the threats that matter to your organization. Not all techniques are created equal.

Truly understanding your detection and response capabilities comes from a combination of incident response readiness, threat modeling, and relevant threat intelligence: Understand the adversaries and scenarios you actually need to defend against, continuously testing whether your telemetry, detections, investigation paths, and response processes would hold up, and using the gaps from those exercises to drive investments in new security controls.

Asking about what you can or can’t detect are good questions, but the better version of those questions is this: “What am I not prepared to detect and respond to, given the threats that actually matter to me?”

So how do you answer that question? 

Here are a few of the tools we deploy to close security gaps and limit misses or false negatives. 

Detection engineering

When we develop new detections, or enable existing detections for a newly added integration, we also check historical telemetry for matches on our new detection capability. This helps us understand whether similar activity occurred before the detection was available. Sometimes this allows us to retroactively identify suspicious or malicious behavior that we can then investigate further and remediate. Other times we see our new detections or integrations showing activity we already detected by other means. This kind of redundancy makes our overall detective controls more resilient. Speaking of redundancy…

Overlapping detection coverage 

We try to cover important tactics, techniques and procedures (TTPs) from more than one source or detection path. For example, we often detect suspicious activity using Daylight detections on AWS CloudTrail before GuardDuty generates a similar alert. That “duplication” can help us detect activity faster, but it also provides resilience against false negatives in any single source or detection.

Multiple detections for the same activity

Sometimes the same stage of an attack triggers several different alerts. For example, during a red team exercise against a Salesforce instance we monitor, multiple alerts fired because the activity involved several suspicious behaviors occurring in sequence, which is very much by design in a red team exercise. We were certainly detecting different attack stages like initial access and lateral movement techniques, but we were also detecting these techniques, these same underlying activities in multiple distinct ways. As a general rule, when you detect malicious or suspicious activity, it’s best to have multiple alerts or detections firing on the bad behavior.

Incident-driven detection improvements

After true positives, we ask, “What signal existed before we detected this?” We investigate where the threat came from and what it did before we detected it. We use what we learn to develop new detection capabilities, earlier in the threat lifecycle, to improve coverage so similar activity can be detected earlier and from multiple angles in the future.

Detection validation

We simulate attacks against our tenants to identify gaps in detection coverage. We have development environments across Azure, GCP, and AWS, as well as access to security tools such as CrowdStrike, Upwind, Wiz, Okta and more, against which we run realistic threat emulations to continually validate whether our detections behave as expected.

Threat hunting

Not everything lends itself to detection. Also detection engineering takes time. Threat hunting allows us to look for threats in the cracks between our detection coverage. It also allows us to quickly search for threats based on threat intelligence or other information without waiting to develop and validate new detection analytics. The added bonus of threat hunting is that we can analyze successful hunts (i.e., those that find threatening or risky behavior) and use our hunting queries to develop new detection analytics. 

Operationalizing threat intelligence

Threat intel by itself doesn’t necessarily reduce false negatives. The value comes from turning it into action. When new threat advisories, campaigns, or TTPs emerge, we use them to create or update detections, hunt historical telemetry, and validate whether we already have sufficient coverage. Since we provide security services to a wide variety of different organizations, threat reports and advisories are very often relevant to us because they relate to technology resident in one or more of our customers. However, if you’re running an internal security team, then you have to verify that the contents of threat intelligence are relevant to your organization, which brings us to our last point.

Threat modeling

The sheer number of threat reports, advisories, and vulnerability disclosures published on a daily or weekly basis is completely overwhelming. That’s the bad news. The good news is that most of them are irrelevant to most organizations. So it’s important that organizations spend time defining their threat model. Understand the technology they use, their hygiene, the kind of data they have, and the ways they use and store that data (to name a few things). From there, security teams can self-select for threat reports or advisories that are relevant to their tech stack or that are targeting organizations like theirs. 

Conclusion

Organizations should think about false-negative reduction not as a single capability but as the combination of different security disciplines coming together to improve resilience and accomplish defense-in-depth.

Organizations who want to better understand their security coverage and work toward improving their resilience can start by taking stock of their IT inventory and hygiene. This is, unfortunately, much easier said than done, but it’s the first and most important step toward understanding their threat model. 

In parallel, security teams should attempt to enumerate their detection coverage, which is also easier said than done, and might include documenting rules in their SIEM, asking vendors about product coverage, and analyzing the extent to which existing security controls are reducing their attack surface. 

Lastly, if they haven’t already, security teams should start developing a threat intelligence function. It doesn’t have to be fancy or expensive, but focus on TTPs in intel reports or advisories that are relevant to the technologies deployed at their organization and detail prevalent threats that are targeting organizations indiscriminately. They can then cross-reference those TTPs with their coverage analysis to identify gaps and prioritize the implementation of security controls to close those gaps accordingly.