Measuring Security Intelligence Value

There was recently a discussion around measuring the value of a security intelligence program in the Twitterverse. Several well-known security experts took part in the discussion, and it was quite interesting to see everyone's thoughts on the subject. Further, this is a discussion that I am hearing more and more in the security operations space, and rightfully so. Security intelligence is a complex topic that requires more elaboration than Twitter's character limit allows for, and in fact, it requires more elaboration than I can realistically put into a blog posting. That being said, I will give it a shot. I have a large amount of operational experience in this area, so if you'll indulge me, I'll provide some thoughts on the topic here in this blog posting.

Generating security intelligence data (i.e., functioning as a source of intelligence) is an interesting topic, but not one that I will discuss in this post. Threat assessment is another fascinating topic, but also not one that I will discuss in this post. Additionally, intelligence sharing is also a hot topic, but again, not one that I will discuss in this post. Instead, I will focus on security intelligence as it relates to defending a large network in the context of a broader security operations program (i.e., functioning as a consumer of intelligence). In this context, at a high level, security intelligence involves consuming a piece of information (e.g., domain name, URL pattern, file name, MD5 hash, etc.), along with some context (e.g., exploit site, callback domain, drop site, malicious attachment MD5, etc.), and subsequently leveraging that information, in the right context, against host data and/or network traffic data.

Over the course of my career, I have seen security intelligence programs that work well, along with those that do not work as well. Before I get into a discussion of metrics around security intelligence programs, here are a few observations relating to challenges that organizations often encounter when implementing a security intelligence program:

• Information lacks context: The best information in the world is useless unless we know in which context to use it -- context is key. Remember, only information with the proper context can qualify as intelligence.
• Confusion of quantity with quality: If we have 5,000 "malicious" domain names, but 4,995 of them generate almost entirely false positives, that provides far less value than 10 reliable, high fidelity malicious domain names. Not only does the first example detect less true positives, but the volume of false positives overwhelm the work queue to the detriment of security operations.
• Lack of indicator reliability and fidelity: It is important to vet indicators and sources before they are introduced into the alerting queue and workflow. Failure to do this properly can result in an overwhelming volume of false positives that dominate the work queue and squander valuable analyst cycles.
• Lack of appropriate data: The best intelligence in the world is useless if we can't search for it over large quantities of host and network data over long periods of time rapidly.
• Improper tracking of intelligence and sources: This leads well into the metrics discussion -- it is extremely important to warehouse and track intelligence and its sources with enough granularity to enable metrics and measurement.
• Lack of integration with the workflow: If leveraging security intelligence is a pain, analysts will do it less, or they won't do it at all. This is a workflow/efficiency issue that can come at a great cost to an organization's overall security posture.

Measuring the value of a security intelligence program can be a difficult task. I'm sure there are many ways to approach the challenge. As mentioned above, proper warehousing and tracking of intelligence and its sources is a necessary precursor. Assuming that is in place, here are a few measurement approaches that I have found helpful over the course of my career:

• Overall number of incidents/percentage of incidents identified via security intelligence vs. identified via other means.
• Percentage of false positives per intelligence source.
• Percentage of overall false positives resulting from security intelligence.
• Percentage of incidents identified through security intelligence per intelligence source (also known as percentage of true positives per intelligence source).
• Percentage of overall true positives resulting from security intelligence.
• Mean time to detection (should decrease as security intelligence program matures).
• Number of long-time (i.e., long undetected) intrusions uncovered via security intelligence.

This is not an exhaustive list, but it does provide a few metrics that I have found helpful for measuring and showing the value of a security intelligence program. Although it's implied here, it's perhaps worth stating explicitly that proper instrumentation of the network, for both collection and analysis is critical here. If an organization does not have total visibility into the network traffic and host data, along with the ability to incisively query that data rapidly, that organization will not be particularly successful in building a security intelligence program nor in measuring it.

A security intelligence program is a great thing, and every large organization should have one. It pays to consider how to make your security intelligence program the best one it can be.