Don't Forget to Dig

Detection is the first stage of the incident handling life cycle, and it is an important one. Detection allows us to identify and respond to issues, compromises, and breaches on our networks before they become big headlines. Given this, I’ve always been surprised at how much of the dialogue around detection is dominated by signature-based approaches to detection. When news of a new attack hits the airwaves, everyone rushes to grab the latest signatures to detect that attack. Of course, this is important and should be continued, but it is not sufficient. Signatures only find what we know about -- the known knowns, and they are reactive at best. What about a variant of the latest attack that might leave different footprints on the network than the signatures are designed to detect? What about another equally worrisome but yet unidentified attack that might be occurring as we speak? These are the unknown unknowns, and it is usually because of the unknown unknowns that organizations wind up in the press.

So how can we find the unknown unknowns? The answer is to dig. Digging involves performing analysis of the network traffic data. No one method or technique will meet the digging needs of an organization or find all intrusions. In my experience, it is most productive to try multiple different approaches and record the results of each one (e.g., in some sort of a knowledge base). Although there are many different approaches that can be tried, I have included some interesting examples here to help illustrate my point:

• Mining for domains that resolved to “parked” IP addresses for a period of time, but then began resolving to “live” IP addresses
• Aggregating over different combinations of fields (e.g., source IP, source port, destination IP, destination port, protocol, application protocol, domain, URL, byte size, etc.) and looking at outbound traffic to identify any patterns or “clustering/clumping” of activity
• Hunting for malformed packets (e.g., byte size below minimum required by RFC)
• Searching for a large number of DNS TXT records from the same system in a relatively short amount of time
• Watching for domain names with excessively short TTLs or domain names that continually change the IP addresses they resolve to

This is by no means an exhaustive list of approaches that can be tried, but I believe it does serve to illustrative the concept of digging. Results will vary with different approaches to digging, and it is important to track and document what was tried, and what resulted. This allows the team to continually build on and improve its digging. When an approach to digging is considered successful and mature, it can be automated, and the output/results from it can be sent to the alert queue. This allows successful digging techniques to integrate with the operational workflow just like successful signature-based detection techniques.

It will always be important to watch for the known knowns of course, but it is the unknown unknowns that really get us in trouble. I’m often surprised at how little digging is going on in many Security Operations Centers (SOCs). My hope is that this will soon change, and that more and more organizations will begin to dig.