Aggregation and Outbound Denies, A Powerful Combination

I have previously blogged (separately) about the merits of both aggregation and looking at outbound denied traffic.  It occurs to me that it is worth a separate post to blog about the powerful combination of aggregation and outbound denies.

If one takes a rich data source (such as proxy logs), looks at the outbound denied traffic, and aggregates by certain key fields, such as:

• Source IP Address
• Destination IP Address
• Domain
• URL
• Request Method (e.g., GET, POST, etc.)
• Count (ordering by Count in descending order)

The results (say, over a 24 hour time slice of data) are generally quite interesting.  Taking a step back, we see that what we're essentially doing is slicing the data in such a way so as to extract repetitive activity that is being denied by the proxy (for whatever reason).  Generally, humans do not create traffic that fits this criteria, but rather, machines do.  Machine generated activity is generally quite interesting from an analytical perspective, though sometimes it is a mere nuisance (e.g., toolbar generated traffic).  On the frequent occasion that the traffic is malicious, this approach to slicing the data is quite helpful in finding the activity of concern.

Big Data Requires a Surgical Approach

I've previously posted about the overwhelming volume of data confronting enterprises today, as have countless others.  Although I have hinted at it through many of my blog posts and provided several tangible, hands-on examples in this blog, it occurs to me that I have never overtly stated that "big data requires a surgical approach".  What does this mean?  Essentially, with billions of transactions/records/sessions per day, the volume of data has grown beyond the capability of organizations to handle it.  The enterprise's network data must be sliced up surgically using a variety of different techniques.  Each technique takes a slightly different view/vantage point of the data and produces a reduced, filtered, and more manageable volume of data for investigation.  If this sounds familiar, it is because it is essentially the same approach as I've blogged about previously in posts discussing what I call the "Jumping Off Points" approach.  The approach is simple, straightforward, and effective.  The challenge is finding the most relevant and value-added jumping off points, and continuously working to improve them and to find the next group of relevant and value-added jumping off points.

Less Tangible Elements of World Class Security Operations/Incident Response Functions

I have had the privilege of working with a number of different incident response/security operations functions within a number of different enterprises.  What I have come to realize is that in addition to all the tangible elements that running a successful security operations/incident response function entails, there are also several key elements that are less tangible, but equally as important.  Although harder to measure, these points are nonetheless equally as important:

• Strong leadership presence in the incident response/security operations community: The best incident response/security operations functions are run by people who have walked the walk and who are active members in the relatively small and close-knit community.  Quite simply put, incident response/security operations functions run by people who understand the challenges, can think strategically about how to approach them, and have the contacts, respect, and earned authority to implement the required strategic approach are more mature than those run by people who do not fit the above description.
• Realization that information sharing and incident response are one in the same: I often see that organizations have "Timely Incident Response" and "Information Sharing" as two separate strategic objectives.  Both are important, but if one thinks about it, they are effectively one in the same.  What does this mean?  That a) strong information sharing relationships can be one of the most effective ways to detect/understand/be notified that an incident is underway requiring response and conversely that b) when an incident is underway, having solid and strong information sharing relationships can be one of the most effective ways to handle/contain the incident (e.g., having close relationships with hosting providers that can take down sites for an organization).
• Proactive intelligence: Many organizations do decently well with reactive intelligence.  For example, if it becomes known that a given URL pattern is an indicator of malicious command and control (C2) activity, most organizations can immediately leverage this in their alerting.  Naturally, this is extremely important, but it is, in its essence, reactive.  Proactive intelligence is something that most organizations do less well.  It involves tracking the attackers and threat landscape to understand the direction in which threats/attacks are moving and how to translate that into actionable intelligence that can be implemented operationally.  This is no easy task, but it is something that separates the world class organizations from the rest of the pack.
• User/insider threat: The most serious compromises generally involve theft and/or misuse of user accounts, certificates, and/or other credentials.  Because of this, tracking, profiling, modeling, and identifying anomalous/suspicious/malicious user activity is essential to a world class security operations/incident response function.  Identifying anomalous user activity is a challenge, but it is one that the best organizations do not shy away from.
• Incident response/security operations is a cerebral business: It is tempting/easy to pay an overwhelming amount of attention to the operational component of incident response/security operations without paying enough attention to the cerebral component.  It is true that incident response/security operations necessitates a strong operational component.  What the best organizations understand is that the operational component supports the strategic, well-structured, intelligently-approached (cerebral) component/foundation, and not the other way around.

Hopefully these thoughts are helpful to those looking to build, strengthen, and/or enhance their incident response/security operations function.  Feedback is, of course, always welcome.

Process

To most analysts, the word process is a scary one that conjures up images of rote, check-box type work.  Although that does sometimes occur, in the Incident Response/Security Operations world, process is extremely important.  Why is this so?  Because in a field where data is so overwhelming, expectations are so high, and resources are so very limited, having an organized, well-structured, well-defined approach to the day-to-day workflow is extremely important.  Organizations that have a well-defined incident response process (at all different levels -- from the highest, strategical level down to the lowest, operational level) generally do much better in incident response than organizations that do not.

A good incident response process can help focus resources (software, hardware, and wetware) and maximize the value they provide.  Process isn't the sexiest of endeavors, but if done properly, it is one of the most productive and value-added.

Disconnect

Lately, I've noticed that there is a bit of a disconnect in the security community between security researchers/malware researchers and security operations personnel.  Perhaps disconnect is too strong of a word -- I'm not sure.  What I've noticed is that security researchers/malware researchers are most interested in attacker techniques, exploit kits, and the actual malware/payload delivery itself, while security operations personnel are most interested in timely and reliable detection and response.  Where's the disconnect you ask?  Well, for one, techniques, exploit kits, and payload delivery sometimes (or often times depending on the environment) fail/don't result in successful infection.  So, while researchers are chasing the ever evolving/changing exploit/payload delivery landscape, security operations personnel are hungry for reliable/actionable indicators of compromise.  As has been discussed previously on this blog, the most reliable/actionable indicators of compromise are usually post-infection, as it's much easier to look for unusual behavior/activity after a machine has been infected than it is to look for an infection that is about to happen or is in progress.

So, on one hand, we have a strong, bright, and energetic community discovering new techniques, exploits, and payload delivery (all pre-infection), while on the other hand, we have a dedicated and hard working community desperately seeking reliable post-infection indicators of compromise.  There is a void between the pre-infection and post-infection data/intelligence, and unfortunately, there aren't a lot of people or organizations currently filling that void.

Interesting observation, but what can be done?  My hope is that researchers will become more interested in post-infection activity, while at the same time, security operations personnel will become better at tracking the great pre-infection research that is going on and correlating/relating that to post-infection indicators of compromise.  I am optimistic that the community will continue to improve in this regard with the proper attention and focus.

Content Development

I often speak about how streamlining the CIRC/SOC workflow and producing high reliability, high fidelity, low noise alerts is a key element to an organization's success in this arena.  The question I often get asked after this is "Where do the alerts come from?".

Well, that is a very good question.  The answer is that it can vary.  One thing that's for sure though is that the process that leads to the alerts -- content development -- is extremely important.  It's important to assess your organization's risk and exposure, and then determine what you would like to look for (conceptually) to monitor/address that risk.  Once you define what you want to look for, then a careful study of the data needs to be performed in order to guide the organization to alerting that will identify activity of concern with low noise.

It's an art, rather than a science, but allowing the data, risk/exposure, and operational requirements to guide the content development process will produce better results than any other approach I've seen.

Collecting, Vetting, Retaining, and Leveraging Intelligence

If you've done a good job building bridges and relationships for your incident response center or SOC, you will receive intelligence/indicators of compromise from time to time.  Once you receive them, what do you do with them?  If you search back in your logs a few weeks or months to see if you've got any hits, that's a good start.  But what if an attack hits tomorrow, after you've already run your search and (likely) "discarded" that intelligence?

That's where a robust intelligence analysis function and process can help.  Joining information sharing groups, purchasing intelligence from vendors, working collaboratively with peer organizations, and building bridges and trusted relationships can all net you decent intelligence.  Once you collect it, it should be vetted.  In other words, given the context (which is extremely important) of a particular piece of intelligence (e.g., is it a payload delivery site, C2 site, malicious email sender, etc.), is it reliable as an indicator of compromise?  Does it produce a large number of false positives, or is the noise relatively tame (making it more useful/reliable as an indicator of compromise)?

Once an indicator has been vetted and deemed reliable, it should be retained.  I've seen a number of organizations use some sort of an intelligence repository to retain the vetted, reliable, high fidelity, actionable intelligence they have.  Once it's retained, of course, it should be fully leveraged.  This includes writing alerts to check the intelligence repository regularly and run the data against recent logging.

I'm sure that this sounds conceptually simple, but it's amazing how many organizations don't properly retain and leverage the intelligence they receive.  Take a look within your own organization -- if you can better retain and leverage intelligence, it will serve you well in the long run!