Less Tangible Elements of World Class Security Operations/Incident Response Functions

I have had the privilege of working with a number of different incident response/security operations functions within a number of different enterprises.  What I have come to realize is that in addition to all the tangible elements that running a successful security operations/incident response function entails, there are also several key elements that are less tangible, but equally as important.  Although harder to measure, these points are nonetheless equally as important:

• Strong leadership presence in the incident response/security operations community: The best incident response/security operations functions are run by people who have walked the walk and who are active members in the relatively small and close-knit community.  Quite simply put, incident response/security operations functions run by people who understand the challenges, can think strategically about how to approach them, and have the contacts, respect, and earned authority to implement the required strategic approach are more mature than those run by people who do not fit the above description.
• Realization that information sharing and incident response are one in the same: I often see that organizations have "Timely Incident Response" and "Information Sharing" as two separate strategic objectives.  Both are important, but if one thinks about it, they are effectively one in the same.  What does this mean?  That a) strong information sharing relationships can be one of the most effective ways to detect/understand/be notified that an incident is underway requiring response and conversely that b) when an incident is underway, having solid and strong information sharing relationships can be one of the most effective ways to handle/contain the incident (e.g., having close relationships with hosting providers that can take down sites for an organization).
• Proactive intelligence: Many organizations do decently well with reactive intelligence.  For example, if it becomes known that a given URL pattern is an indicator of malicious command and control (C2) activity, most organizations can immediately leverage this in their alerting.  Naturally, this is extremely important, but it is, in its essence, reactive.  Proactive intelligence is something that most organizations do less well.  It involves tracking the attackers and threat landscape to understand the direction in which threats/attacks are moving and how to translate that into actionable intelligence that can be implemented operationally.  This is no easy task, but it is something that separates the world class organizations from the rest of the pack.
• User/insider threat: The most serious compromises generally involve theft and/or misuse of user accounts, certificates, and/or other credentials.  Because of this, tracking, profiling, modeling, and identifying anomalous/suspicious/malicious user activity is essential to a world class security operations/incident response function.  Identifying anomalous user activity is a challenge, but it is one that the best organizations do not shy away from.
• Incident response/security operations is a cerebral business: It is tempting/easy to pay an overwhelming amount of attention to the operational component of incident response/security operations without paying enough attention to the cerebral component.  It is true that incident response/security operations necessitates a strong operational component.  What the best organizations understand is that the operational component supports the strategic, well-structured, intelligently-approached (cerebral) component/foundation, and not the other way around.

Hopefully these thoughts are helpful to those looking to build, strengthen, and/or enhance their incident response/security operations function.  Feedback is, of course, always welcome.

Process

To most analysts, the word process is a scary one that conjures up images of rote, check-box type work.  Although that does sometimes occur, in the Incident Response/Security Operations world, process is extremely important.  Why is this so?  Because in a field where data is so overwhelming, expectations are so high, and resources are so very limited, having an organized, well-structured, well-defined approach to the day-to-day workflow is extremely important.  Organizations that have a well-defined incident response process (at all different levels -- from the highest, strategical level down to the lowest, operational level) generally do much better in incident response than organizations that do not.

A good incident response process can help focus resources (software, hardware, and wetware) and maximize the value they provide.  Process isn't the sexiest of endeavors, but if done properly, it is one of the most productive and value-added.

Disconnect

Lately, I've noticed that there is a bit of a disconnect in the security community between security researchers/malware researchers and security operations personnel.  Perhaps disconnect is too strong of a word -- I'm not sure.  What I've noticed is that security researchers/malware researchers are most interested in attacker techniques, exploit kits, and the actual malware/payload delivery itself, while security operations personnel are most interested in timely and reliable detection and response.  Where's the disconnect you ask?  Well, for one, techniques, exploit kits, and payload delivery sometimes (or often times depending on the environment) fail/don't result in successful infection.  So, while researchers are chasing the ever evolving/changing exploit/payload delivery landscape, security operations personnel are hungry for reliable/actionable indicators of compromise.  As has been discussed previously on this blog, the most reliable/actionable indicators of compromise are usually post-infection, as it's much easier to look for unusual behavior/activity after a machine has been infected than it is to look for an infection that is about to happen or is in progress.

So, on one hand, we have a strong, bright, and energetic community discovering new techniques, exploits, and payload delivery (all pre-infection), while on the other hand, we have a dedicated and hard working community desperately seeking reliable post-infection indicators of compromise.  There is a void between the pre-infection and post-infection data/intelligence, and unfortunately, there aren't a lot of people or organizations currently filling that void.

Interesting observation, but what can be done?  My hope is that researchers will become more interested in post-infection activity, while at the same time, security operations personnel will become better at tracking the great pre-infection research that is going on and correlating/relating that to post-infection indicators of compromise.  I am optimistic that the community will continue to improve in this regard with the proper attention and focus.

Content Development

I often speak about how streamlining the CIRC/SOC workflow and producing high reliability, high fidelity, low noise alerts is a key element to an organization's success in this arena.  The question I often get asked after this is "Where do the alerts come from?".

Well, that is a very good question.  The answer is that it can vary.  One thing that's for sure though is that the process that leads to the alerts -- content development -- is extremely important.  It's important to assess your organization's risk and exposure, and then determine what you would like to look for (conceptually) to monitor/address that risk.  Once you define what you want to look for, then a careful study of the data needs to be performed in order to guide the organization to alerting that will identify activity of concern with low noise.

It's an art, rather than a science, but allowing the data, risk/exposure, and operational requirements to guide the content development process will produce better results than any other approach I've seen.

Collecting, Vetting, Retaining, and Leveraging Intelligence

If you've done a good job building bridges and relationships for your incident response center or SOC, you will receive intelligence/indicators of compromise from time to time.  Once you receive them, what do you do with them?  If you search back in your logs a few weeks or months to see if you've got any hits, that's a good start.  But what if an attack hits tomorrow, after you've already run your search and (likely) "discarded" that intelligence?

That's where a robust intelligence analysis function and process can help.  Joining information sharing groups, purchasing intelligence from vendors, working collaboratively with peer organizations, and building bridges and trusted relationships can all net you decent intelligence.  Once you collect it, it should be vetted.  In other words, given the context (which is extremely important) of a particular piece of intelligence (e.g., is it a payload delivery site, C2 site, malicious email sender, etc.), is it reliable as an indicator of compromise?  Does it produce a large number of false positives, or is the noise relatively tame (making it more useful/reliable as an indicator of compromise)?

Once an indicator has been vetted and deemed reliable, it should be retained.  I've seen a number of organizations use some sort of an intelligence repository to retain the vetted, reliable, high fidelity, actionable intelligence they have.  Once it's retained, of course, it should be fully leveraged.  This includes writing alerts to check the intelligence repository regularly and run the data against recent logging.

I'm sure that this sounds conceptually simple, but it's amazing how many organizations don't properly retain and leverage the intelligence they receive.  Take a look within your own organization -- if you can better retain and leverage intelligence, it will serve you well in the long run!

Xenophobia

Over the years, I've worked with, developed trusted relationships with, and shared information with people holding a wide variety of passports.  A person's professional work, professional reputation, and trusted relationships generally speak to a person much better than the color of their passport.  Unfortunately, there are some people and organizations who think along the very analog lines of countries of origin/passport in hand.  This is foolish in my opinion -- judge the person not by the color of their passport, but by the content of their work.  To not do so could be to needlessly exclude fresh information/a different perspective so often missing in very insular places.

What's the concern?

I find it interesting that some people have a knee jerk reaction/aversion to information sharing, or proceed to turn any mention of it into an over-complicated mess of a conversation.  I always seem to hear the same types of statements:

"We have privacy concerns"
"There is a lot of regulation that prohibits/impedes information sharing"
"That information is classified/sensitive/protected"
"We are not permitted to share information"

In my experience, when there is doubt or fear of the unknown, it's always easier for people to say no and then provide reasons that appear quite official and legitimate to support that position.  After all, no one ever got fired for not sharing information, right?  These individuals that choose this path, however, put their organizations at serious risk by needlessly limiting the organization's access to timely, valuable, high fidelity information necessary for incident response/security operations.

What's interesting to me is where this doubt/fear comes from.  As far as I can tell, it comes from a profound lack of knowledge regarding what is valuable from an information sharing perspective.  If long lists of sensitive internal assets were valuable from an information sharing perspective, I could totally understand the hestitation.  But, as it turns out, to our fortune, the most valuable information for incident response/security operations is publicly available Indicators of Compromise (IOCs), such as malicious domain names, malicious callback URL patterns, malicious email attachment names/MD5s, etc.!  It takes a skilled analyst and trusted circles of peer review to vet/filter the vast maze of information until it is boiled down to its most valuable essence.  But what is eventually shared is entirely focused on "footprints" that attackers leave in the sand after an intrusion, and contains no sensitive, private, or personal information about an organization.

So, I'm left asking, what exactly is the concern with sharing valuable, actionable, high fidelity IOCs within trusted peer information sharing groups?  Seems to me the concern is a fear of the unknown/lack of understanding what is valuable from an information sharing perspective.  I think it's about time that changed.