Aggregation and Outbound Denies, A Powerful Combination

I have previously blogged (separately) about the merits of both aggregation and looking at outbound denied traffic.  It occurs to me that it is worth a separate post to blog about the powerful combination of aggregation and outbound denies.

If one takes a rich data source (such as proxy logs), looks at the outbound denied traffic, and aggregates by certain key fields, such as:

• Source IP Address
• Destination IP Address
• Domain
• URL
• Request Method (e.g., GET, POST, etc.)
• Count (ordering by Count in descending order)

The results (say, over a 24 hour time slice of data) are generally quite interesting.  Taking a step back, we see that what we're essentially doing is slicing the data in such a way so as to extract repetitive activity that is being denied by the proxy (for whatever reason).  Generally, humans do not create traffic that fits this criteria, but rather, machines do.  Machine generated activity is generally quite interesting from an analytical perspective, though sometimes it is a mere nuisance (e.g., toolbar generated traffic).  On the frequent occasion that the traffic is malicious, this approach to slicing the data is quite helpful in finding the activity of concern.

Big Data Requires a Surgical Approach

I've previously posted about the overwhelming volume of data confronting enterprises today, as have countless others.  Although I have hinted at it through many of my blog posts and provided several tangible, hands-on examples in this blog, it occurs to me that I have never overtly stated that "big data requires a surgical approach".  What does this mean?  Essentially, with billions of transactions/records/sessions per day, the volume of data has grown beyond the capability of organizations to handle it.  The enterprise's network data must be sliced up surgically using a variety of different techniques.  Each technique takes a slightly different view/vantage point of the data and produces a reduced, filtered, and more manageable volume of data for investigation.  If this sounds familiar, it is because it is essentially the same approach as I've blogged about previously in posts discussing what I call the "Jumping Off Points" approach.  The approach is simple, straightforward, and effective.  The challenge is finding the most relevant and value-added jumping off points, and continuously working to improve them and to find the next group of relevant and value-added jumping off points.

Less Tangible Elements of World Class Security Operations/Incident Response Functions

I have had the privilege of working with a number of different incident response/security operations functions within a number of different enterprises.  What I have come to realize is that in addition to all the tangible elements that running a successful security operations/incident response function entails, there are also several key elements that are less tangible, but equally as important.  Although harder to measure, these points are nonetheless equally as important:

• Strong leadership presence in the incident response/security operations community: The best incident response/security operations functions are run by people who have walked the walk and who are active members in the relatively small and close-knit community.  Quite simply put, incident response/security operations functions run by people who understand the challenges, can think strategically about how to approach them, and have the contacts, respect, and earned authority to implement the required strategic approach are more mature than those run by people who do not fit the above description.
• Realization that information sharing and incident response are one in the same: I often see that organizations have "Timely Incident Response" and "Information Sharing" as two separate strategic objectives.  Both are important, but if one thinks about it, they are effectively one in the same.  What does this mean?  That a) strong information sharing relationships can be one of the most effective ways to detect/understand/be notified that an incident is underway requiring response and conversely that b) when an incident is underway, having solid and strong information sharing relationships can be one of the most effective ways to handle/contain the incident (e.g., having close relationships with hosting providers that can take down sites for an organization).
• Proactive intelligence: Many organizations do decently well with reactive intelligence.  For example, if it becomes known that a given URL pattern is an indicator of malicious command and control (C2) activity, most organizations can immediately leverage this in their alerting.  Naturally, this is extremely important, but it is, in its essence, reactive.  Proactive intelligence is something that most organizations do less well.  It involves tracking the attackers and threat landscape to understand the direction in which threats/attacks are moving and how to translate that into actionable intelligence that can be implemented operationally.  This is no easy task, but it is something that separates the world class organizations from the rest of the pack.
• User/insider threat: The most serious compromises generally involve theft and/or misuse of user accounts, certificates, and/or other credentials.  Because of this, tracking, profiling, modeling, and identifying anomalous/suspicious/malicious user activity is essential to a world class security operations/incident response function.  Identifying anomalous user activity is a challenge, but it is one that the best organizations do not shy away from.
• Incident response/security operations is a cerebral business: It is tempting/easy to pay an overwhelming amount of attention to the operational component of incident response/security operations without paying enough attention to the cerebral component.  It is true that incident response/security operations necessitates a strong operational component.  What the best organizations understand is that the operational component supports the strategic, well-structured, intelligently-approached (cerebral) component/foundation, and not the other way around.

Hopefully these thoughts are helpful to those looking to build, strengthen, and/or enhance their incident response/security operations function.  Feedback is, of course, always welcome.

Process

To most analysts, the word process is a scary one that conjures up images of rote, check-box type work.  Although that does sometimes occur, in the Incident Response/Security Operations world, process is extremely important.  Why is this so?  Because in a field where data is so overwhelming, expectations are so high, and resources are so very limited, having an organized, well-structured, well-defined approach to the day-to-day workflow is extremely important.  Organizations that have a well-defined incident response process (at all different levels -- from the highest, strategical level down to the lowest, operational level) generally do much better in incident response than organizations that do not.

A good incident response process can help focus resources (software, hardware, and wetware) and maximize the value they provide.  Process isn't the sexiest of endeavors, but if done properly, it is one of the most productive and value-added.

Disconnect

Lately, I've noticed that there is a bit of a disconnect in the security community between security researchers/malware researchers and security operations personnel.  Perhaps disconnect is too strong of a word -- I'm not sure.  What I've noticed is that security researchers/malware researchers are most interested in attacker techniques, exploit kits, and the actual malware/payload delivery itself, while security operations personnel are most interested in timely and reliable detection and response.  Where's the disconnect you ask?  Well, for one, techniques, exploit kits, and payload delivery sometimes (or often times depending on the environment) fail/don't result in successful infection.  So, while researchers are chasing the ever evolving/changing exploit/payload delivery landscape, security operations personnel are hungry for reliable/actionable indicators of compromise.  As has been discussed previously on this blog, the most reliable/actionable indicators of compromise are usually post-infection, as it's much easier to look for unusual behavior/activity after a machine has been infected than it is to look for an infection that is about to happen or is in progress.

So, on one hand, we have a strong, bright, and energetic community discovering new techniques, exploits, and payload delivery (all pre-infection), while on the other hand, we have a dedicated and hard working community desperately seeking reliable post-infection indicators of compromise.  There is a void between the pre-infection and post-infection data/intelligence, and unfortunately, there aren't a lot of people or organizations currently filling that void.

Interesting observation, but what can be done?  My hope is that researchers will become more interested in post-infection activity, while at the same time, security operations personnel will become better at tracking the great pre-infection research that is going on and correlating/relating that to post-infection indicators of compromise.  I am optimistic that the community will continue to improve in this regard with the proper attention and focus.

Content Development

I often speak about how streamlining the CIRC/SOC workflow and producing high reliability, high fidelity, low noise alerts is a key element to an organization's success in this arena.  The question I often get asked after this is "Where do the alerts come from?".

Well, that is a very good question.  The answer is that it can vary.  One thing that's for sure though is that the process that leads to the alerts -- content development -- is extremely important.  It's important to assess your organization's risk and exposure, and then determine what you would like to look for (conceptually) to monitor/address that risk.  Once you define what you want to look for, then a careful study of the data needs to be performed in order to guide the organization to alerting that will identify activity of concern with low noise.

It's an art, rather than a science, but allowing the data, risk/exposure, and operational requirements to guide the content development process will produce better results than any other approach I've seen.

Collecting, Vetting, Retaining, and Leveraging Intelligence

If you've done a good job building bridges and relationships for your incident response center or SOC, you will receive intelligence/indicators of compromise from time to time.  Once you receive them, what do you do with them?  If you search back in your logs a few weeks or months to see if you've got any hits, that's a good start.  But what if an attack hits tomorrow, after you've already run your search and (likely) "discarded" that intelligence?

That's where a robust intelligence analysis function and process can help.  Joining information sharing groups, purchasing intelligence from vendors, working collaboratively with peer organizations, and building bridges and trusted relationships can all net you decent intelligence.  Once you collect it, it should be vetted.  In other words, given the context (which is extremely important) of a particular piece of intelligence (e.g., is it a payload delivery site, C2 site, malicious email sender, etc.), is it reliable as an indicator of compromise?  Does it produce a large number of false positives, or is the noise relatively tame (making it more useful/reliable as an indicator of compromise)?

Once an indicator has been vetted and deemed reliable, it should be retained.  I've seen a number of organizations use some sort of an intelligence repository to retain the vetted, reliable, high fidelity, actionable intelligence they have.  Once it's retained, of course, it should be fully leveraged.  This includes writing alerts to check the intelligence repository regularly and run the data against recent logging.

I'm sure that this sounds conceptually simple, but it's amazing how many organizations don't properly retain and leverage the intelligence they receive.  Take a look within your own organization -- if you can better retain and leverage intelligence, it will serve you well in the long run!

Xenophobia

Over the years, I've worked with, developed trusted relationships with, and shared information with people holding a wide variety of passports.  A person's professional work, professional reputation, and trusted relationships generally speak to a person much better than the color of their passport.  Unfortunately, there are some people and organizations who think along the very analog lines of countries of origin/passport in hand.  This is foolish in my opinion -- judge the person not by the color of their passport, but by the content of their work.  To not do so could be to needlessly exclude fresh information/a different perspective so often missing in very insular places.

What's the concern?

I find it interesting that some people have a knee jerk reaction/aversion to information sharing, or proceed to turn any mention of it into an over-complicated mess of a conversation.  I always seem to hear the same types of statements:

"We have privacy concerns"
"There is a lot of regulation that prohibits/impedes information sharing"
"That information is classified/sensitive/protected"
"We are not permitted to share information"

In my experience, when there is doubt or fear of the unknown, it's always easier for people to say no and then provide reasons that appear quite official and legitimate to support that position.  After all, no one ever got fired for not sharing information, right?  These individuals that choose this path, however, put their organizations at serious risk by needlessly limiting the organization's access to timely, valuable, high fidelity information necessary for incident response/security operations.

What's interesting to me is where this doubt/fear comes from.  As far as I can tell, it comes from a profound lack of knowledge regarding what is valuable from an information sharing perspective.  If long lists of sensitive internal assets were valuable from an information sharing perspective, I could totally understand the hestitation.  But, as it turns out, to our fortune, the most valuable information for incident response/security operations is publicly available Indicators of Compromise (IOCs), such as malicious domain names, malicious callback URL patterns, malicious email attachment names/MD5s, etc.!  It takes a skilled analyst and trusted circles of peer review to vet/filter the vast maze of information until it is boiled down to its most valuable essence.  But what is eventually shared is entirely focused on "footprints" that attackers leave in the sand after an intrusion, and contains no sensitive, private, or personal information about an organization.

So, I'm left asking, what exactly is the concern with sharing valuable, actionable, high fidelity IOCs within trusted peer information sharing groups?  Seems to me the concern is a fear of the unknown/lack of understanding what is valuable from an information sharing perspective.  I think it's about time that changed.

IDS

I find it fascinating that although IDS has the potential to be used as a sharpened scalpel to pick out abnormal network activity from various places within the packet, it is mostly used as a packet by packet log collector.  What do I mean by this?  IDS is a technology that can be equipped with a relatively small number of highly potent and actionable signatures designed to look for activity that organizations need to take notice of.  Unfortunately though, most organizations go the complete opposite direction, deploying IDS with thousands of weak/mediocre signatures, perhaps out of fear of "missing something".  The result?  Anything that might have been worth looking at gets buried in 10,000 false positives per day (or more!).  Worse yet, sometimes IDS becomes every analyst's favorite data source to ignore.  It's a shame really -- so much potential in modern IDS devices, yet so underutilized.

BS Filter

It sounds a bit crude, but a good BS filter can be an analyst's best friend.  Analysts are confronted by an overwhelming amount of information on a daily basis.  Whether it be from blogs, mailing lists, management, a vendor, an intelligence feed, a tool, or one or more log sources, it can be overwhelming.  Diving into the wrong lead can tie up precious analyst resources for hours or even days, often taking away from other events or leads that need to be investigated.  So what is an analyst to do?  Follow only the leads most likely to yield fruit!  Easier said than done, of course.  Knowing how to separate out the good leads from those that are a fool's errand is an acquired skill that takes years of false starts to develop.  It can be a skill that is extremely valuable to an organization though and should be respected.

Big Data

There is a lot of buzz lately about big data.  Almost every vendor and most pundits seem to be talking about big data -- and telling us we need it.  While I am seeing a lot of hype and build-up, I'm not seeing a whole lot of useful advice or helpful tips about *how* to actually leverage big data.  Where I work, we've been leveraging big data for over a year now (we just never called it that).  I see this as an opportunity.  We've developed some effective techniques for slicing through big data and producing a reasonable volume of highly actionable alerting from over 4 billion network events a day.  It's time to take this show on the road and share the information.  I've put in some papers for some upcoming conferences -- wish me luck!

To The Endpoint

As time has progressed, monitoring has moved further and further inside the enterprise.  Once upon a time, an enterprise could monitor only its perimeter and still be in decent shape.  Then, attackers started moving up the OSI stack (as has been discussed previously in this blog and elsewhere).  Now, with attackers using legitimate domains/sites, encrypted command and control, and a whole host of other techniques to hide in the noise/legitimate traffic, it is getting increasingly difficult for the defender to keep pace.  It appears to me that the only long term monitoring solution is to go all the way to the endpoint somehow (in tandem with instrumenting the network for monitoring of course).  I'm not sure I see how else we can obtain the visibility on the interior segments of the network that we so desperately need.

A sad reality I fear.

Practical Passive DNS Example

I (along with many others) have blogged in the past about the tremendous value of passive DNS data.  Recently, I have encountered a practical example of an instance where passive DNS data may provide one of the only ways to try and stay on top of a threat.

Recently, some members of the community have encountered/been working with a piece of malicious code that has the following properties:

1) A malicious JAR file is delivered with a content type of text/html.  This JAR file assembles two executables on the victim host and infects it.
2) The infected vicitim host initiates a callback to a randomly generated four character sub-domain (e.g., ..com).  This traffic is encrypted (via SSL) and goes to the same netblock (a /24), regardless of the domain name (which changes regularly).

The purpose of this malware is not yet known, and it is not widely written about or particularly well understood.

Typically, as has been discussed on this blog and elsewhere in the past, it's much easier to look for artifacts of infection, rather than pre-infection activity.  Typically that involves creating monitoring/alerting for one of the following:

1) Looking for command and control URL patterns (which are much more reliable and have lower false positive rates than just domain names or IP addresses).  In this case, the callback traffic is encrypted, and thus, there is no URL pattern that can be monitored for (unless one intercepts the SSL traffic, but that is a separate discussion).
2) Looking for command and control domain names.  Here again, since the domain names change, this is not going to be very effective as a long term monitoring solution.
3) Looking for the malicious code payload delivery.  In this case, since it is a JAR being delivered with a content type of text/html, the noise is overwhelming.  Creating alerting based on this type of logic would inundate incident response personnel with alerts, the overwhelming majority of which would be false positives.  As I've learned, delivering a JAR via content type text/html is actually fairly common, even for legitimate applications.

So what does one do?  Well, since the callbacks all go to the same netblock, one can monitor passive DNS data for any new domains that are requested and resolve to that netblock matching the pattern described above.  That way, if the callback domains change (which they have been), and a host is infected, the incident response team can pick up on the new callback domain(s) and respond accordingly.  True, the incident response team is still one step behind the malware, but that's a lot better than being in the dark.

Pretty neat real world use of passive DNS data in my opinion.