To The Endpoint

As time has progressed, monitoring has moved further and further inside the enterprise.  Once upon a time, an enterprise could monitor only its perimeter and still be in decent shape.  Then, attackers started moving up the OSI stack (as has been discussed previously in this blog and elsewhere).  Now, with attackers using legitimate domains/sites, encrypted command and control, and a whole host of other techniques to hide in the noise/legitimate traffic, it is getting increasingly difficult for the defender to keep pace.  It appears to me that the only long term monitoring solution is to go all the way to the endpoint somehow (in tandem with instrumenting the network for monitoring of course).  I'm not sure I see how else we can obtain the visibility on the interior segments of the network that we so desperately need.

A sad reality I fear.

Practical Passive DNS Example

I (along with many others) have blogged in the past about the tremendous value of passive DNS data.  Recently, I have encountered a practical example of an instance where passive DNS data may provide one of the only ways to try and stay on top of a threat.

Recently, some members of the community have encountered/been working with a piece of malicious code that has the following properties:

1) A malicious JAR file is delivered with a content type of text/html.  This JAR file assembles two executables on the victim host and infects it.
2) The infected vicitim host initiates a callback to a randomly generated four character sub-domain (e.g., ..com).  This traffic is encrypted (via SSL) and goes to the same netblock (a /24), regardless of the domain name (which changes regularly).

The purpose of this malware is not yet known, and it is not widely written about or particularly well understood.

Typically, as has been discussed on this blog and elsewhere in the past, it's much easier to look for artifacts of infection, rather than pre-infection activity.  Typically that involves creating monitoring/alerting for one of the following:

1) Looking for command and control URL patterns (which are much more reliable and have lower false positive rates than just domain names or IP addresses).  In this case, the callback traffic is encrypted, and thus, there is no URL pattern that can be monitored for (unless one intercepts the SSL traffic, but that is a separate discussion).
2) Looking for command and control domain names.  Here again, since the domain names change, this is not going to be very effective as a long term monitoring solution.
3) Looking for the malicious code payload delivery.  In this case, since it is a JAR being delivered with a content type of text/html, the noise is overwhelming.  Creating alerting based on this type of logic would inundate incident response personnel with alerts, the overwhelming majority of which would be false positives.  As I've learned, delivering a JAR via content type text/html is actually fairly common, even for legitimate applications.

So what does one do?  Well, since the callbacks all go to the same netblock, one can monitor passive DNS data for any new domains that are requested and resolve to that netblock matching the pattern described above.  That way, if the callback domains change (which they have been), and a host is infected, the incident response team can pick up on the new callback domain(s) and respond accordingly.  True, the incident response team is still one step behind the malware, but that's a lot better than being in the dark.

Pretty neat real world use of passive DNS data in my opinion.

Managing Up

It's always amazed me how a high profile incident seems to turn up management and executives in an incident response environment.  When a serious incident hits, everyone wants to hang out with the incident responders, even people who one might never see day to day.  In these type of situations, managing up is key.  Management and executives have the best intentions, but they don't work with the data day to day, and their technical skills may be a bit dated.  Suggestions on how to proceed, what analysis to do, and how to do it will fly at an incident responder faster than he/she can process them.  Unfortunately, many ideas that seem good in theory are not good ideas in practice.  The volume and variety of data makes finding the right approach tricky, and many behaviors that seem like they would indicate malicious activity don't.  Trust your senior team members and manage up in the best way you can.  Your team will be more productive because of it, and your senior team members will thank you for it.

Focus

When performing incident response, focus is extremely important.  A significant incident can produce innumerable leads and avenues to investigate.  Unfortunately, not all of these leads/avenues are productive ones.  Choosing poorly can have the unintended consequence of locking up resources for days while producing very little value-added analysis.  It is often difficult to know which direction or directions to go in analytically.  In my experience, senior members of the incident response team, who base recommendations on past experiences, lessons learned, and day to day familiarity with the network and its data have good advice to offer here.  That being the case, I've always wondered why management ends up driving high profile incidents.  It's a bit of a wonder if you think about it....

Over Classification

In theory, as a large enterprise and/or critical infrastructure provider, building a working relationship with the US government can provide valuable intelligence (in both directions).  Where the theory often breaks down in practice, however, is that the government suffers from issues arising from severe over-classification of data.  I understand that there are certain sensitivities and secrets that must remain closely guarded, but lists of malicious domain names do not fall within that realm.  The attackers already know where they are attacking us from, so we are not keeping anything from them.  They also already know that we know about them (it is very difficult to truly disguise network defense measures).  Moreover, malicious domain names themselves are not really all that valuable anymore (reference earlier blog posts), and on top of that, are often re-purposed by multiple actors/groups.  I frequently see domain names that used to mean one thing, but today mean something else or multiple different things.

So, my question remains, why guard these so tightly?  Can't we all agree that withholding this intelligence hurts the overall security posture of the United States?  Seems like the opposite of what we were aiming for.

Anomaly Detection

Most network security monitoring techniques use one of two approaches:

• Signature-based detection (i.e., "I know this specific activity is bad")
• Pattern-based detection (i.e., "I know this pattern of activity is bad")

Those are both well and good, but they leave a gaping hole.  What is the answer to the question: "Is this previously unknown activity normal and expected, or is it weird and unexpected?"

The way to answer that question is through anomaly-based detection techniques.  Unfortunately, at the present time, we as a community do not have so many mature, production-ready approaches to anomaly-based detection, nor do we have many vendor options.

I am cautiously optimistic that in the coming years, we will begin to mature out capabilities in this area.  It is sorely needed.

The Final Frontier

If you think about it, the final frontier for network monitoring is most likely the internal network.  We as a community have become quite good at instrumenting the edge and somewhat proficient at monitoring it.  Most of us, interestingly enough, have no idea what is going on inside our perimeter.  This is something that requires serious thought and attention in my opinion.  What lies beneath?  That is the question that we should seek to answer.