The Right Pivot is Everything

Every incisive query into network traffic data needs to be anchored, or keyed, on some field.  This is, essentially, the pivot field.  Pivoting on the right field is crucial -- I've seen inexperienced analysts spend days mired in data that is off-topic and non-covergent.  In some cases, simply changing their pivot vantage point results in an answer/convergence in a matter of minutes.

For example, consider the simple case of a malicious binary download detected in proxy logs.  If we want to understand what else the client endpoint was doing around the time of the download, we would pivot on source IP address and search a variety of different data sources keyed on source IP address during the given time period.  If we want to quickly assess who else may have also downloaded the malicious binary, we would pivot on domain and/or URL.

Naturally, these are simple pivots, but the point is a good one.  Take care to use the right pivot.  Otherwise, the results may be confusing, divergent, and inconclusive.

Peer Inside The Tangled Web

Most enterprises do a reasonably good job monitoring their edges, or at the very least logging network transactions traversing those edges these days.  What's interesting though is that most enterprises have almost no visibility (or interest for that matter) to understand what's happening in the interior of their networks.  I have found there to be great value in peering inside the tangled web that is an enterprise network.

There are several data sources that can prove extremely valuable for monitoring the interior of a network:

1. Interior firewall logs
2. DNS logs
3. Network flow data (netflow)

Allow me to illustrate with two examples I commonly see.  The first example deals with detection an infected system, while the second example deals with tracing back an indicator of compromise (IOC) to an endpoint (infected) system.

First, let's consider the example where an enterprise monitors proxy logs and DPI (Deep Packet Inspection) at the edge.  Let's say that a client endpoint was re-directed via a drive-by re-direct attack (e.g., Blackhole Exploit Kit), downloaded, and was subsequently infected by a malicious executable.  Further, let's say that the malicious executable that was downloaded is not proxy aware.  If we miss the executable download (which happens somewhat regularly, even in an enterprise that is monitored 24x7), then our proxy and DPI will likely be of little help to us in detecting the artifacts of intrusion.  This is because of two main reasons:

1. The malicious code is not proxy aware, and thus its callback/C&C (command and control) attempts will most likely be blocked by an interior firewall.
2. The infected system will likely attempt domain name lookups for callback/C&C domain names.  Even if these domain name requests resolve (they don't always resolve, i.e., in cases where the domain name has been taken down), there will no subsequent HTTP request (remember, it was blocked by an interior firewall).  Because of this, there will be no detectable footprint in the proxy log.  In the DPI data, the DNS request will be co-mingled with the millions of other DNS requests and will show as coming from an enterprise DNS server.  This makes detection and endpoint identification nearly impossible.

I believe we can now see the value in retaining, monitoring, and analyzing the three data types I've identified above:

1. Interior firewall logs will allow us to detect attempts to connect to callback/C&C sites that have been blocked/denied.
2. DNS logs will allow us to identify endpoints requesting suspicious/malicious domain names.
3. Netflow data will allow us to very quickly identify other systems that may be exhibiting the same suspicious/malicious behavior.

 Consider a second example as well.  Let's assume that an indicator of compromise (IOC) has been detected leaving and/or entering the edge of the enterprise network.  What needs to happen first and foremost is endpoint identification.  How many endpoints are infected?  Where are they located?  When did the activity begin?  What was the infection vector?  In the modern enterprise, with many layers of NATing and firewalls, the right logging is key here.  Without it, tracing back the infection to the endpoint(s) will be nearly impossible.  None of the preceding questions can be answered before endpoint identification is completed successfully.

Hopefully this hard earned advice is helpful to all.

Best Practices

Around the information security community, you'll often hear people talking about "implementing best practices".  While best practices provide helpful guidance at a high level, I'd argue they can't be "implemented".  Implementing something involves wrestling with the realities, nuances, and imperfections of a real enterprise network.  This pain is felt particularly acutely when developing reliable and actionable jumping off points for analysis.  As I'm sure you're aware, the number of false positives caused by blindly implementing "best practices" is enormous and is enough to stifle any incident response workflow.  What I've found over the course of my career is that the best and most reliable jumping off points are created through sweat equity that comes from an iterative cycle of intuition, data-driven refinement, and automation.  No manual, white paper, or vendor can sell you the secret sauce.

So how does one get there?  By using analysis (of real data) to navigate the difficult path from conception to implementation.  Know your network.

Actions Speak Louder Than Words

We all know the popular phrase, "actions speak louder than words". What some of us may not realize is just how wise and true that statement really is. In my experience, I've seen people express words of respect for others, but it's rarely that I witness people demonstrating repect for others through their actions. The difference is important and is one that people notice. Here's to action.

Artifacts of Intrusion

As many of us in the network forensics/network traffic analysis field know, looking for the artifacts of infection/intrusion can often be much easier than catching the actual infection.  Regardless of how one "latches on" to or otherwise becomes aware of an infected system, one should also take the time to perform due diligence in confirming that the system is indeed infected.  Here are some points that can be helpful during this process:

1. Look for the infection vector.  Was it web-based?  Email-based?  Some other means?  Can information relating to the infection vector be used to harden the enterprise or otherwise improve the enterprise's security posture?
2. Check anti-virus or host-based IDS/IPS logs.  Was the threat detected and remediated by either of those?  Could there have been more than one threat, one or more of which was perhaps not caught by either of those?
3. Can a sample of the malicious code be isolated for analysis?  Does analysis of the malicious code sample provide any information or intelligence (such as callbacks or other artifacts of intrusion) that can be fed back into the analysis process and/or used to harden the enterprise or otherwise improve the enterprise's security posture?
4. Look for the artifacts of intrusion in the network traffic data.  Does the network traffic data provide evidence that the malicious code successfully infected the machine?
5. Based on the information gathered in steps 1-4, make an educated decision, rather than a happenstance decision, regarding remediation.

This certainly isn't an exhaustive list, but it does provide a skeleton analysis process for malicious code incidents.  Perhaps useful to those who may be less familiar with or less experienced in this realm....

An Inspiring Phone Call

Last night, I received a phone call from a neighbor asking if he should renew his home anti-virus subscription.  I typically shy away from giving people advice on this sort of question, but it did give me pause and caused me to do some thinking before I answered him.

I thought about how in most large enterprises that I advise, anti-virus is <25% effective at detecting malicious code threats.  Worse yet, most anti-virus vendors have not only sold large enterprises software that is largely ineffective, but they have also sold those enterprises on a mantra/workflow/philosophy that is nearly useless.  In other words, most anti-virus vendors will suggest that you review their logs regularly to identify systems infected with malicious code.  I remember from graduate school that this is, by definition, the quintessential biased sample.  In other words, since anti-virus is <25% effective at detecting and identifying threats, the enterprise is left with a gaping hole/blind spot of >75%.  Stated another way, if you trust anti-virus logs to guide your analysis, workflow, and/or process, you will be left largely in the dark.

The "Know Your Network" philosophy discussed in this blog and elsewhere is really the only way to effectively monitor a large, enterprise network.  Unfortunately, anti-virus vendors do not seem to be interested in improving through novel or cutting edge techniques.  They are quite content to sell both home users and enterprises the 21st century equivalent of a bridge.

Of course, anti-virus logs are somewhat useful as a supporting/corroborating data source for investigations that begin with true network forensics/network traffic analysis, but not much else.  In other words, anti-virus logs can tell you if the malicious code that you see a user downloading in the proxy logs or via deep packet inspection (DPI) data was somehow detected and cleaned by anti-virus.  As an aside, even this is not 100% reliable though, as it is not uncommon for malicious code infection vectors to try several different exploits and/or executable downloads.  Just because anti-virus caught one of them, doesn't mean it caught all of them.  The analyst still needs to examine the proxy logs, firewall logs, DPI data, and/or any other relevant data looking for artifacts of infection.

So back to my neighbor.  After thinking about this for a moment, I explained to him that the anti-virus vendor he was thinking of sending additional money to was really no more effective than some of the free products that are available.  I advised him to hold onto his money.  After all, if one ignores the hype and thinks about it logically, it only makes sense.

Passive DNS Expansion

Recently, I learned that an analytical method I had experimented with a while back was being used by an organization to generate intelligence.  It would have been nice to have been given proper credit, but regardless, it's still great that the method is in use.  I've called the method "Passive DNS Expansion" for lack of a better term.  The basic idea is that you use a combination of forward DNS resolution and passive DNS data to generate lists of malicious domain names that can be used for analysis and/or network defense purposes.  It turns out to be a pretty handy way to generate some actionable intelligence.  The basic algorithm is as follows:

1) Begin with a known malicious domain name
2) Forward resolve the known malicious domain name and obtain its current IP address
3) Leverage passive DNS data to identify the domain's IP address history
4) Search passive DNS data for the IP addresses to obtain a list of domain names associated with those IP addresses
5) Perform some cursory research on the domain names to determine whether or not they can be reliably used for analysis/network defense

I'd like to illustrate this with an example.  I picked the domain name at the top of the Zeus Tracker list this morning to use as an example for illustrative purposes:

1) I begin with the domain name freetop[.]mobi, which I obtained from the Zeus Tracker list
2) The domain name forward resolves to 69[.]175[.]127[.]82
3) Passive DNS data shows only one IP address for this domain: 69[.]175[.]127[.]82
4) In this case, there is only one additional domain name associated with this IP address in passive DNS data: mymobilewap[.]info
5) A Google search for mymobilewap[.]info indicates that the domain name is most likely malicious and could probably be used reliably for network traffic analysis purposes.


I'm a big fan of this method.  In the past, I've scripted these steps to automate the process with positive results.  Feel free to give it a try and let me know your thoughts!