Moving up the Stack

Over the past decade, attackers have moved deeper into the packet. As network defenses, controls, and detection techniques have improved, attackers have had to "move up the stack" to avoid detection and maintain persistence. A decade ago, attackers would often use dedicated resources on known malicious networks to communicate with their malicious code. Back then, malicious IP blacklists were common, and they were used widely to detect malicious activity. Back then, it was often possible to detect malicious activity using only data available at layer 4 of the OSI model.

Over the years, attackers were forced to move away from dedicated resources to more dynamic resources. This was due in part to dynamic resources becoming easier and cheaper to use, as well as efforts by law enforcement, telecoms, and others to shut down dedicated resources. Because of this transition, attackers were forced to change the way they operate and communicate with their malicious code. Domain names and URL patterns, both of which are only found in the data available at layer 7 of the OSI model, became the preferred means for malicious communication. Moving up the stack allows attackers to change where their malicious code communicates easily and at a moment's notice. This new paradigm requires layer 7 data to detect intrusions -- layer 4 data is no longer sufficient.

Attackers have already made the move up the stack. On the enterprise side, we need to make sure we have the tools to move up the stack with them. Layer 7 enriched meta-data provides us a solid foundation upon which to perform the network forensics we need to perform in order to detect and respond to intrusions in a timely manner.

Shrinking the Rack

It will come as no surprise that most enterprise security organizations are perpetually understaffed and resource constrained. This is especially true with regard to the resources required to deploy, maintain, and use enterprise security technology. Because of this, choosing which security technologies to deploy is an important decision that has many ramifications and consequences. This blog posting explores the concept of collecting fewer data sources of higher value, rather than a larger number of data sources of lower value. In effect, "shrinking the rack" of network instrumentation equipment without adversely affecting visibility, monitoring, and operations.

Most enterprises instrument their network to collect highly specialized forms of data. For example, an organization may have a netflow collector, a DNS tap, and a variety of other technologies to collect highly specialized forms of data. This creates a stream of various different data types and formats that complicates and clouds the operational workflow. I note this as an observation and do not mean it to be critical. Having worked on the operational side for over a decade, I understand quite well that for historical and other reasons, enterprises have had to make difficult choices to meet operational needs under less than ideal conditions. What I am suggesting is that as equipment comes to end of life, as priorities change, and as the threat landscape evolves, enterprises take a second look at what goes into their network instrumentation and monitoring rack.

In addition to the variety and complexity of specialized forms of data, the volume of data confronting enterprises these days is also overwhelming. So, this adds huge quantities of data to the already complicated operational security workflow. This perfect storm of circumstances creates a very real big data challenge.

Let's consider a unified, consolidated approach provided by PCAP data and its associated meta-data. Full packet capture (PCAP) records all traffic transiting the network. The meta-data that can be parsed out of both layer 4 (network layer) and layer 7 (application layer) provides an incredibly rich data source for the big data challenge. This rich meta-data also includes the specialized data forms often collected within enterprises. For example, if we look at the data obtained from the DNS tap example, we see that it is functionally the same as the DNS protocol meta-data parsed out of PCAP data. The potential this provides for simplification, streamlining, and optimization of workflow and operations is incredible.

We live in a world that seems to get increasingly more complex. Those of us that work in the security operations field feel this on a daily basis. Shrinking the rack allows us to simplify, optimize, and reduce complexity within our environments without sacrificing visibility and functionality. As you think about replacing, upgrading, or modifying your network instrumentation equipment, it may be helpful to consider shrinking the rack. Sometimes, less is more.

Breaches Happen

Recently, a number of high profile data breaches have been in the news quite a bit. The general public is quickly becoming aware of something that security professionals have known for quite some time -- that compromises and breaches are going to happen. What separates good organizations from great organizations is really their preparedness and response during these types of high profile incidents.

Typically, immediately after an enterprise becomes aware of a data breach, several important, focused, and pointed questions will be asked of the security organization by the company's leadership, as well as the members of the company's legal and privacy organizations:

• How did this happen? 
• When did this begin?
• Is this activity still occurring?
• How many systems/brands/products have been affected?
• What sensitive, proprietary, and/or confidential/private data has been taken?
• What can be done to stop this activity/prevent it from happening again?

These questions are appropriately and pointedly aimed at assessing the damage, understanding the legal, privacy, and business ramifications of what has occurred, and formulating a plan to respond as required. The answers to these questions require hard facts that can only be found in the enterprises's network traffic data.

Given this, it is perhaps a bit surprising that when many organizations attempt to answer the above questions, they will find that they do not have the appropriate data in place to do so. There can be many reasons why this is the case, but chief among them are:

• Incomplete/partial instrumentation of the network (i.e., the required data is not being collected at all required points of presence to provide the appropriate answers)
• Inadequate data retention (i.e., the required data is not being retained sufficiently long enough to provide the appropriate answers)
• Inability to leverage the data for analysis (i.e., the required data may or may not be collected, but is retained in a system or systems that do not allow for it to be exploited analytically to provide the appropriate answers) 

People, process, and technology are said to be the elements upon which great organizations are built. When data breach response time comes, having the right people and the right processes in place is undoubtedly critical. Equally critical though is having the right platform in place that allows for both collection AND analysis, so that a great organization's people and process can shine and perform incident response properly.

Aggregation and Outbound Denies, A Powerful Combination

I have previously blogged (separately) about the merits of both aggregation and looking at outbound denied traffic.  It occurs to me that it is worth a separate post to blog about the powerful combination of aggregation and outbound denies.

If one takes a rich data source (such as proxy logs), looks at the outbound denied traffic, and aggregates by certain key fields, such as:

• Source IP Address
• Destination IP Address
• Domain
• URL
• Request Method (e.g., GET, POST, etc.)
• Count (ordering by Count in descending order)

The results (say, over a 24 hour time slice of data) are generally quite interesting.  Taking a step back, we see that what we're essentially doing is slicing the data in such a way so as to extract repetitive activity that is being denied by the proxy (for whatever reason).  Generally, humans do not create traffic that fits this criteria, but rather, machines do.  Machine generated activity is generally quite interesting from an analytical perspective, though sometimes it is a mere nuisance (e.g., toolbar generated traffic).  On the frequent occasion that the traffic is malicious, this approach to slicing the data is quite helpful in finding the activity of concern.

Big Data Requires a Surgical Approach

I've previously posted about the overwhelming volume of data confronting enterprises today, as have countless others.  Although I have hinted at it through many of my blog posts and provided several tangible, hands-on examples in this blog, it occurs to me that I have never overtly stated that "big data requires a surgical approach".  What does this mean?  Essentially, with billions of transactions/records/sessions per day, the volume of data has grown beyond the capability of organizations to handle it.  The enterprise's network data must be sliced up surgically using a variety of different techniques.  Each technique takes a slightly different view/vantage point of the data and produces a reduced, filtered, and more manageable volume of data for investigation.  If this sounds familiar, it is because it is essentially the same approach as I've blogged about previously in posts discussing what I call the "Jumping Off Points" approach.  The approach is simple, straightforward, and effective.  The challenge is finding the most relevant and value-added jumping off points, and continuously working to improve them and to find the next group of relevant and value-added jumping off points.

Less Tangible Elements of World Class Security Operations/Incident Response Functions

I have had the privilege of working with a number of different incident response/security operations functions within a number of different enterprises.  What I have come to realize is that in addition to all the tangible elements that running a successful security operations/incident response function entails, there are also several key elements that are less tangible, but equally as important.  Although harder to measure, these points are nonetheless equally as important:

• Strong leadership presence in the incident response/security operations community: The best incident response/security operations functions are run by people who have walked the walk and who are active members in the relatively small and close-knit community.  Quite simply put, incident response/security operations functions run by people who understand the challenges, can think strategically about how to approach them, and have the contacts, respect, and earned authority to implement the required strategic approach are more mature than those run by people who do not fit the above description.
• Realization that information sharing and incident response are one in the same: I often see that organizations have "Timely Incident Response" and "Information Sharing" as two separate strategic objectives.  Both are important, but if one thinks about it, they are effectively one in the same.  What does this mean?  That a) strong information sharing relationships can be one of the most effective ways to detect/understand/be notified that an incident is underway requiring response and conversely that b) when an incident is underway, having solid and strong information sharing relationships can be one of the most effective ways to handle/contain the incident (e.g., having close relationships with hosting providers that can take down sites for an organization).
• Proactive intelligence: Many organizations do decently well with reactive intelligence.  For example, if it becomes known that a given URL pattern is an indicator of malicious command and control (C2) activity, most organizations can immediately leverage this in their alerting.  Naturally, this is extremely important, but it is, in its essence, reactive.  Proactive intelligence is something that most organizations do less well.  It involves tracking the attackers and threat landscape to understand the direction in which threats/attacks are moving and how to translate that into actionable intelligence that can be implemented operationally.  This is no easy task, but it is something that separates the world class organizations from the rest of the pack.
• User/insider threat: The most serious compromises generally involve theft and/or misuse of user accounts, certificates, and/or other credentials.  Because of this, tracking, profiling, modeling, and identifying anomalous/suspicious/malicious user activity is essential to a world class security operations/incident response function.  Identifying anomalous user activity is a challenge, but it is one that the best organizations do not shy away from.
• Incident response/security operations is a cerebral business: It is tempting/easy to pay an overwhelming amount of attention to the operational component of incident response/security operations without paying enough attention to the cerebral component.  It is true that incident response/security operations necessitates a strong operational component.  What the best organizations understand is that the operational component supports the strategic, well-structured, intelligently-approached (cerebral) component/foundation, and not the other way around.

Hopefully these thoughts are helpful to those looking to build, strengthen, and/or enhance their incident response/security operations function.  Feedback is, of course, always welcome.

Process

To most analysts, the word process is a scary one that conjures up images of rote, check-box type work.  Although that does sometimes occur, in the Incident Response/Security Operations world, process is extremely important.  Why is this so?  Because in a field where data is so overwhelming, expectations are so high, and resources are so very limited, having an organized, well-structured, well-defined approach to the day-to-day workflow is extremely important.  Organizations that have a well-defined incident response process (at all different levels -- from the highest, strategical level down to the lowest, operational level) generally do much better in incident response than organizations that do not.

A good incident response process can help focus resources (software, hardware, and wetware) and maximize the value they provide.  Process isn't the sexiest of endeavors, but if done properly, it is one of the most productive and value-added.