Collect It, Keep It, Need It

According to the 2014 Verizon Data Breach Investigations Report (DBIR), as well as the Mandiant M-Trends 2014 Threat Report, intrusions remain on networks for many months before they are detected.  Further, organizations do not discover the breaches themselves, but rather, are notified by third parties a majority of the time.  These two facts bring with them an unfortunate reality -- when it comes time to do incident response, the data required for incident response is often not available to the incident response team.

There are several common reasons why the required data may not be available:

• Collection: In some cases, organizations may not have their network properly instrumented for collection.  In other cases, organizations may not be properly equipped to retain and expose for analysis the volume of data created by the network instrumentation.  Either way, when it comes time to investigate, the relevant data will not be available.
• Visibility: Some organizations may have portions of their network instrumented for collection.  But what if the breach occurs in an area of the network that is not included in the area of visibility?  In those cases, data that is relevant to the breach investigation will not be available.
• Retention: Sometimes, the network is properly instrumented in the appropriate places, but there is simply nowhere to put the volume of data that is generated.  As the volume of data grows, either the retention period shrinks, or the storage capacity grows to compensate.  It is not uncommon for the retention period to get down to one month or even less, making incident response for breaches with long-term presence extremely difficult.

The collection and visibility issues can be addressed through improving instrumentation of the network for collection.  But what about the retention issues?  Collecting fewer data sources of higher value to security operations can help, as I discussed in my “Data Value vs. Data Volume” post (http://ananalyticalapproach.blogspot.com/2014/04/data-value-vs-data-volume.html).  Ultimately, even with optimization of data collection, a large network will generate a large volume of data.  Given the amount of time breaches remain present before detection, and the key role network traffic data plays in investigating a breach, organizations are retaining network traffic data for increasingly longer periods.  It’s probably a good idea to retain 6-12 months of meta-data (or more) if possible.  Granted, this involves budgeting resources to accomplish this.  Given the financial damage recent breaches have inflicted, if done properly and optimally, increased retention seems like a good investment.

A Way Forward In Information Sharing

In a recent piece in Security Week, I discussed the various challenges I perceive relating to information sharing (http://www.securityweek.com/understanding-challenges-information-sharing). Although I won’t rehash the details of the piece here, I did want to discuss a related point. I’m sure we can all understand the challenges, but what can an organization do about it?

In spite of, or perhaps because of, the challenges in information sharing, much information sharing happens informally through ad hoc trusted relationships and informal information sharing forums. It may be preferable for organizations to have formal agreements and policies in place, but we are not there as a community yet. Until that time, security practitioners still need to exchange data to perform their jobs properly, and as such, informal relationships exist. That being said, there are still steps that can be taken to formalize information sharing efforts a bit more.

Communication and education are extremely important and seem to me to be good starting points. Security leadership within an organization can communicate the information sharing vision. A dialogue can begin with legal, privacy, and other relevant stakeholders within the organization, so that they can be educated as to the value and importance of information sharing and included in the efforts. People can begin tackling the information sharing challenge and working to build the organization’s street cred and foster collaborative relationships. The importance of information sharing can be communicated to external organizations and entities that can better facilitate formalized information sharing.

Once communication and education are underway, a formal information sharing process can be developed. This process will include details regarding what type of information may and may not be shared, what to do if information is shared that should not have been, as well as the actual nuts and bolts around how information is collected, handled, used, and shared. Process in itself brings more formality to an information sharing effort and is an important part of the overall picture.

Technology is also important. Technology that facilitates, rather than fights, information sharing is a must. The data of record should be recorded with no losses or gaps. Searches for evidence of Indicators of Compromise (IOCs) should complete rapidly. It should be straightforward and smooth to both both receive and share information. All of these factors contribute to enabling and empowering successful information sharing, rather than fighting it.

Like most security endeavors, information sharing comes back to people, process, and technology. All of them are important, and all of them play an important role in a successful information sharing effort. Most people say that information sharing is a critical piece of a complete security operations program, and, as a result, it should be given the proper attention accordingly.

Servers, Shovels, and On-line Sales

eBay is in the news this week, having been the latest high profile victim of a breach. From the media reports I’ve seen, it appears that the attackers compromised certain eBay employee credentials and then used those credentials to log into internal databases containing information about eBay’s users. While the story has been widely covered, I thought it would be interesting to look at it from a different perspective than I’ve seen in the media coverage.

This breach interests me because it brings together two concepts I’ve written about in the past: “The Forgotten Servers” (http://ananalyticalapproach.blogspot.com/2014/03/the-forgotten-servers.html) and “Don’t Forget to Dig” (http://ananalyticalapproach.blogspot.com/2014/05/dont-forget-to-dig.html).

I won’t rehash the details of those posts here, but in essence, identifying server compromises is extremely difficult. Detecting infected endpoints is much easier in comparison, and because of that, it typically dominates the security operations workflow. As we know though, servers generally contain much more valuable information, but it is much more difficult to detect when they have been compromised. Server compromises generally live in the realm of the unknown unknowns, and they are usually far more serious than endpoint compromises.

This is where digging becomes important. While it may someday be possible to write reliable, high fidelity, low noise alerting for server environments, we’re not quite there as a community yet. No matter how rich and actionable our work queue of alerts is, we still need to dedicate some resources to work outside that linear flow. Those resources should be used to perform intensive analysis and “dig” through the network data using a variety of techniques. Server environments seem, to me at least, to be a great place to begin a digging initiative. Servers are valuable resources with valuable information that is worth keeping an eye on.

Bring your shovels. You’re going to need them.

Shift Perspective

If you’ve ever worked in a security operations environment, you’re aware of how much day-to-day work revolves around systems (identified by IP address, MAC address, hostname, or otherwise). Alerts and events are generated per system. Investigations are based on detecting, analyzing, containing, and remediating infected systems. Intelligence is primarily leveraged to identify systems of interest. What’s interesting to me, though, is that if we take a step back, we see that systems aren’t actually the true pivot point -- users are.

Each user in an organization may use a number of different systems. For example, a given user may have a desktop computer, a laptop computer, a tablet, smartphone, and other devices. Likewise, each system may be used by a number of different users. For example, a virtual desktop environment may have one IP address but be used by dozens of users. If all of our analysis is centered on systems, we miss the correlation that arises from linking a single user to multiple systems, or conversely, multiple users to a single system.

Why is this important? Let’s have a look at what happens when we shift our perspective for a few different use cases.

Insider Threat: Insider threat is topic on many people’s minds these days. Whether the concern is a rogue employee, espionage, or something else, insider threat is a challenge designed to be approached from the user perspective. Trying to identify insider threat activity is already extremely difficult. Trying to identify it solely by analyzing the activity of systems, rather than analyzing the activity of users is nearly impossible.

Serial Offender: What is the difference between five different systems infected over a period of a few months and a serial offender? Correlation at the user level. Sometimes users have bad security “hygiene” that causes them to pose a greater risk to the organization. Taking a user perspective allows us to identify serial offenders and take steps to address the issue.

Lateral Movement/Staging for Exfiltration: From the system perspective, lateral movement and staging of data for exfiltration look very similar to legitimate network activity. The difference lies mainly in intent, which is nearly impossible to infer when looking at the problem from a system perspective. Looking at the problem from the user perspective allows us to gain an edge. Correlating activity to the user allows us to see if users are logging in from unusual places or logging into unusual places, among other suspect behaviors. Perspective changes everything here.

Stolen Credentials: Two systems may log in to a server or access a file share at the same time, and we would think nothing of it. But if the same user account was used at the same time from two different systems in two different divisions of the organization on two different sides of the globe, that activity becomes a bit more suspect. Looking at the activity through the lens of user-level correlation allows us to tease out the difference.

Essentially, systems are merely tools leveraged by users. Taking a different vantage point that allows us to correlate activity by user, rather than by system alone gives us a very different perspective. That different perspective allows us to better identify and analyze certain types of activity on the network that we may want to investigate further.

Knowledge Without Borders

Knowledge knows no borders. Contributions to the collective human understanding have come from all different types of people. Not surprisingly, this is also the case within the security profession. We all learn from an incredibly diverse set of peers, and this is a good thing. Each person’s experience brings with it a fresh perspective that can be applied to the challenges at hand, and in the security profession, the challenges are many.

Unfortunately, there are still some people in this world that will discount, dismiss, or exclude ideas, experience, and expertise because of the race, religion, creed, ethnicity, or national origin of the people they belong to. Aside from the obvious personal pain this inflicts on the victims of this type of hatred, there is also a professional issue that arises from this that I would like to discuss.

Each security professional has a professional duty to protect his or her organization to the best of his or her ability. The attackers are constantly learning new tricks, improving their skills, and modifying their behavior. The risks to an organization resulting from an attack continue to rise. The threat landscape is continually evolving. Even under the best circumstances, we as defenders can barely keep up. If we are to have any hope of successfully protecting the valuable assets and information we are charged with protecting, we need all the help we can get, from any reliable and trustworthy source available.

When someone turns away, discounts, dismisses, or excludes ideas, experience, and expertise because of their origin, that person is putting his or her organization at great risk for no good reason. In essence, politics and personal prejudices are being put ahead of professional duty. It should come as no surprise that this is not acceptable -- the organization and its clients, shareholders, partners, leadership, and employees deserve and demand better. Security knowledge should be valued and respected regardless of its source. Anything less would simply be unprofessional.

Operational Experience

I firmly believe that there is no substitute for operational experience. I try to make it a daily practice to read blogs, articles, and other posts from around the information security community. This allows me to keep up with the latest news and developments, as well as to understand and learn from the views of others. I find it rather interesting that I can usually tell the difference between authors who have operational experience in the security field and those who do not. I often check my assessment via LinkedIn, Google, and other means, and I am usually correct. I’ve always wondered why there is such a clear delineation between the writings of those with operational experience and the writings of those without.

As the Albert Einstein quote reminds us, “In theory, theory and practice are the same. In practice, they are not.” This is a salient point. There is no shortage of ideas, theories, and suggestions for improving the state of security, but how many of them are rational, practical, and realistic? Operational experience causes people to see the world from a different perspective. It causes people to identify practical suggestions that can be implemented and operationalized in a realistic timeframe and without an unrealistic amount of resources. Further, operational experience causes people to place more of weight on the ratio of the resulting impact to the effort required to produce that impact, rather than other potential decision making metrics. In my experience, operational experience enables better decision making and produces a better result, whatever the undertaking. This is particularly true in the security community, where resources are quite limited and expectations are quite high.

So, perhaps it is important to think about key personnel and decision makers within your security organization, at your vendors, and at your consultancies. What is their level of operational experience and familiarity with the issues you face? Have they spent time in the trenches? Are they making decisions and offering advice based on a solid foundation of formal training and on-the-job experience? In my experience, these are important questions to consider when hiring, selecting vendors, and retaining consultants. There is really no substitute for operational experience.

Difference Between Logging and Alerting

Based upon some conversations I’ve had lately, I feel comfortable stating that some people don’t understand the difference between logging and alerting particularly well. There is an important difference, and it becomes noticeably more important as the volume, velocity, and variety of data within a security operations setting continue to grow. In this post, I hope to illustrate the difference in order to help people mature their security operations programs.

In essence, I think much of the confusion stems from ambiguity around the meaning of the word “alert”. In a security operations setting, an alert is something that needs to be qualified, vetted, and acted on appropriately. Every alert that presents itself to the work queue should be reviewed. Due to resource limitations, an organization can realistically handle only a small number of alerts per day -- say on the order of hundreds per day at a typically resourced organization. We’ve all met people who make statements like, “Our SOC handles 100,000 alerts per day”. Bollocks.

Unfortunately, many technologies use the term “alerts” when they are really producing “logs”. For illustrative purposes, consider the following example. Say we install an IDS, outfit it with the latest ruleset, and set it loose. It will produce tens of thousands or hundreds of thousands of events per day. The vendor will call them alerts, but let’s take a step back and think about what the IDS is producing in actuality. The IDS is producing a notification every time it sees a packet matching a signature that the IDS maintains. To me, this is a log -- a log of network events. I don’t intend to belittle or pick on IDS -- there is huge value to be brought to security operations by IDS and the events it produces. Rather, I am saying that IDS is essentially producing logs to be sent to the SIEM or data warehouse, just like any other network sensor produces.

So, I’m sure you will ask me, “Well, what do you consider an alert then?”. Great question -- thank you for asking. To me, an alert consists of one or more events that match a defined risk, threat, concern, and/or priority to the business with a low incidence of false positives. For example, it is possible that thousands of IDS events, hundreds of proxy log entries, and a DLP event could meet the criteria of a particular alert we have developed. Thus, one alert would fire that happened to involve thousands of event logs.

All of the various network sensors we have are important and contribute to our overall security posture. But, it’s important to remember that they produce logs, and not alerts. Alerts should be specifically designed around our requirements and use cases. Alerts are what get bubbled up to our work queue and provide us with jumping off points into analysis, forensics, and investigation around activity of interest. Because of this, alerts need to be higher in sophistication and fewer in number than logs. The logs that our network sensors produce are an important component of our alerts, but they are not alerts in and of themselves. In my experience, this is an important concept that, when properly understood, contributes to the differentiation between weaker security programs and stronger security programs.