Drowning in Information, Starved for Knowledge

In his 1982 book, Megatrends, author John Naisbitt penned the famous quote, "We are drowning in information but starved for knowledge."

This quote is particularly relevant to the security operations field. Information, or data, comes at us faster than we can make sense of it. In a large enterprise, daily log volumes can quickly rise to 5, 10, 20 billion rows of data or more. We can gain access to 30 or 40 "intelligence" feeds in the blink of an eye. Threat reports are more plentiful than the eyes available to read them. Information sharing groups and mailing lists may deluge us with hundreds of emails per day. As we know, there is a big difference between information and knowledge. We have more than enough information. How can we turn that information into knowledge?

In my experience, most Security Operations Centers (SOCs) feel this pain continually. As security professionals, we have quantities of information at our fingertips that would have been unimaginable 10 or 20 years ago. Yet, with all that information, we struggle to answer simple questions such as, "What is happening on the network right now?". The data, feeds, reports, and emails come at us continuously, and most organizations understandably struggle to make sense of them. While there is no easy solution to this complex challenge, I can offer some guidelines and advice that I have found helpful in bringing order to the chaos:

• Evaluate risk to the organization: Boiled down to its essence, security is about managing, reducing, and accepting risk, rather than eliminating it. Different organizations will be exposed to and concerned about different risks. Understanding what needs to be protected (e.g., sensitive information, intellectual property, money, reputation, etc.) is the first step towards understanding how to protect it. It is difficult to build a security operations program without a clear vision of what exactly that program is designed to protect.
• Create human language goals and priorities: Before diving into any technology or creating any alert logic, it is helpful to write down, in human language sentences, what you would like to accomplish. This is similar to the programming practice of writing pseudo-code before writing any actual code. This accomplishes two important things. First, it helps the organization to organize its goals and priorities, which can subsequently be used to frame and execute a work plan. Second, it serves to document those goals and priorities. Documentation (at all levels) is seldom fun, but it is an extremely important activity for a number of reasons.
• Identify appropriate data sources: Once goals and priorities are identified, the appropriate data sources can be identified to meet those goals and priorities. The relevant data sources can include network traffic data, various types of logs (network, end-point, and malware), intelligence feeds, threat reports, and other sources. It is important to consider all data sources relevant to a particular goal or priority and to explicitly identify and document the relevant sources alongside it.
• Identify appropriate technologies: Different technologies suit different needs. For example, for certain goals and priorities, a SIEM or data warehouse may be the appropriate tool. For others, a network forensics platform might be the right fit. Or, for yet others, something different entirely.
• Throw out the default rule set: This may sound radical, but for each technology, throw out the default or standard rule set. Why? Because it wasn’t written specifically for your organization. Are there specific signature sets, rules, logic, alerts, etc. that meet the needs of your organization? Absolutely, and those should be retained selectively. It’s important to remember, though, that many elements of the system’s default set won’t meet the needs of your organization. Keeping them in there will create noise and false-positives that won’t help you accomplish your goals and priorities.
• Write targeted alerting: Alerts are a powerful force in security operations and should be leveraged accordingly. Write alerting designed to identify suspicious, malicious, or anomalous activity as defined by your goals and priorities. Don’t bother writing any alerts that don’t fit your goals and priorities. Why? Because that will just produce additional noise and false positives that you’ve already decided you aren’t interested in.
• Streamline the workflow: Regardless of how and where alerts are generated, they should all flow to one unified work queue. Priority should be used to assist the team in identifying what to work on first, second, etc. Highest priority goes to the highest fidelity, most reliable alerts covering the most critical assets. Lowest priority goes to the lowest fidelity, least reliable alerts covering the least critical assets. The most important takeaway here is that one work queue allows your team to focus and provides them jumping off points into analysis, forensics, and investigation. More than one work queue leads to complication and confusion.
• Practice Continuous Security Monitoring (CSM): Once you set up alerting and a work queue, use it. Every alert should be reviewed by the team and investigated appropriately to build context around it and understand what occurred. Some alerts will require more investigation, while others will require less investigation. There is really no point in setting up alerts that you never intend to look at. That’s really the point of CSM -- reviewing each alert and performing analysis, forensics, and investigation as required.
• Follow a mature process: Process is the glue that binds people and technology together. If you have great people and have set up great technology (perhaps leveraging the suggestions provided in this blog), a great process is also required. Process helps the team focus on what tasks are value-added and converge to a conclusion. After all, analysis and forensics are not done for analysis’ and forensics’ sake, but rather, to reach a conclusion.
• Leverage automation where appropriate: Once a mature process is in place, study the application of that process operationally. If time-consuming, manual labor exists for certain aspects of the process, consider automation. If time can be saved in one place, it means that more time can be spent elsewhere. This leads to greater overall visibility on the network.
• Maintain a communal presence: We can learn a lot from our peers. Maintaining a presence in the broader security operations community ensures that an organization is in the loop regarding current events, topics, discussions, and issues. All of these factors play a role in ensuring that the security operations program changes with the times, and that information is shared in a timely and relevant manner. Just remember -- this relationship involves both giving and receiving.
• Continuously improve: Never believe that the work has been completed. Technologies, methodologies, and the threat landscape change continuously and quickly. It is important to continually seek feedback and improve. The steps described here are iterative and should continually be stepped through in a cyclical fashion.

There is no one size fits all solution to security operations. That being said, I have found the above guidelines to be quite helpful when working with enterprises to improve the state of their security operations programs. My hope is that if individuals find this advice and other advice on this blog helpful, that they will share with their peers, colleagues, and friends. Contact me on Twitter (@ananalytical) or drop me a comment here to let me know what you think. Your input is important to me as well.