The Business of Security Operations

When an organization decides to build, mature, or improve its security operations function, that decision will give rise to a building or maturing process.  That process will involve many important questions and a long list of tasks to complete.  The questions and tasks are a means to an end – the outcome of the process is what we’re ultimately interested in.  In my experience, some organizations end the process with a mature capability, while other organizations struggle to achieve maturity.  Why is this?  I’d like to take a look at some potential reasons in this post.  At a high level, security operations should be approached more like a business function and less like a laboratory exercise.

Self-Awareness:  Acknowledging that capabilities need to improve is often half the battle.  Self-awareness comes with a dose of humility that allows us to learn from others that have come before us.  There are a lot of lessons learned that can be leveraged, but the listener needs to be receptive to the input.  There is no shame in acknowledging the need to improve.  Quite the contrary -- it is to be applauded.

Vision:  In my experience, when it comes time to build or mature the security operations program, it is a natural tendency for an organization to immediately begin deploying technologies.  It’s important to remember that a clear and concise leadership vision is an essential pre-requisite to any action.  Before we can begin building capabilities, we need to understand towards what goal we are building.  Ultimately, a successful security operations program involves carefully leveraging people, process, and technology toward a focused vision.

People:  Once a vision has been articulated, it needs talented people to carry it out.  People form the foundation of a security operations program, and the right people are critical to its success.  It’s important to take adequate time to recruit and retain the appropriate staff.  It may be the case that an organization needs to bring in a trusted partner for the staffing process, and that is okay.  I have seen instances where poor staffing choices have been made, and they work against the vision, no matter how grand it is.

Process:  Like any business function, a good process is critical to a successful security operations program.  The process should be written at multiple levels, and it should serve to guide the organization’s human resources as they go about their daily tasks.  At the strategic level, the process should address goals, priorities, and high-level workflow.  At the operational and tactical levels, more depth and context should be included to provide specific instruction around categories of incidents and/or families of analysis, investigation, and response actions.  Aside from guiding the team, a detailed process also demonstrates to executives, partners, customers, and other stakeholders that the organization takes a formal approach to security.

Technology:  Obviously, technology is the third component of the people, process, and technology triad.  Technology should enable and empower people to execute the process and make the vision a reality, rather than work against that endeavor.  When purchasing technology, it’s important to identify technology that addresses operational needs – namely, gaps in the execution of the process that technology can help address.  Before technology is acquired, it should be matched to an operational need and integrated into the operational workflow.  I’ve seen many instances where numerous different technologies were procured without thinking strategically about where the different pieces fit operationally, and the results made creating a streamlined and efficient workflow difficult.

Workflow:  In most cases, resources are scarce in a security operations environment, and in particular, human resources are usually the scarcest.  Because of this, it’s important to develop alerting content designed to identify strategic risks to the organization.  This content should generate fewer alerts of higher quality and fidelity that are more actionable.  Quality is far more important than quantity here, and the signal-to-noise ratio should be high enough to facilitate timely detection.  All alerts should be sent to a unified work queue that analysts can be focused on, and each alert should be reviewed, vetted, analyzed, investigated, and responded to appropriately.  In my experience, there is no point in generating 100,000 alerts each day if you can realistically only handle several hundred of them properly on a given day.  The risk of missing an intrusion or breach because it was lost in the noise is simply too high.

Communication:  During execution of the incident response process, and during the course of daily security operations as well, communication is key.  Aside from metrics and other important information that need to be regularly communicated to leadership, communication serves another important purpose as well.  Relationships with upstream providers, peer organizations, professional associations, partners, customers, legal, privacy, and other stakeholders are incredibly important.  Having those relationships in place ahead of time can help ensure that when crunch time comes, the appropriate channels exist to disseminate, receive, and act upon information in a timely manner.

Information Sharing:  The knowledge of 100 organizations will always be greater than just one.  Sharing information allows us to broaden our perspective and view the challenges of security operations through a much larger lens.  Techniques, methodologies, and Indicators of Compromise (IOCs) are all great information that can be shared between organizations.  Those who give the most generally receive the most, and building street cred for your organization is important.  Sometimes, being remembered can mean the difference between getting timely intelligence and not getting that intelligence.

Building or maturing a security operations program is a serious endeavor and a time-consuming undertaking.  Because of this, it warrants a strategic business approach.  Taking this approach may require additional resources initially, but in the end, the forethought will be reflected in the quality of the program it produces.  With the sophistication of the modern attacker and the constantly evolving threat landscape, don’t we owe it to ourselves to approach security operations like we would any other important business function?

Integrating Actionable Intelligence

Last week, I authored a piece in SecurityWeek discussing the integration of actionable intelligence into an operational setting.  The piece can be found here: http://www.securityweek.com/integrating-actionable-intelligence.  In my experience, integrating intelligence into a security operations program is important and is recognized as such, but it is not a topic that is widely understood.  Because of that, there is quite a bit of buzz around the topic, but organizations realize widely varying results.  I tried to bring some practical approaches towards progress in what is a challenging problem.  Have a look and let me know what you think.  My hope is that many can benefit from the column.

Chronic Ocean Boilers

Recently, I met with an organization that was interested in speaking with me because of my experience in the security operations realm. After a few minutes, it became apparent that the organization had many of the same challenges I often see in organizations that have immature security operations functions. These challenges include, but are not limited to, incomplete logging, lack of visibility into network traffic, no communicated leadership vision, no formal process, no unified work queue of events, incomplete staffing, inadequate training, and other challenges. That didn’t surprise me in the least, but what did surprise me was the direction in which the organization wanted to take the conversation.

The organization began asking me about machine learning and other sophisticated data mining techniques, insisting that “we already have data, but we need to know what to do with that data”. Long term, yes, absolutely -- “digging” (through a variety of techniques, whether manual or automated) is an important part of a mature security operations function. But lacking a mature security operations function, does it make sense to jump ahead to machine learning without first visiting the foundational components of security operations? I don’t think so, and I’ll explain why.

I’ve noticed over the course of my career that people sometimes want to boil the ocean. In other words, rather than proceed step by step through the process of building and maturing a security operations function, they want to move immediately into very advanced topics. This is more than just impractical and nearly impossible -- in my experience, it prevents the step by step progression that ultimately leads to a mature security operations function.

In my experience, there is a hierarchy of needs -- almost like Maslow’s hierarchy of needs, but for security operations. That hierarchy looks something like this:

• Awareness: The first step to a mature security operations function is understanding that you need one.
• Vision: Leadership vision and the communication of that vision are an essential foundation for a successful security operations function.
• Process: A formal incident response process from the strategic level down to the tactical level is critical. This instructs and informs the security team, and serves to show executives, partners, customers, and other stakeholders that the organization takes a formal approach to security.
• Instrumentation: Proper network and endpoint instrumentation provides us the data we need to understand what’s going on within our organization.
• Content: Content development (explained in a previous post) allows us to leverage our network and endpoint data to produce reliable, high fidelity, actionable alerting.
• Unified Work Queue: Sending our actionable alerts to a unified work queue allows us to focus our security operations resources and provide an orderly workflow in an often chaotic environment.
• Staffing: Talented people are needed alongside process and technology to make a successful security operations program.
• Training: The team needs to be trained not only on the technology, but also the process, as well as the strategic vision and philosophy of the organization.
• Operations: Smooth operations require adequate staffing, good communication, proper shift handover, and a large amount of coordination.
• Intelligence: The knowledge of 100 organization will always be greater than the knowledge of just one. As such, integrating actionable intelligence is an important need that arises when the organization has almost reached maturity.
• Information Sharing: Organizations with mature security operations functions will often share intelligence, techniques, and process with one another. Achieving this level is a tremendous accomplishment and usually comes after a significant amount of time has been invested in maturing the security operations function.

This hierarchy is very high level and really only scratches the surface of course, but you can see that a mature security operations function doesn’t build itself. If an organization works its way up the hierarchy of needs, I would argue that at that point, the incorporation of sophisticated data mining techniques would be warranted as a next step in maturity. Before that point though, I’m not sure it is productive to discuss or pursue that angle. Data mining will produce results that need to be investigated further, which requires a strong foundation and a complete hierarchy of needs. Before the security operations function is mature, it's not clear to me that the organization would know how to make sense of the output from data mining techniques. Put another way, investing resources in data mining before the security operations function is mature puts the organization at great risk. Why? Because there are many risks and priorities that take precedent and require more immediate attention. Instead, I recommend a step by step progression to mature the security operations function before moving on to more advanced topics.

Boiling the ocean has never done anyone any good in my experience. First things first.