Where to Look?

When I work with clients to build their Security Operations Centers (SOCs)/Incident Response Centers (IRCs), I often see a common challenge.  As I've mentioned previously, most organizations spend a good deal of time instrumenting their network to collect data.  Unfortunately, they don't often give enough thought to how one might analyze all that data.  In other words, the questions of "Where do we put all this data?" and "What type of questions do we want to ask of all this data?" should be asked at the same time the instrumentation of the network is being planned.  As you might expect, this is almost never the case.  As a result, organizations often end up with large amounts of data in various different locations.  There are some data types where there is a good deal of overlap with other data types, which results in redundancy, waste, and long query times (due to excessive volumes of data).  In other data types, there may be (potentially large) gaps in the data, which results in the inability to ask certain (sometimes crucial) questions of the data.

What's striking is that the first question an analyst usually needs to ask is "Where do I go to get the data to answer my question?", rather than "What is the answer that the data provides to my question?".  It's unfortunate.  The good news is that better coordination between the collection side of the enterprise and the analysis side of the enterprise can result in incredible gains analytically.  Something to keep in mind when building a SOC/IRC for sure.