Merits of Meta-Data

In my previous post, I discussed the logic behind keeping a record of all traffic traversing the network. As we all know, the traffic that a large, enterprise network generates is incredibly voluminous. So what does one do to best keep eyes on the network? I believe the key here is meta-data. Meta-data describes the envelope information about transactions/conversations on the network, but doesn't include the content of the actual conversation. Network flow data is one type of meta-data, while layer 7 enriched meta-data (discussed in a previous blog post) is another type. This allows for several key advantages:

• Long term retention of data for auditing and forensics purposes without the need for large amounts of expensive disk space.
• The ability to see all the data without needing to sample, filter, or drop certain traffic.
• Rapid search capability over vast quantities of data collected over long periods of time.
  

Now, for sure there is information in the packet data that is helpful for identifying the true nature of malicious or suspicious traffic. I believe that meta-data based technologies and packet-based technologies can work together beautifully here. Meta-data allows one to craft incisive queries designed to interrogate the data so as to identify network traffic that requires further investigation. I call these jumping off points (also discussed in a previous blog post). From there, the packet data can be consulted to assist in the investigation (presuming that the retention window for the packet data has not already expired).

As the amount of traffic on our networks continues to grow, I believe that we as a community will need to get used to the network traffic analysis model/work flow described above. I sometimes refer to it as breadth, then depth. I believe it to be a model capable of scaling with the data volumes of the present and future.