Aggregation and Outbound Denies, A Powerful Combination

I have previously blogged (separately) about the merits of both aggregation and looking at outbound denied traffic.  It occurs to me that it is worth a separate post to blog about the powerful combination of aggregation and outbound denies.

If one takes a rich data source (such as proxy logs), looks at the outbound denied traffic, and aggregates by certain key fields, such as:

• Source IP Address
• Destination IP Address
• Domain
• URL
• Request Method (e.g., GET, POST, etc.)
• Count (ordering by Count in descending order)

The results (say, over a 24 hour time slice of data) are generally quite interesting.  Taking a step back, we see that what we're essentially doing is slicing the data in such a way so as to extract repetitive activity that is being denied by the proxy (for whatever reason).  Generally, humans do not create traffic that fits this criteria, but rather, machines do.  Machine generated activity is generally quite interesting from an analytical perspective, though sometimes it is a mere nuisance (e.g., toolbar generated traffic).  On the frequent occasion that the traffic is malicious, this approach to slicing the data is quite helpful in finding the activity of concern.