This article originally appeared in the Sept. 2012 issue of INTERNET TELEPHONY
Why is everyone so hyped over big data? Possibly it’s because people are now realizing the power of big data.
The security industry has realized that log data is an incredibly rich source of information for detecting security intrusions, and has since developed a taste for more and more logs.
Log correlation has since then followed as IT professionals realized that individual log entries by themselves meant very little, but when placed into context against one another illustrate more than just system-level events. They illustrate behavioral context – clusters of individual log lines that can be translated into records of human-readable actions.
Security is still in the early days of this science and practice of event correlation. Methods and results are rarely shared with the community, the target for what is effective keeps moving, and yet we're already talking about big data.
Vast databases of information being mined for emergent patterns and used to process simulations over and over are hardly new to the world. The finance, medical and aerospace industries have spent years in this realm.
How is it, then, that the security world has not previously tapped into this pool of expertise before now to help us glean the knowledge lying dormant within our vast supplies of data? Quite simply, it's because we still don't know what questions to ask in the first place.
It's worth performing a short recap on emerging big data technologies out there and why they differ from being just large databases.
Although there are many implementations of these technologies, they all derive from two core functions: NoSQL and MapReduce.
NoSQL is a difficult beast to define even among the experts in that field.
Unlike an RDBMS, where the schema must be well-defined before data is stored and changes to that schema, when live data is present it becomes increasingly more unfeasible, NoSQL data stores may freely adapt the nature of the records they store over time.
They are optimized for rapid retrieval of information at the possible expense of consistency of data (they do not comply to ACID). To wit, they are excellent systems with which to do analytical work but have inherent issues if treated as the authoritative repository.
Accordingly for the same audience, MapReduce's key features include the ability to perform information retrieval and calculation over a widely distributed data storage. A practical example would be that if individual devices had their log storage implemented in a MapReduce-capable manner, then a centralized log storage mechanism may no longer be required – a single query could be performed across all logs on all devices simultaneously.
Generally speaking, there is comparatively little need for the end user to optimize their query sets to take advantage of MapReduce's distributed nature.
So, we can immediately see some of the reasons these two technologies have raised excitement and promise to the information security world.
Increased speed on complex queries across large quantities of data is a vital force-multiplier for security analysts; the ability to query every machine that has accessed a particular URI in the last 90 days in minutes (not hours or even days) cannot be overlooked.
The flexibility to bring additional data to supplement existing records works in lockstep with the inherent nature of security information: that it is comparatively a domain of unstructured data. Freedom from data schemas that fail to take into account the information that is vital to the organization we are trying to defend will allow us to make better correlations and ask better questions from our data.
Between these two factors, we can see where the excitement comes from, and yet we still have to return back to the same issues we've struggled with before the advent of big data.
We still aren't very good at asking the right questions from our data.
In security analytics, it's often the relations between the data (not the data itself) that is important. Just as detective work is a matter of connecting the dots, so are the relations between our data points for the true information. (Log Correlation itself is about looking for and exposing those relations.)
As IT professionals, we share a particular reticence to trust anything we didn't do hands-on ourselves; as security professionals, this trait becomes magnified. Perhaps that’s because of the fact that the concepts we are looking for (exposures, risks, threat surfaces) are so difficult to define that we are still stuck in the stone ages of bar charts and keyword searches when it comes to data analytics.
No amount of big data is going to save us until we can learn to formulate better questions for that data. Perhaps it's time that we accept that the problems we're approaching now (trying to boil an ocean of data points into digestible information) is not unique to us. Information security as a discipline may have much to learn from other technology fields. It's a tough pill to swallow when you think of how much we collectively berate the rest of IT as being the source of all our issues in the first place.
Bioinformatics places emphasis on discovering the nature of interactions and relations between their points of data, since this is intrinsic to how biology operates too. It won't take long before you find a plethora of advanced (and aesthetically pleasing) visualization techniques being used to present and explore data relations, like the CIRCOS system.
This field has made great strides in distilling down complex data relationships into advanced visualization techniques that maximize the ability of human pattern recognition abilities to discern inferences that are difficult to make programmatically.
Ask better questions, discover relationships, create hypotheses and test them against more data; rinse, repeat – the scientific method.
Big data will not magically enable us to discern better answers until we come up with better questions to explore the relationships between our data more thoroughly.
The field of log correlation could make great strides if we were to establish an open format for exchanging ideas for correlations in a vendor-neutral manner and collectively discussing what is effective within the field instead of how we operate today. Information security is evolving into areas well explored within other fields. Our issues with discovering relations and implications from our oceans of unstructured data are at the heart of the field of complex event processing.
We're moving into territory where we are not as alone as we think; if we are going to reap the benefits that big data promises and not let this become another failed fad, then we have to start overcoming our isolationist attitude and start inviting experts from other disciplines to join us and teach us how to use this new toolset.
Edited by Stefania Viscusi