Architecting data platforms for cybersecurity

Data is becoming a crucial weapon to secure an organization against cyber threats. Charaka Goonatilake shares strategies for designing effective data platforms for cybersecurity using big data technologies, such as Spark and Hadoop, and explains how these platforms are being used in real-world examples of data-driven security.

In today’s world, with cyber incidents reported almost daily, security teams are increasingly turning to data for answers. Data-driven approaches can prove immensely valuable in providing visibility to support decision making and drive action across the entire cybersecurity lifecycle. Data platforms to support security operations teams in their reactive efforts to detect and respond to security incidents have existed for a long time, from SIEMs historically and Apache Metron to Spot. In addition, a whole new class of traditionally underserved stakeholders and use cases is emerging from security executives who need strategic decision support to deliver proactive initiatives that measure and mitigate cyber risks. Designing successful data solutions for the cybersecurity domain can be a daunting task. The diversity of problems to be solved for various stakeholders in and around a security function leads to an array of complex and potentially competing data and analysis requirements. This complexity initially arises from the need to collect and prepare data of any type from wherever it resides and however it’s exposed. The data must be then stored in a way that can accommodate a range of access patterns. Finally, interfaces must exist to promote wide accessibility to allow the range of platform users to analyze the data and consume insights, taking these users’ varying data analysis skill levels into account. Once you’ve understood your users and their needs, you face the challenge of navigating the vast sea of data technologies vying for your attention to arrive at a solution. But with the proliferation of open source and proprietary technology options, each with their own trade-offs, how do you deliver a scalable and flexible data platform that will serve your security organization for years to come? Charaka Goonatilake explores the key drivers that influence the architecture of a cyber data platform and explains how to deliver on these requirements using open source big data technologies like Spark and the Hadoop ecosystem. Charaka walks you through real-life lessons learned and the successes and failures experienced while building and evolving data platforms. Topics include: