Building stream processing as a service at Netflix

Steven Wu explains how Netflixs SPaaS platform empowers users to focus on extracting insights from data streams and build stream processing applications and shares lessons learned building and operating the largest SPaaS use case: Netflixs Keystone data pipeline, a self-serve platform for creating near-real-time event pipelines that processes three trillion events and 12 PB of data every day.

Over 109 million subscribers enjoy more than 125 million hours of TV shows and movies per day on Netflix. This leads to a massive amount of data flowing through its data pipeline that can be used to improve service and user experience and power various data analytic cases like personalization, operational insight, and fraud detection. Netflix is now building a stream-processing-as-a-service (SPaaS) platform on top of Apache Flink that is self-serve, operable, scalable, fault tolerant, and multitenant. Steven Wu explains how Netflix’s SPaaS platform empowers users to focus on extracting insights from data streams and build stream processing applications. He also shares lessons learned building and operating the largest SPaaS use case: Netflix’s Keystone data pipeline, a self-serve platform for creating near-real-time event pipelines that processes three trillion events and 12 PB of data every day.