February 12, 2020

174 words 1 min read

Building a distributed real-time stream processing system

Building a distributed real-time stream processing system

Amy Boyle walks you through building, scaling, and monitoring a stream processing pipeline.

Talk Title Building a distributed real-time stream processing system
Speakers A boyle (New Relic)
Conference O’Reilly Velocity Conference
Conf Tag Build systems that drive business
Location London, United Kingdom
Date October 31-November 2, 2018
URL Talk Page
Slides Talk Slides
Video

The future of software is distributed. If you run a backend service of consequence, you’re probably dealing with some sort of distributed system. Stream processing applications form the backbone of New Relic’s data pipeline processing billions of data points a minute. As a result, the company has learned a few useful things about building scalable distributed stream processing systems. While there are many great tools such as Kafka and Docker orchestration upon which to build feature-rich systems, you still need to understand how these building blocks work and how to apply them effectively and reliably at scale. Amy Boyle walks you through building, scaling, and monitoring a stream processing pipeline, drawing on examples from New Relic’s data pipeline. Topics include:

comments powered by Disqus