November 29, 2019

158 words 1 min read

Approximation data structures in streaming data processing

Approximation data structures in streaming data processing

Debasish Ghosh explores the role that approximation data structures play in processing streaming data. Typically, streams are unbounded in space and time, and processing has to be done online using sublinear space. Debasish covers the probabilistic bounds that these data structures offer and shows how they can be used to implement solutions for fast and streaming architectures.
Talk Title Approximation data structures in streaming data processing
Speakers Debasish Ghosh (Lightbend)
Conference Strata Data Conference
Conf Tag Big Data Expo
Location San Jose, California
Date March 6-8, 2018
URL Talk Page
Slides Talk Slides
Video

Debasish Ghosh explores the role that approximation data structures (Bloom filtera, sketches, HyperLogLog, etc.) play in processing streaming data. Typically, streams are unbounded in space and time, and processing has to be done online using sublinear space. Debasish covers the probabilistic bounds that these data structures offer and shows how they can be used to implement solutions for fast and streaming architectures. Topics include:

streaming
comments powered by Disqus
Apache Spark programming

Apache Spark programming

November 29, 2019

Brooke Wenig walks you through the core APIs for using Spark, fundamental mechanisms and basic internals of the framework, SQL and other high-level data access tools, and Sparks streaming capabilities and machine learning APIs.
Best practices for productionizing Apache Spark MLlib models

Best practices for productionizing Apache Spark MLlib models

November 29, 2019

Joseph Bradley discusses common paths to productionizing Apache Spark MLlib models and shares engineering challenges and corresponding best practices. Along the way, Joseph covers several deployment scenarios, including batch scoring, Structured Streaming, and real-time low-latency serving.
Continuous machine learning over streaming data

Continuous machine learning over streaming data

November 28, 2019

Roger Barga, Nina Mishra, Sudipto Guha, and Ryan Nienhuis detail continuous machine learning algorithms that discover useful information in streaming data. They focus on explainable machine learning, including anomaly detection with attribution, the ability to reduce false positives through user feedback, and the detection of anomalies in directed graphs.
Enough data engineering for a data scientist; or, How I learned to stop worrying and love the data scientists

Enough data engineering for a data scientist; or, How I learned to stop worrying and love the data scientists

November 26, 2019

Stephen O'Sullivan takes you along the data science journey, from onboarding data (using a number of data/object stores) to understanding and choosing the right data format for the data assets to using query engines (and basic query tuning). You'll learn some new skills to help you be more productive and reduce contention with the data engineering team.
How to build leakproof stream processing pipelines with Apache Kafka and Apache Spark

How to build leakproof stream processing pipelines with Apache Kafka and Apache Spark

November 25, 2019

When Kafka stream processing pipelines fail, they can leave users panicked about data loss when restarting their application. Jordan Hambleton and Guru Medasani explain how offset management provides users the ability to restore the state of the stream throughout its lifecycle, deal with unexpected failure, and improve accuracy of results.
Kafka streaming applications with Akka Streams and Kafka Streams

Kafka streaming applications with Akka Streams and Kafka Streams

November 24, 2019

Dean Wampler compares and contrasts data processing with Akka Streams and Kafka Streams, microservice streaming applications based on Kafka. Dean discusses the strengths and weaknesses of each tool for particular design needs and contrasts them with Spark Streaming and Flink, so you'll know when to choose them instead.