December 5, 2019

239 words 2 mins read

StreamDM: Advanced data science with Spark Streaming

StreamDM: Advanced data science with Spark Streaming

Heitor Murilo Gomes and Albert Bifet offer an overview of StreamDM, a real-time analytics open source software library built on top of Spark Streaming, developed at Huawei's Noahs Ark Lab and Tlcom ParisTech.
Talk Title StreamDM: Advanced data science with Spark Streaming
Speakers Heitor Murilo Gomes (Télécom ParisTech), Albert Bifet (Télécom ParisTech)
Conference Strata Data Conference
Conf Tag Making Data Work
Location London, United Kingdom
Date May 22-24, 2018
URL Talk Page
Slides Talk Slides
Video

Heitor Murilo Gomes and Albert Bifet offer an overview of StreamDM, a real-time analytics open source software library built on top of Spark Streaming, developed at Huawei’s Noah’s Ark Lab and Télécom ParisTech. StreamDM’s tools and algorithms are specifically designed for data streaming. Due to the large amount of data that is created—and must be processed—in real-time streams, such methods need to be extremely time efficient while using very small amounts of memory. StreamDM is the first library to include advanced stream-mining algorithms for Spark Streaming and is intended to be the open source gathering point for the research and implementation of data streams, while also allowing practical deployments on real-world datasets. This library contains methods for classification, regression, clustering, and frequent pattern mining. Heitor and Albert explain how these advanced methods work in practice, discuss some big data analytics applications in telecommunication networks, compare them with the methods available in MLlib and Spark ML, and demonstrate their ease of use and extensibility.

huawei streaming algorithm dataset spark data analytics data science analytics open source network big data telecom cluster data stream
comments powered by Disqus
The ultimate data scientist's playground: Building a multipetabyte analytic infrastructure for cyber defense

The ultimate data scientist's playground: Building a multipetabyte analytic infrastructure for cyber defense

December 5, 2019

Lee Blum offers an overview of Verint's large-scale cyber-defense system built to serve its data scientists with versatile analytic operations on petabytes of data and trillions of records, covering the company's extremely challenging use case, decision considerations, major design challenges, tips and tricks, and the systems overall results.
Smart agriculture: Blending IoT sensor data with visual analytics

Smart agriculture: Blending IoT sensor data with visual analytics

November 21, 2019

Mike Prorock offers an overview of mesur.io, a game-changing climate awareness solution that combines smart sensor technology, data transmission, and state-of-the-art visual analytics to transform the agricultural and turf management market. Mesur.io enables growers to monitor areas of concern, providing immediate benefits to crop yield, supply costs, farm labor overhead, and water consumption.
Accelerating deep learning on Apache Spark using BigDL with coarse-grained scheduling

Accelerating deep learning on Apache Spark using BigDL with coarse-grained scheduling

November 30, 2019

The BigDL framework scales deep learning for large datasets using Apache Spark. However there is significant scheduling overhead from Spark when running BigDL at large scale. Shivaram Venkataraman and Sergey Ermolin outline a new parameter manager implementation that along with coarse-grained scheduling can provide significant speedups for deep learning models like Inception and VGG.
Continuous machine learning over streaming data

Continuous machine learning over streaming data

November 28, 2019

Roger Barga, Nina Mishra, Sudipto Guha, and Ryan Nienhuis detail continuous machine learning algorithms that discover useful information in streaming data. They focus on explainable machine learning, including anomaly detection with attribution, the ability to reduce false positives through user feedback, and the detection of anomalies in directed graphs.
Deep learning for domain-specific entity extraction from unstructured text

Deep learning for domain-specific entity extraction from unstructured text

November 27, 2019

Mohamed AbdelHady and Zoran Dzunic demonstrate how to build a domain-specific entity extraction system from unstructured text using deep learning. In the model, domain-specific word embedding vectors are trained on a Spark cluster using millions of PubMed abstracts and then used as features to train a LSTM recurrent neural network for entity extraction.
Kafka streaming applications with Akka Streams and Kafka Streams

Kafka streaming applications with Akka Streams and Kafka Streams

November 24, 2019

Dean Wampler compares and contrasts data processing with Akka Streams and Kafka Streams, microservice streaming applications based on Kafka. Dean discusses the strengths and weaknesses of each tool for particular design needs and contrasts them with Spark Streaming and Flink, so you'll know when to choose them instead.