January 8, 2020

359 words 2 mins read

Mastering data with Spark and machine learning

Mastering data with Spark and machine learning

Enterprise data on customers, vendors, and products is often siloed and represented differently in diverse systems, hurting analytics, compliance, regulatory reporting, and 360 views. Traditional rule-based MDM systems with legacy architectures struggle to unify this growing data. Sonal Goyal offers an overview of a modern master data application using Spark, Cassandra, ML, and Elastic.
Talk Title Mastering data with Spark and machine learning
Speakers Sonal Goyal (Nube)
Conference Strata Data Conference
Conf Tag Making Data Work
Location London, United Kingdom
Date April 30-May 2, 2019
URL Talk Page
Slides Talk Slides
Video

Enterprise data on customers, vendors, and products is often siloed and represented differently in diverse systems, hurting analytics, compliance, regulatory reporting, and 360 views. Traditional rule-based MDM systems with legacy architectures struggle to unify this growing data. Further, each source and data type has its own schema and format, data volumes run into millions of records, and linking similar records is a fuzzy matching and computationally expensive exercise—making this a challenging undertaking. Sonal Goyal offers an overview of the design and architecture of a modern master data application using Spark, Cassandra, ML, and Elastic. The application unifies nontransactional master data in multiple data domains like customer, organization, and product through multiple systems like ERP, CRM, and custom applications of different business units, using the Spark Data Source API and machine learning. Sonal explains how the abstraction offered by the Data Source API allows users to consume and manipulate the different datasets easily. After aligning required attributes, Spark is used to cluster and classify probable matches using a human-in-the-loop feedback system. These matched and clustered records are persisted to Cassandra and exposed to data stewards through an AJAX-based GUI. The Spark job also indexes the records to Elastic, which lets the data steward query and search clusters more effectively. Sonal covers the end-to-end flow, design, and architecture of the different components as well as the configuration per source and type to support the different and unknown datasets and schemas. Along the way, she details the performance gains using Spark, machine learning for data matching, and stewardship as well as the role of Cassandra and Elastic in the application.

api performance dataset spark analytics machine learning cassandra cluster
comments powered by Disqus
Building high-performance text classifiers on a limited labeling budget

Building high-performance text classifiers on a limited labeling budget

December 26, 2019

Robert Horton, Mario Inchiosa, and Ali Zaidi demonstrate how to use three cutting-edge machine learning techniquestransfer learning from pretrained language models, active learning to make more effective use of a limited labeling budget, and hyperparameter tuning to maximize model performanceto up your modeling game.
LSTM-based time series anomaly detection using Analytics Zoo for Spark and BigDL

LSTM-based time series anomaly detection using Analytics Zoo for Spark and BigDL

January 8, 2020

Collecting and processing massive time series data (e.g., logs, sensor readings, etc.) and detecting the anomalies in real time is critical for many emerging smart systems, such as industrial, manufacturing, AIOps, and the IoT. Guoqiong Song explains how to detect anomalies in time series data using Analytics Zoo and BigDL at scale on a standard Spark cluster.
Spark NLP in action: How Indeed applies NLP to standardize rsum content at scale

Spark NLP in action: How Indeed applies NLP to standardize rsum content at scale

January 6, 2020

Alexander Thomas and Alexis Yelton demonstrate how to use Spark NLP and Apache Spark to standardize semistructured text, illustrated by Indeed's standardization process for rsum content.
The vegan data diet: How Wikipedia cuts down privacy issues while keeping data fit

The vegan data diet: How Wikipedia cuts down privacy issues while keeping data fit

January 4, 2020

Analysts and researchers studying Wikipedia are hungry for long-term data to build experiments and feed data-driven decisions. But Wikipedia has a strict privacy policy that prevents storing privacy-sensitive data over 90 days. Marcel Ruiz Forns explains how the Wikimedia Foundation's analytics team is working on a vegan data diet to satisfy both.
Industrialized capsule networks for text analytics

Industrialized capsule networks for text analytics

December 30, 2019

Vijay Agneeswaran and Abhishek Kumar offer an overview of capsule networks and explain how they help in handling spatial relationships between objects in an image. They also show how to apply them to text analytics. Vijay and Abhishek then explore an implementation of a recurrent capsule network and benchmark the RCN with capsule networks with dynamic routing on text analytics tasks.
Turn devices into data scientistsat the edge

Turn devices into data scientistsat the edge

December 28, 2019

Todays approach to processing streaming data is based on legacy big-data centric architectures, the cloud, and the assumption that organizations have access to data scientists to make sense of it allleaving organizations increasingly overwhelmed. Simon Crosby shares a new architecture for edge intelligence that turns this thinking on its head.