November 29, 2019

219 words 2 mins read

Big data insights equal big money: Stories from the trenches at GoDaddy

Big data insights equal big money: Stories from the trenches at GoDaddy

GoDaddy ingests and analyzes over 100,000 data points per second. Felix Gorodishter discusses the company's big data journey from ingest to automation, how it is evolving its systems to scale to over 10 TB of new data per day, and how it uses tools like anomaly detection to produce valuable insights, such as the worth of a reminder email.
Talk Title Big data insights equal big money: Stories from the trenches at GoDaddy
Speakers Felix Gorodishter (GoDaddy)
Conference Strata Data Conference
Conf Tag Big Data Expo
Location San Jose, California
Date March 6-8, 2018
URL Talk Page
Slides Talk Slides
Video

Harnessing the power of data is the future of business. GoDaddy is using its massive amounts of data to improve customer experience and internal operations, performing real-time anomaly detection, log analysis, alerting, remediation, and batch reporting on hundreds of thousands of events per second across its products and systems data. Felix Gorodishter discusses the company’s transformation from a farm of data silos to a centralized platform to ingest, analyze, and visualize data throughout the enterprise. You’ll learn how GoDaddy collects and uses data from business units like hosting and domains as well as from network and hardware events across its fleet of servers and network devices, using a wide range of technologies, including Kafka, Spark, Hadoop, and Elasticsearch.

kafka spark hadoop network big data anomaly detection elasticsearch hardware
comments powered by Disqus
Playing well together: Big data beyond the JVM with Spark and friends

Playing well together: Big data beyond the JVM with Spark and friends

November 22, 2019

Holden Karau and Rachel Warren explore the state of the current big data ecosystem and explain how to best work with it in non-JVM languages. While much of the focus will be on Python + Spark, the talk will also include interesting anecdotes about how these lessons apply to other systems (including Kafka).
Smart agriculture: Blending IoT sensor data with visual analytics

Smart agriculture: Blending IoT sensor data with visual analytics

November 21, 2019

Mike Prorock offers an overview of mesur.io, a game-changing climate awareness solution that combines smart sensor technology, data transmission, and state-of-the-art visual analytics to transform the agricultural and turf management market. Mesur.io enables growers to monitor areas of concern, providing immediate benefits to crop yield, supply costs, farm labor overhead, and water consumption.
Automatic 3D MRI knee damage classification with 3D CNN using BigDL on Spark

Automatic 3D MRI knee damage classification with 3D CNN using BigDL on Spark

November 29, 2019

Damage to the meniscus is a physically limiting injury that can lead to further medical complications. Automatically classifying this damage at the time of an MRI scan would allow quicker and more accurate diagnosis. Jennie Wang, Valentina Pedoia, Berk Norman, and Yulia Tell offer an overview of their classification system built with 3D convolutional neural networks using BigDL on Apache Spark.
Classifying job execution using deep learning

Classifying job execution using deep learning

November 28, 2019

Ash Munshi shares techniques for labeling big data apps using runtime measurements of CPU, memory, I/O, and network and details a deep neural network to help operators understand the types of apps running on the cluster and better predict runtimes, tune resource utilization, and increase efficiency. These methods are new and are the first approach to classify multivariate time series.
Cuttlefish: Lightweight primitives for online tuning

Cuttlefish: Lightweight primitives for online tuning

November 28, 2019

Tomer Kaftan offers an overview of Cuttlefish, a lightweight framework prototyped in Apache Spark that helps developers adaptively improve the performance of their data processing applications by inserting a few library calls into their code. These calls construct tuning primitives that use reinforcement learning to adaptively modify execution as they observe application performance over time.
The secret sauce behind LinkedIn's self-managing Kafka clusters

The secret sauce behind LinkedIn's self-managing Kafka clusters

November 20, 2019

LinkedIn runs more than 1,800+ Kafka brokers that deliver more than two trillion messages a day. Running Kafka at such a scale makes automated operations a necessity. Jiangjie Qin shares lessons learned from operating Kafka at scale with minimum human intervention.