December 10, 2019

178 words 1 min read

Deep Dive: Big Data SIG

Deep Dive: Big Data SIG

This presentation will cover two projects from sig-big-data: Apache Spark on Kubernetes and Apache Airflow on Kubernetes. Kubernetes became a native scheduler backend for Spark in 2.3 and we have been …
Talk Title Deep Dive: Big Data SIG
Speakers Erik Erlandson (Principal Software Engineer, Red Hat), Yinan Li (Software Engineer, Google)
Conference KubeCon + CloudNativeCon North America
Conf Tag
Location Seattle, WA, USA
Date Dec 9-14, 2018
URL Talk Page
Slides Talk Slides
Video

This presentation will cover two projects from sig-big-data: Apache Spark on Kubernetes and Apache Airflow on Kubernetes. Kubernetes became a native scheduler backend for Spark in 2.3 and we have been working on expanding the feature set as well as hardening the integration since then. Apache Airflow on Kubernetes achieved a big milestone with the new Kubernetes Operator for natively launching arbitrary Pods and the Kubernetes Executor that is a Kubernetes native scheduler for Airflow. We will give an overview of the current state and present the roadmap of both projects, and give attendees opportunities to ask questions and provide feedback on roadmaps.

roadm big-data apache spark airflow big data kubernetes oadm
comments powered by Disqus
Making Big Data Processing Portable. The Story of Apache Beam and gRPC

Making Big Data Processing Portable. The Story of Apache Beam and gRPC

December 7, 2019

Big data applications have been an almost exclusive domain of Java and Scala developers. This not only frustrates engineers who prefer other languages and their ecosystems, but also impedes companies …
The ultimate data scientist's playground: Building a multipetabyte analytic infrastructure for cyber defense

The ultimate data scientist's playground: Building a multipetabyte analytic infrastructure for cyber defense

December 5, 2019

Lee Blum offers an overview of Verint's large-scale cyber-defense system built to serve its data scientists with versatile analytic operations on petabytes of data and trillions of records, covering the company's extremely challenging use case, decision considerations, major design challenges, tips and tricks, and the systems overall results.
Smart diagnosis in healthcare with deep learning

Smart diagnosis in healthcare with deep learning

December 3, 2019

Deep learning with ConvNet in particular has emerged as a promising tool in medical research labs and diagnostic centers to help analyze images and scans, and systems are now surpassing human capability for manual inspection. Nishant Sahay explains how to apply deep learning to analyze high-end microscope images and X-ray scans to provide accurate diagnosis.
Accelerating deep learning on Apache Spark using BigDL with coarse-grained scheduling

Accelerating deep learning on Apache Spark using BigDL with coarse-grained scheduling

November 30, 2019

The BigDL framework scales deep learning for large datasets using Apache Spark. However there is significant scheduling overhead from Spark when running BigDL at large scale. Shivaram Venkataraman and Sergey Ermolin outline a new parameter manager implementation that along with coarse-grained scheduling can provide significant speedups for deep learning models like Inception and VGG.
Automatic 3D MRI knee damage classification with 3D CNN using BigDL on Spark

Automatic 3D MRI knee damage classification with 3D CNN using BigDL on Spark

November 29, 2019

Damage to the meniscus is a physically limiting injury that can lead to further medical complications. Automatically classifying this damage at the time of an MRI scan would allow quicker and more accurate diagnosis. Jennie Wang, Valentina Pedoia, Berk Norman, and Yulia Tell offer an overview of their classification system built with 3D convolutional neural networks using BigDL on Apache Spark.
Cuttlefish: Lightweight primitives for online tuning

Cuttlefish: Lightweight primitives for online tuning

November 28, 2019

Tomer Kaftan offers an overview of Cuttlefish, a lightweight framework prototyped in Apache Spark that helps developers adaptively improve the performance of their data processing applications by inserting a few library calls into their code. These calls construct tuning primitives that use reinforcement learning to adaptively modify execution as they observe application performance over time.