February 26, 2020

209 words 1 min read

Lessons Learned from the Migration to Apache Airflow

Lessons Learned from the Migration to Apache Airflow

Apache Airflow is an open-source tool for orchestrating complex workflows and data processing pipelines.In this talk, Radek Maciaszek will present his learnings from the migration of machine learning …
Talk Title Lessons Learned from the Migration to Apache Airflow
Speakers Radek Maciaszek (Chief Architect, Skimlinks)
Conference Open Source Summit + ELC North America
Conf Tag
Location San Diego, CA, USA
Date Aug 19-23, 2019
URL Talk Page
Slides Talk Slides
Video

Apache Airflow is an open-source tool for orchestrating complex workflows and data processing pipelines.In this talk, Radek Maciaszek will present his learnings from the migration of machine learning and big data processing pipelines to Apache Airflow.Radek will discuss examples of how are they using Airflow to power their company big data infrastructure where they analyze hundreds of terabytes of data. Examples will cover the building of the ETL pipeline and use of Airflow to manage the machine learning Spark pipeline workflow.This talk will cover the basic Airflow concepts and show real-life examples of how to define your own workflows in the Python code. The talk will finish with more advanced topics related to Apache Airflow, such as adding custom task operators, sensors and plugins as well as best practices and both the pros and cons of this tool.

open-source code apache spark etl airflow infrastructure big data machine learning python pipeline
comments powered by Disqus
Analytics Zoo: Distributed TensorFlow in production on Apache Spark

Analytics Zoo: Distributed TensorFlow in production on Apache Spark

December 27, 2019

Yuhao Yang and Jennie Wang demonstrate how to run distributed TensorFlow on Apache Spark with the open source software package Analytics Zoo. Compared to other solutions, Analytics Zoo is built for production environments and encourages more industry users to run deep learning applications with the big data ecosystems.
Cloud, Containerization, Virtualization and SDN Controllers

Cloud, Containerization, Virtualization and SDN Controllers

February 19, 2020

To develop and build a modern Cloud infrastructure or DevOps implementation that both Docker and Kubernetes have revolutionized the era of software development and operations.Together we will develop …
Machine learning challenges at LinkedIn: Spark, TensorFlow, and beyond

Machine learning challenges at LinkedIn: Spark, TensorFlow, and beyond

February 18, 2020

From people you may know (PYMK) to economic graph research, machine learning is the oxygen that powers how LinkedIn serves its 630M+ members. Zhe Zhang provides you with an architectural overview of LinkedIns typical machine learning pipelines complemented with key types of ML use cases.
Architecting a data analytics service both in the public cloud and in the on-premise private cloud: ETL, BI, and machine learning (sponsored by SK Holdings)

Architecting a data analytics service both in the public cloud and in the on-premise private cloud: ETL, BI, and machine learning (sponsored by SK Holdings)

February 16, 2020

Jungwook Seo walks you through a data analytics platform in the cloud by the name of AccuInsight+ with eight data analytic services in the CloudZ (one of the biggest cloud service providers in Korea), which SK Holdings announced in January 2019.
Deep learning on Apache Spark at CERNs Large Hadron Collider with Analytics Zoo

Deep learning on Apache Spark at CERNs Large Hadron Collider with Analytics Zoo

February 15, 2020

Sajan Govindan outlines CERNs research on deep learning in high energy physics experiments as an alternative to customized rule-based methods with an example of topology classification to improve real-time event selection at the Large Hadron Collider. CERN uses deep learning pipelines on Apache Spark using BigDL and Analytics Zoo open source software on Intel Xeon-based clusters.
Scalable anomaly detection with Spark and SOS

Scalable anomaly detection with Spark and SOS

February 10, 2020

Jeroen Janssens dives into stochastic outlier section (SOS), an unsupervised algorithm for detecting anomalies in large, high-dimensional data. SOS has been implemented in Python, R, and, most recently, Spark. He illustrates the idea and intuition behind SOS, demonstrates the implementation of SOS on top of Spark, and applies SOS to a real-world use case.