Quantopian integrates financial data from vendors around the globe. As the scope of its operations outgrew cron, the company turned to Apache Airflow, a distributed scheduler and task executor. James Meickle explains how in less than six months, Quantopian was able to rearchitect brittle crontabs into resilient, recoverable pipelines defined in code to which anyone could contribute.

Apache Airflow is an open source workflow orchestration engine that allows users to write Directed Acyclic Graph (DAG)-based workflows using a simple Python library. Airflow offers a wide range of nat …

This presentation will cover two projects from sig-big-data: Apache Spark on Kubernetes and Apache Airflow on Kubernetes. Kubernetes became a native scheduler backend for Spark in 2.3 and we have been …