Building an Enterprise/Cloud Analytics Platform with Jupyter Enterprise Gateway

Data science and analytics departments are now common place for enterprises determined to maximize their operations. While Jupyter Notebooks have significantly decreased the cost of admission into this space, enterprises are finding that data science at scale is difficult within the current framework. Jupyter Enterprise Gateway is designed to address these scalability issues for the enterprise.

Notebook kernel processes are the heart and soul of the Notebook application and are responsible for the submission of potentially extreme and resource intensive operations against vast amounts of data, particularly in big data analytics. Running these resource-consuming processes on the same node results in a crippled server struggling to meet the needs of subsequent kernel creation requests. This is hardly a recipe for success within an enterprise servicing dozens or even hundreds of data scientists simultaneously striving to unlock the secrets within their data sets. With Jupyter Enterprise Gateway, enterprises are able to distribute kernels across the compute cluster, consisting of different capabilities (e.g., GPUs, Cores, Memory, etc.) and leveraging the resource allocation semantics of the underlying resource managers. This is accomplished via a pluggable framework that enables support for additional resource managers allowing your enterprise to leverage this functionality. Jupyter Enterprise Gateway currently supports for Hadoop YARN, IBM Spectrum Conductor, and Kubernetes resource managers with contributions welcome for others.