Collaboration and automated operation as literate computing for reproducible infrastructure

Jupyter is useful for DevOps. It enables collaboration between experts and novices to accumulate infrastructure knowledge, while automation via notebooks enhances traceability and reproducibility. Yoshi Nobu Masatani shows how to combine Jupyter with Ansible for reproducible infrastructure and explores knowledge, workflow, and customer support as literate computing practices.

We have been applying the Jupyter Notebook to operational engineering for years to meet various research and educational customers’ needs for cloud infrastructure, but Jupyter is useful for DevOps too. It enables collaboration between experts and novices to accumulate infrastructure knowledge, while automation via notebooks enhances traceability and reproducibility. Yoshi Nobu Masatani shows how to combine Jupyter with Ansible for reproducible infrastructure and explores knowledge, workflow, and customer support as literate computing practices. Nobu shares best practices and use cases for utilizing Jupyter for collaborative communication about design and process and managing reproducible infrastructure throughout its lifecycle. The National Institute of Informatics has centralized the Jupyter Notebook in its cloud operations. Both ad hoc and recurring operations are conducted under Jupyter: every daily ad hoc operation is recorded as a notebook for traceability and reproducibility, and elaborated workflows are shared and standardized as reproducible notebooks and automate operations using Ansible. For reproducibility, it is as important to be able to share information and processes with others as it is to actually automate complex operations for infrastructures. Jupyter enables users to describe automated operations as live code and share predicted and reproducible outcomes with others. Its narrative stories allow collaborative communication between experts and novices in order to accumulate infrastructure knowledge within an operation team. Moreover, it enables an efficiently shared understanding of how an infrastructure is usable and actually works between tech ops and nontech users. Narrative stories also help communicate with users about delivered services and customized conditions considering reproducibility. NII’s practices cover the entire infrastructure lifecycle, from initial design and deployment to daily operations and maintenance, rebuilding for new target capacity and throughput, trouble shooting, and customer support and tuned customization. There are available DevOps tools for these tasks, but each automation is within a silo—just a portion of the lifecycle. Enter literate computing, which augments collaboration and makes automation beyond silos possible. Nobu shares NII’s literate computing practices for reproducible infrastructure. Topics include: For more information, check out a short video, Literate Computing for Reproducible Infrastructure, exploring these practices. You can find a Hadoop example on GitHub, which includes a set of notebooks and Ansible playbooks that cover several phases of a Hadoop cluster’s lifecycle, the initial design (accommodation) and automated deployment, operational alignment, and postdeployment expansion and decommission. Notebook extensions for robust infrastructure operations can be found here.