Genesis: Automating data center management with help from PXE and Chef

The flexibility and speed offered by cloud computing solutions have raised the bar for bare metal deployments. Automation is essential to speedy, reliable provisioning and capacity management. David Radcliffe explores the tools Shopify uses, such as Genesis, to automate its data center and empower developers to move quickly and keep up with the times.

The flexibility and speed offered by cloud computing solutions have raised the bar for bare metal deployments. Automation is essential to speedy, reliable provisioning and capacity management. David Radcliffe explores the tools Shopify uses, such as Genesis, to automate its data center and empower developers to move quickly and keep up with the times. Three years ago, Shopify kept track of its data center infrastructure in a Google sheet. (Surprisingly, this is not an uncommon practice in the industry.) Shopify embraced Tumblr’s Collins and took a leap forward into the world of automation. Genesis, its replacement for Collins, started as a hack days project and has grown to become an essential tool for managing its data center deployments. With an iPad in each data center, custom barcodes, and SAML logins, we’re able to provide remote hands with the ability to share a view of the data center. Automation begins when servers arrive and inventory themselves. Every server is production ready within mere minutes. Utilizing a Chef cookbook CI, built around Docker and Test Kitchen, Shopify is able to produce server images on every merge to master. These are cached in the data center, allowing the PXE install system to reimage and repurpose a system in minutes. Shopify’s “spare allocation service” allows a user to specify the number, type, and data center where they want servers, and they are supplied behind the scenes. Within 15 minutes, 80 servers can be made fully operational production nodes—all from scratch.