Docker on YARN

Docker makes it easy to bundle an application with its dependencies and provide full isolation, and YARN now supports Docker as an execution engine for submitted applications. Daniel Templeton explains how YARN's Docker support works, why you'd want to use it, and when you shouldn't.

Docker makes it easy to bundle an application with its dependencies and provide full isolation, and YARN now supports Docker as an execution engine for submitted applications. Daniel Templeton explains how YARN’s Docker support works, why you’d want to use it, and when you shouldn’t. Daniel starts with a very short summary of Docker and the use cases that drive its adoption before jumping into how Docker is supported on YARN. Daniel covers container executors and the various options and digs into the Docker support provided by the LinuxContainerExecutor, and he’ll talk about what it doesn’t do yet, where that creates problems, and the work going on in the community to address the gaps. Along the way, Daniel also explores Apache Slider and the Docker capabilities it brings as it’s being merged into YARN and the use cases for when you’d want to use Docker in a Hadoop environment and when there might be a better alternative, covering both traditional applications, like MapReduce jobs, and longer-running services. You’ll leave with a concrete idea of what’s possible today, what’s coming soon, and when to look to Docker as the solution.